Skip to content

Troubleshooting

Jump to bottom
Noah Harasz edited this page Apr 18, 2023 · 15 revisions

Each heading here is an issue you could run into.

Contents:

Python

If the environment setup script fails to install Python3 for you, report the error to Numerai and then install it manually with one of the following options:

Docker

If the environment setup script fails to install Docker for you, report the error to Numerai then read the following to get a working installation on your machine. For PCs, you may need to activate virtualization in your BIOS before installing.

MacOS and Windows 10

Just install Docker Desktop to get it running. You should also increase the RAM allocated to the VM:

  1. Open the Docker Desktop GUI then
  2. Click Gear in top right corner
  3. Select Resources > Advanced in left sidebar
  4. Use slider to allocate more memory (leave a few gigs for your OS and background applications, otherwise your computer might crash)

Linux

Check here for instructions for your distro: https://docs.docker.com/engine/install/

Older PCs: Docker Toolbox

If your machine is older and/or doesn't have Hyper-V enabled, then you will have to follow these steps to install docker toolbox on your machine:

  1. Install Oracle VirtualBox for your Operating System
  2. Restart your computer
  3. Install Docker Toolbox
  4. Restart your computer
  5. After it's installed, open the "Docker QuickStart Terminal" and run the following to increase its RAM:
docker-machine rm default
docker-machine create -d virtualbox --virtualbox-cpu-count=2 --virtualbox-memory=4096 --virtualbox-disk-size=50000 default
docker-machine restart default

NOTE: your code must live somewhere under your User directory (ie. C:\Users\USER_NAME\ANY_FOLDER). This is a restriction of docker toolbox not sharing paths correctly otherwise.

Common Errors

Error:

...
subprocess.CalledProcessError: Command 'docker run --rm -it -v /home/jason/tmp/.numerai:/opt/plan -w /opt/plan hashicorp/terraform:light init' returned non-zero exit status 127.

Reason: Docker is not installed.

Solutions:

  • If you're certain that docker is installed, make sure that your user can execute docker, ie. try to run docker ps. If that's the issue, then depending on your system, you can do the following:

  • Windows/OSX

    • Make sure the Docker Desktop is running and finished booting up. It can take a few minutes to be completely ready. When clicking on the docker tray icon, it should say "Docker Desktop is Running".
    • If you're using Docker Toolbox on Windows, then make sure you've opened the "Docker Quickstart Terminal".

  • Linux

    • Run sudo usermod -aG docker $USER
    • Then reboot or logout/login for this to take effect.

Error:

docker: Error response from daemon: Drive has not been shared

Solutions

numerai command not in PATH

Error:

numerai: command not found

Solutions:

  • osx/linux: Try and run ~/.local/bin/numerai
  • Windows: %LOCALAPPDATA%\Programs\Python\Python37-32\Scripts\numerai.exe
  • Alternatively, exit then re-open your terminal/command prompt.

Wrong AWS API key

Error:

...
Error: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: 32500359-7d9e-11e9-b0ed-596aba1b72c5
...
subprocess.CalledProcessError: Command 'docker run -e "AWS_ACCESS_KEY_ID=..." -e "AWS_SECRET_ACCESS_KEY=..." --rm -it -v /home/jason/tmp/.numerai:/opt/plan -w /opt/plan hashicorp/terraform:light apply -auto-approve' returned non-zero exit status 1.

Solutions

  • Run numerai configure to re-write your API Keys

Cloud Out-Of-Memory Issues

If you want to use larger instances to generate your predictions first run numerai list-constants to list the vCPU/mem presets available, then you can configure a node to use one of the presets via:

numerai node config -s mem-lg

You can also customize these values in nodes.json (see https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#task_size for valid combinations) and then run numerai node config.

CPU / Memory configuration is above your limit

Error:

ERROR numerapi.base_api: Error received from your webhook server: {"tasks":[],"failures":[{"reason":"The requested CPU configuration is above your limit"}]}

Solution:

  1. Go to the Quota Dashboard for EC2
  2. Search for "On-Demand", this will list all instance types and their limits for your account.
  3. Click the bubble next to "Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances"
  4. click "Request quota increase" in the top right
  5. Input a higher value than the currently applied quota and finish the request

NOTES

  • If after AWS increases your quota, you still get this error, try again 1-2 more times
  • You may have to complete a quota request for other types of instances too depending on how resource intensive your setup is

Billing Alerts

There's no automated way to setup billing alerts, so you'll need to configure one manually. We estimate costs to be less than $5 per month unless your compute takes more than 12 hours a week, but increasing the RAM/CPU will increase your costs.

Special Thanks:

Clone this wiki locally