GPU TensorFlow

Nvidia docker 2 installation guide

check the nvidia-container-toolkit is installed correctly.

sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

install image

docker pull tensorflow/tensorflow:latest-gpu-jupyter

list image

docker image ls -a

run image with jupyter notebook

docker run -it -p 8888:8888 tensorflow/tensorflow:latest-gpu-jupyter

See containers status

sudo docker container ls

Restart docker container

docker restart container

Mount in current folder

docker run --rm -it -p 8888:8888 -v $(pwd):/usr/src/project tensorflow/tensorflow:latest-gpu-jupyter

Troubleshooting

Tensor flow doesnt detect my gpu problem

Docker GPU tensorflow

Cleaning up the command

-it interactive console

--rm remove the file system where the container exists

Entorno que usa gpus

docker run -it --rm -u $(id -u):$(id -g) --name tensor_gpu --gpus all -p 8888:8888 tensorflow/tensorflow:latest-gpu-jupyter jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root --NotebookApp.allow_origin='https://colab.research.google.com'

conectarme por consola al entorno

docker exec -it tensor_gpu bash

tensorflow message

________                               _______________                
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ / 
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/


WARNING: You are running this container as root, which can cause new files in
mounted volumes to be created as the root user on your host machine.

To avoid this, run the container by specifying your user's userid:

$ docker run -u $(id -u):$(id -g) args...

Container main folder /tf

local path /home/luis/Documents/personal_projects/tensor_flow_gpu/docker_mount

Comando final con volumenes

docker run -it --rm -u $(id -u):$(id -g) --name tensor_gpu --gpus all -p 8888:8888 -v /home/luis/Documents/personal_projects/tensor_flow_gpu/docker_mount:/tf/notebooks tensorflow/tensorflow:latest-gpu-jupyter jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root --NotebookApp.allow_origin='https://colab.research.google.com'

sin rm

docker run -it -u $(id -u):$(id -g) --name tensor_gpu --gpus all -p 8888:8888 -v /home/luis/Documents/personal_projects/tensor_flow_gpu/docker_mount:/tf/notebooks tensorflow/tensorflow:latest-gpu-jupyter jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root --NotebookApp.allow_origin='https://colab.research.google.com'

running as root for vscode attach

docker run -it --rm --name tensor_gpu --gpus all -p 8888:8888 -v /home/luis/Documents/personal_projects/tensor_flow_gpu/docker_mount:/tf/notebooks tensorflow/tensorflow:latest-gpu-jupyter jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root --NotebookApp.allow_origin='https://colab.research.google.com'

conect bash to container

docker exec -it tensor_gpu bash

Jupyter notebook command

jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root --NotebookApp.allow_origin='https://colab.research.google.com'

Reusable container

docker run -it --name tensor_gpu --gpus all -p 8888:8888 -v /home/luis/Documents/personal_projects/tensor_flow_gpu/docker_mount:/tf/notebooks tensorflow/tensorflow:latest-gpu-jupyter bash

Workflow

Creating container

docker run -it --name tensor_gpu --gpus all -p 8888:8888 -v /home/luis/Documents/personal_projects/tensor_flow_gpu/docker_mount:/tf/notebooks tensorflow/tensorflow:latest-gpu-jupyter bash

Running Jupypter in the container

jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root --NotebookApp.allow_origin='https://colab.research.google.com'

Stopping container

docker stop tensor_gpu

Removing container

docker rm tensor_gpu

Restarting container in bash

docker restart tensor_gpu

Connecting to restarted container

docker exec -it tensor_gpu bash

Docker Compose

Helper that turns docker command into docker file

Gpu support with docker compose

Checking docker images

docker-compose ps

Building docker image

docker build --rm --tag tensor_gpu_image:dev .

Remove dangling images after build

docker image prune

Do NOT run unless SURE!!!

$docker image prune -a # this will remove even base images used as reference to local images

Running docker compose

Creating the service

docker-compose up

For user ids

To run as a normal user not root (some libraries might have trouble installing)

UID=$(id -u) GID=$(id -g) docker-compose up

Compose use

Starting the service

docker-compose start tensorflow

Stoping the service

docker-compose stop tensorflow

restarting the service to keep environment changes

docker-compose restart tensorflow

Connecting to the service in any scenario

docker exec -it tensor_gpu bash

Running to turn up jupyter

jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root --NotebookApp.allow_origin='https://colab.research.google.com'

Maintenance

Check actual space of docker images in local

docker system df