Update, Handout

Dockerfile

@@ -36,12 +36,15 @@ RUN apt-get update && apt-get install -y \
 RUN R -e "install.packages(c('crayon', 'pbdZMQ', 'devtools', 'IRdisplay'), repos='http://cran.us.r-project.org')"
 RUN R -e "devtools::install_github(paste0('IRkernel/', c('repr', 'IRdisplay', 'IRkernel')))"
 
+RUN pip install --upgrade pip
+
 # Install virtualenv to let users install more libs
 RUN pip install -U virtualenv
 
+RUN pip install -U cython
+
 # Install KERAS + SCIKIT + Data Science Libs
 RUN pip install -U \
-		cython \
 		scipy \
 		numpy \
 		keras \
@@ -59,7 +62,10 @@ RUN pip install -U \
 		catboost \
 		opencv-python \
 		tqdm \
-		tslearn
+		tslearn \
+        bert-serving-server \
+        bert-serving-client \
+        handout
 
 # Install OpenCV + HDF5
 RUN apt-get update && apt-get install -y \
@@ -80,8 +86,7 @@ RUN apt-get update && apt-get install -y \
 	rm -rf /var/lib/apt/lists/*
 
 # Install PYTORCH
-RUN pip install https://download.pytorch.org/whl/cu100/torch-1.0.1.post2-cp35-cp35m-linux_x86_64.whl && \
-    pip install torchvision
+RUN pip install torch===1.3.0 torchvision===0.4.1 -f https://download.pytorch.org/whl/torch_stable.html
 
 # TensorBoard
 EXPOSE 6006
@@ -123,9 +128,33 @@ RUN jupyter contrib nbextension install --user && \
 
 USER $NB_USER
 
+# add alias to include system site packages into virtualenvs
+RUN echo "alias virtualenv='virtualenv --system-site-packages'" >> ~/.bashrc
+
 # Install and enable nbextensions
 RUN jupyter contrib nbextension install --user && \
     jupyter nbextensions_configurator enable --user && \
     jupyter tensorboard enable --user && \
     jupyter nbextension enable --py nbzip && \
     jupyter nbextension enable --py rise
+
+
+# Install anaconda
+
+## TODO fix this
+
+#USER root
+
+#RUN curl -O https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh
+#RUN bash Anaconda3-2019.03-Linux-x86_64.sh -b -p "/home/$NB_USER/anaconda3"
+
+#RUN chown -R $NB_USER:dbvis /home/$NB_USER/anaconda3
+
+#USER $NB_USER
+
+#RUN echo "export PATH=~/anaconda3/bin:$PATH" >> ~/.bashrc
+#RUN /bin/bash -c "source ~/.bashrc"
+
+#RUN export PATH=~/anaconda3/bin:$PATH
+
+#RUN /bin/bash -c "conda install -c anaconda cython"

README.md

@@ -1,6 +1,6 @@
 # Start
 
-Hi User,
+Dear user,
 
 this ReadMe should prepare you to work with the DBVIS GPU Server.  
 It is conceived as a guide to using every possibility of the server.
@@ -13,55 +13,90 @@ There are two major options to use the server:
 
 ## Jupyter Notebooks
 
-Jupyter Notebooks are built as a running Python or R terminal emulation.
-It enables an easier access to build small programs faster and more efficient.
+Jupyter Notebooks are built as a Python or R terminal emulation.
+It enables easier access to build small programs faster and more efficient.
 You can create a new one by pressing the New button and select a new Python or R kernel.
-It is advised to use them as you can test your Python or R code more easily with them and if your development stage is over you can export them as Python or R scripts.
+It is advised to use them as you can test your Python or R code more easily with them and if your development stage is over, you can export them as Python or R scripts.
 
 ## Terminal Access
 
 Another option is the built-in terminal and editor of Jupyterhub.
-You can open nearly every text file with the editor provided by Jupyterhub. This works like a normal desktop environment by either clicking on the file or by creating a new one.
+You can open nearly every text file with the editor provided by Jupyterhub. This works like a typical desktop environment by either clicking on the file or by creating a new one.
 Opening a terminal works with New button and then the terminal button.
 Afterward, you can navigate to your script and run it with the command line.
+Default the terminal is started with shell, but with `bash` you can change that.
 
 # Advises
 
-## Tensorflow 2.0.0 alpha
+## Virtual Environment
 
-If you are using Tensorflow or a framework, which runs Tensorflow in the background, it is advised to use:
-```
-import tensorflow as tf
-from tensorflow.keras import backend as K
+The server is equipped with the many python frameworks and libraries. In case you need additional libraries you can create a virtual environment to add them and start Jupyter notebook via the new virtual environment. Use the following snippet to create a new virtual environment:
+
+1. Open a terminal:  
+  ```
+  ## CREATE A NEW VIRTUAL ENVIRONMENT ##
+  bash         # start a bash
+  cd           # navigate to the home directory
+  name='venv'  # specify the name of your virtual environment
+  virtualenv --system-site-packages $name  # create the virtual environment + add all already installed packages
+  source $name/bin/activate  # activate the virtual environment
+  pip install --upgrade pip  # upgrade pip
+  ipython kernel install --user --name=$name   # install in user space
+  sed -i -e 's|/usr/bin/python3|'${HOME}'/'$name'/bin/python|g' $HOME'/.local/share/jupyter/kernels/'$name'/kernel.json' # replace python binary of the new kernel with the kernel of the new virtual environment
+  ```
+2. Via the following snippets you can add new packages to the virtual environment:  
+  ```
+  ## ADD PACKAGES TO A VIRTUAL ENVIRONMENT ##
+  bash
+  cd
+  name='venv'
+  source $name/bin/activate
+  pip install <package>
+  ```
+
+3. Now you can create new Jupyter notebooks using the virtual environment from the *New* tab in the jupyter home directory.
+
+## GPU Status
+
+You can see the available GPUs as well as the available VRAM by typing `nvidia-smi` in a terminal.
+
+## GPU Usage
+
+Please make sure to just use **only one** GPU at the same time. After checking for an available GPU, you can specify the GPU inside your Jupyter notebooks using the following snippet:
 
-config = tf.compat.v1.ConfigProto()
-config.gpu_options.allow_growth = True
-K.set_session(tf.compat.v1.Session(config=config))
 ```
-To limit the amount of GPU memory Tensorflow uses.
-Else Tensorflow will use all available GPU memory, which blocks other users and leads to your process being killed.
+import os
+os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"   # order GPUs by a steady ordering (like nvidia-smi)
+os.environ["CUDA_VISIBLE_DEVICES"]="0"         # the ID of the GPU (view via nvidia-smi)
+```
+
+**After finishing using the GPUs, please explicitly stop your Juypter notebooks, since otherwise the VRAM occupied might not be freed.**
 
 ## Tensorflow
 
-If you are using Tensorflow or a framework, which runs Tensorflow in the background, it is advised to use:
+If you are using Tensorflow or a framework (Keras), which runs Tensorflow in the background, it is advised to use:
 ```
 import tensorflow as tf
 config = tf.ConfigProto()
 config.gpu_options.allow_growth = True
 session = tf.Session(config=config, ...)
 ```
-or
+or Keras explicitly
 ```
 from keras import backend as K
 config = K.tf.ConfigProto()
 config.gpu_options.allow_growth = True
 K.set_session(K.tf.Session(config=config))
 ```
+
 To limit the amount of GPU memory Tensorflow uses.
 Else Tensorflow will use all available GPU memory, which blocks other users and leads to your process being killed.
 
 
-# Don'ts
+# Dont's
 
-- Do not use more space than you need. This means if your data is too large, use the database server to store it. There are enough options to use Python and a database server efficiently.
-- Do not block all of the available space of the different GPUs. Either block one GPU completely or try to use the allow growth option of tensorflow.
+- Do not use more space than you need. 
+This means if your data is too large, use the database server to store it. 
+There are enough options to use Python and a database server efficiently.
+- Do not block all of the available space of the different GPUs. 
+Either block one GPU entirely or try to use the allow growth option of tensorflow.