library path not work in notebook

[ZlodeiBaal]

Probably it's my fault and not a bug, but i can't find reason of it.
I install fresh version of Ubunta to work with Caffe framework. Install Caffe, it worked fine. Try to open my project which I used few days ago on old version of Ubunta in notebook (both Ubunta 14.04).
But when I try:
import caffe
It give me: "ImportError: libcudart.so.7.5: cannot open shared object file: No such file or directory".
Same code work well in python and in ipython when i start them in same terminal.
I have:
os.system('export LD_LIBRARY_PATH=/usr/local/cuda-7.5/lib64:$LD_LIBRARY_PATH')
in code and few day ago it's work fine.
I have tryed few different way to load LD_LIBRARY_PATH, but none of them work. I think the problem in notebook loadin, but can't find where.

Simple test that i done:

os.environ['LD_LIBRARY_PATH'] = '/usr/local/cuda-7.5/lib64:$LD_LIBRARY_PATH'
os.system('python Test2.py')

Where Test2.py:

import sys 
caffe_root = '../'  # this file is expected to be in {caffe_root}/examples
sys.path.append(caffe_root + 'python')
import caffe 

Work fine in notebook.
But this not working:

os.environ['LD_LIBRARY_PATH'] = '/usr/local/cuda-7.5/lib64:$LD_LIBRARY_PATH'
import sys 
caffe_root = '../'  # this file is expected to be in {caffe_root}/examples
sys.path.append(caffe_root + 'python')
import caffe 

[ZlodeiBaal]

Ok, i found solution, but still think that it's some problem with notebook: it's not take system variables and not using the ones that i load in code.
Solution:
Generate config file:
jupyter notebook --generate-config
It will be stored i:
jupyter --config-dir
Add this code at start:

import os
c = get_config()
os.environ['LD_LIBRARY_PATH'] = '/usr/local/cuda-7.5/lib64:usr/local/cuda-7.5/lib64/libcudart.so.7.5'
c.Spawner.env.update('LD_LIBRARY_PATH')

And everything work

[takluyver]

This line isn't going to do what you think:

os.system('export LD_LIBRARY_PATH=/usr/local/cuda-7.5/lib64:$LD_LIBRARY_PATH')

os.system() starts a new process to run the system command. You're changing the environment variable in the new process, and then it's immediately getting closed. Child processes can't affect environment variables in their parent. To change an environment variable inside a Python process, you have to use os.environ, as you found.

I don't think the Spawner config is relevant to you - spawners are part of Jupyterhub, an extra layer on top of the notebook to support multiple users. If you're using regular jupyter notebook, that line is probably having no effect.

[yuyouling]

The solution is very simple.

1. use jupyterhub --generate-config to generate the jupyterhub_config.py
2. use text editor such as nano or vim to open the jupyterhub_config.py
3. change some configs
   (1) general setting
   c.LocalAuthenticator.create_system_users = True
   c.PAMAuthenticator.encoding = 'utf8'
   c.PAMAuthenticator.open_sessions = False
   c.JupyterHub.admin_access = True
   c.JupyterHub.ip = '0.0.0.0'
   c.JupyterHub.port = 8888
   c.Spawner.args = ['--allow-root']
   c.Spawner.default_url = '/lab'
   c.Spawner.notebook_dir = '~'
   c.Authenticator.admin_users = set(['root', 'alex'])
   (2) LD_LIBRARY_PATH related setting
   c.Spawner.env_keep = ['LD_LIBRARY_PATH'] #use for keep the environment variable - LD_LIBRARY_PATH
   c.Spawner.cmd = ['/opt/anaconda3/envs/pytorch2/bin/jupyterhub-singleuser'] # !!! very important, you should use the full path of the command jupyterhub-singleuser, due to the Spawner process the activate the jupyterhub-singleuser the load the environment (I think it does as this way)
   (3) save this file
4. run jupyterhub -f jupyterhub_config.py to start jupyterhub.
5. open the jupyterlab website and create the notebook file, try the following scripts, you can see the expected result.
   import os
   os.environ['LD_LIBRARY_PATH']

[edmorley]

This was caused by:
jupyterhub/jupyterhub#4903

And will be fixed by:
jupyterhub/jupyterhub#4904

[minrk]

Thanks @edmorley!