Shear Test fails due to OSError: libgfortran.so.3: cannot open shared object file: No such file or directory

[evevkovacs]

@yymao Just tried running the shear test and it failed with the following error:
File "/global/u1/k/kovacs/descqa2-local/descqa_njobs/descqa/shear_test.py", line 10, in
import camb
File "/global/common/software/lsst/common/miniconda/current/envs/stack/lib/python3.7/site-packages/camb/init.py", line 11, in
from . import baseconfig
File "/global/common/software/lsst/common/miniconda/current/envs/stack/lib/python3.7/site-packages/camb/baseconfig.py", line 51, in
camblib = ctypes.LibraryLoader(ifort_gfortran_loader).LoadLibrary(CAMBL)
File "/global/common/software/lsst/common/miniconda/current/envs/stack/lib/python3.7/ctypes/init.py", line 434, in LoadLibrary
return self._dlltype(name)
File "/global/common/software/lsst/common/miniconda/current/envs/stack/lib/python3.7/ctypes/init.py", line 356, in init
self._handle = _dlopen(self._name, mode)
OSError: libgfortran.so.3: cannot open shared object file: No such file or directory

I tested this myself manually. If I open an interactive python session and do:
import camb
I get OSError: libgfortran.so.3: cannot open shared object file: No such file or directory

Should I report this to Heather or open a ticket with nersc consulting?
FYI: My python setup is
source /global/common/software/lsst/common/miniconda/setup_current_python.sh
Same error happens if I open a notebook using the desc-python kernel

[yymao]

Let's check with @heather999 first.

[heather999]

I can confirm I saw the same issue. Think this is now fixed. We had lost gcc/6.3.0 with the last Cori OS upgrade and I tried to move us to gcc/7.3.0, but clearly such a move would require a complete reinstall of desc-python. Fortunately, NERSC has reinstalled gcc/6.3.0 for us, and I updated the setup_current_python.sh script to go back to gcc/6.3.0. Things should be working now. Just let me know if you see any other problems.

[yymao]

Thanks for the quick response @heather999! @evevkovacs can you report back if the code works (or not)? Thanks!

[evevkovacs]

@yymao Hmm, still seems to be broken. I also tried interactively and could not import camb after setting up python using setup_current_python.sh
I also tried a jupyter notebook using the desc-python kernel. That worked.

So descqa and interactive python are still broken, I suspect because setup_current_python.sh needs to be changed.

[heather999]

desc-python and setup_current_python.sh should be the same environment. I can do the following at a fresh Cori login - this didn't work earlier today before I updated setup_current_python.sh (updated to remove my local DESC_PYTHON_PATH):

heatherk@cori09:~> bash
heatherk@cori09:~> source /global/common/software/lsst/common/miniconda/setup_current_python.sh 
Now using /global/common/software/lsst/common/miniconda/current
(stack) heatherk@cori09:~> which python
/global/common/software/lsst/common/miniconda/py3-4.5.12/envs/stack/bin/python
(stack) heatherk@cori09:~> python
Python 3.7.2 (default, Dec 29 2018, 06:19:36) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import camb
>>> 

[evevkovacs]

I tried again on a fresh login. I normally use tcsh, so have to switch shells to run python. If I switch using exec sh, run the setup, run ipython and try to import camb, I get the error. (Previously, this procedure worked). If I switch to bash as you did above, then it works.
I also tested with DESCQA. The shear test used to run even if I was in tcsh, with no python setup. This is no longer the case. I now have to run desqa by switching to bash, do the python setup, and then run the test. So the environment is not the same as it was. The user now has to be in bash and have setup python. It would be nice if everything worked regardless of the user's shell, but I can live with it the way it is now. Thanks

[yymao]

hmm, @evevkovacs are you using run_master.sh? if so I thought it would be using bash (see https://github.com/LSSTDESC/descqa/blob/master/run_master.sh#L1)