parallel error in HPC #28715
-
Check these boxes if you have followed the posting rules.
QuestionDear MOOSE team, I have installed moose in the hpc at Singapore (NSCC), using the conda framework. However, when I submit a parallel case, actually it doesn't work. My qsub file is as follow. Could you give me some suggestions to address this issue? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 14 replies
-
|
Hello First try this to make sure 32 processes are launched. you can use when you compile and when you run to compare |
Beta Was this translation helpful? Give feedback.
-
|
Hi GiudGiud, if I run which mpicc and which mpirun, it shows that Kindly note that the parallel file I used is The issue still exist. |
Beta Was this translation helpful? Give feedback.
-
|
Hello There are a few issues here Why are you using conda compilers on a computing cluster? This won't scale well. Then after compiling with the MPI from conda, you are likely using MPI from a module to run. which mpirun should be run inside a job to see what is being used then |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the reply. I use conda to install MOOSE in HPC because the module ava in our HPC cannot support the normal installation. I have changed the job file: and the output is seems that the '/opt/cray/pe/pals/1.1.6/bin/mpirun' differs with '~/.conda/envs/moose/bin/mpicc ~/.conda/envs/moose/bin/mpirun'. Is there any suggestions? |
Beta Was this translation helpful? Give feedback.
-
|
Can you try to activate the environment using an alternative method: #!/bin/bash
#PBS -q normal
#PBS -j oe
#PBS -N p12_nognd
#PBS -l select=1:ncpus=32:mpiprocs=32:mem=100GB
#PBS -l walltime=24:00:00
# alternative way to 'activate' a conda environment
# see tip below why I don't use ~/
export PATH=/the/full/path/to/conda/bin:$PATH
source activate /the/full/path/to/.conda/envs/moose
which mpirun
srun -n 32 /scratch/users/nus/guozixu/scut/scut-opt \
-i /scratch/users/nus/guozixu/scut/run.i \
> /scratch/users/nus/guozixu/scut/p12_nognd.txtTip use I think And for completeness, here would be the SLURM equivalent to your above PBS submission script: #SBATCH --time=24:00:00 # walltime
#SBATCH --ntasks-per-node=32 # number of processor cores (i.e. tasks)
#SBATCH --nodes=1 # number of nodes
#SBATCH -J "p12_nognd" # job name
#SBATCH -p normal # queue |
Beta Was this translation helpful? Give feedback.
-
|
It works! Special thanks for your helps! |
Beta Was this translation helpful? Give feedback.
-
|
Welcome! We activate our Conda environments in the same manner on our buildfarm (https://civet.inl.gov). I find it's a cleaner, less intrusive way that does not modify nor rely on, your |
Beta Was this translation helpful? Give feedback.
-
|
Thanks~ |
Beta Was this translation helpful? Give feedback.
-
|
There is a performance issue. The inter-node communication will not use any optimized communication network your HPC cluster team deployed (infiniband for example) On a single node, you should be fine. It will still be slower because the architecture-specific optimizations are not turned on |
Beta Was this translation helpful? Give feedback.
Can you try to activate the environment using an alternative method:
Tip
use
/full/path/to/**paths instead of~/.I've seen shorthand path to home, fail in job submission scripts. Best to use fully declared paths in…