Singularity container with OpenMPI and InfiniBand (UCX)

[VincentDonney]

Hello, I'm working as an intern currently, and I was asked to build a Singularity container for OpenMPI to make distributed programming possible on multiple machines of our HPC cluster using containers.

Basically, I'm wondering what is the correct approach for something like this : do I want my container to take OpenMPI, UCX and Mellanox OFED (that we use together for InfiniBand) from outside, using something like environment-modules, or do I want to compile OFED, UCX and OpenMPI inside my container ? We could also consider hybrid solutions, like compiling OpenMPI and UCX inside the container but taking our OFED libraries from outside the container.

I would like to know if anyone has worked on a similar case before and can help me. What is the best practice ? What limits will I face with the different solutions I spoke of ?

So far I've already built a container with OFED, UCX and OpenMPI compiled inside, one with nothing inside but the small setup for environment-modules, and one sort of hybrid, where I compile OpenMPI and UCX inside the container, and install libraries using dnf, and take the little bits of OFED that are still missing to run InfiniBand from outside.
The thing is that I've built my container with a RHEL8.2 OS, which is the same as my host machine, and I don't know what issues I could face if I were to build a container with a RHEL7 or RHEL9 OS for example.

Now to provide more details about our stack, we use LSF as our job scheduler, though I haven't worked with it yet. Our machines are mostly RHEL8.2 and 8.6. Some older machines have mlx4_0 InfiniBand devices, while the newer have mlx5_0.
I have installed OpenMPI-5.0.3 and Singularity-ce-4.1.2 on my host machine, along with UCX-1.16.0 so that OpenMPI can use our InfiniBand network.

Inside my container, I compile MLNX_OFED_LINUX-4.9-7.1.0.0-rhel8.2-x86_64, UCX-1.16.0 and OpenMPI-5.0.3 from tarballs.
I've created a module for Openmpi-5.0.3 and one for UCX-1.16.0 that my container uses with environment-modules.

Here is my definition file for a hybrid container where we use Mellanox OFED and UCX from the host machine, and compile OpenMPI while referencing that UCX. (I mount the directory where my UCX installation is located). At first I tried to compile both UCX and OpenMPI from inside the container, but it was too complicated to bind every folder of Mellanox OFED, since its files are spread in multiple locations.

Any help is appreciated.

#compiled with :
#singularity build -f -F -B /install/ucx-1.16.0:/install/ucx-1.16.0 mpi_ucx_hybrid.sif mpi_ucx_hybrid.def
#run command :
#mpirun -np 2 --mca plm ^lsf --mca ras ^lsf -x HCOLL_RCACHE=^ucs --mca ompi_display_comm 1 --hostfile hosts.txt singularity exec -B /install/ucx-1.16.0:/install/ucx-1.16.0 mpi_ucx_hybrid.sif /scripts/benchmark

BootStrap: localimage
From: /images/rhel82-base.sif

%files
    /packages/tarball/openmpi-5.0.3.tar.gz /packages/openmpi-5.0.3.tar.gz
    /scripts/pi_trapezoid.c  /scripts/pi_trapezoid.c
    /scripts/benchmark.c /scripts/benchmark.c


%environment
    export OMPI_DIR=/install/openmpi-5.0.3
    export PATH=$OMPI_DIR/bin:${PATH}
    export LD_LIBRARY_PATH=$OMPI_DIR/lib:${LD_LIBRARY_PATH}
    export UCX_DIR=/install/ucx-1.16.0
    export PATH=$UCX_DIR/bin:$PATH
    export LD_LIBRARY_PATH=$UCX_DIR/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=$UCX_DIR/lib/ucx:$LD_LIBRARY_PATH
%setup
    mkdir -p ${SINGULARITY_ROOTFS}/install/ucx-1.16.0
%post

    export LDFLAGS=/lib64/libnsl.so.2

    export UCX_DIR=/install/ucx-1.16.0
    export PATH=$UCX_DIR/bin:$PATH
    export LD_LIBRARY_PATH=$UCX_DIR/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=$UCX_DIR/lib/ucx:$LD_LIBRARY_PATH

    dnf makecache
    dnf install -y -q gcc gcc-gfortran gcc-c++ make file python3 lzo-devel systemd-devel perl-Getopt-Long
    #extracting
    cd /packages

    tar -xzf openmpi-5.0.3.tar.gz

    mkdir /install

    #installing OpenMPI
    cd openmpi-5.0.3
    ./configure --prefix=/install/openmpi-5.0.3 CC=gcc CXX=g++ FC=gfortran --enable-mpirun-prefix-by-default --with-ucx=$UCX_DIR --enable-mca-dso --without-verbs
    make all
    make all install

    # Set env variables so we can find OpenMPI and compile
    export OMPI=/install/openmpi-5.0.3
    export PATH=$OMPI/bin:${PATH}
    export LD_LIBRARY_PATH=$OMPI/lib:${LD_LIBRARY_PATH}
    export MANPATH=$OMPI/share/man:${MANPATH}

    echo "Compiling the MPI application..."
    cd /scripts/ && mpicc -o pi_trapezoid pi_trapezoid.c -lm && mpicc -o benchmark benchmark.c -lm

[VincentDonney]

Update : I have been working on different things and tried to create a "hybrid" container where I try to bind different folders of my machine, like the libraries, to the container, and then add them to $LD_LIBRARY_PATH or $PATH, but I just get an error on the glibc library (it seems that the container is trying to use the version of glibc of the bind dir and not its own).

[dtrudg]

If you give a bit more detail about exactly how you are doing the binds, how you are building the containers then that would be a good starting point for someone to provide advice. It's always good to share the actual definition file you are using for the build, and the exact commands you are using to run it.

There are a lot of different things that people do even within the realms of 'hybrid' containers, so seeing exactly what is being run is useful.

[VincentDonney]

I've updated my post with my definition file for my hybrid container. Basically I don't have any issues building the image, but when I want to use mpirun it uses tcp instead of ucx.