From 7299895094cba9bd2170f7b52f985526255212ec Mon Sep 17 00:00:00 2001 From: David Grote Date: Fri, 13 Dec 2024 02:57:03 -0800 Subject: [PATCH] Replace quartz with dane (#5507) The Quartz system at LLNL was removed and replaced by Dane. This PR updates the scripts and install instructions for the new machine. --- Docs/source/install/hpc.rst | 2 +- .../install/hpc/{quartz.rst => dane.rst} | 78 +++++++++---------- .../quartz.sbatch => dane-llnl/dane.sbatch} | 11 ++- .../dane_warpx.profile.example} | 30 +++---- .../install_dependencies.sh | 46 +++++------ 5 files changed, 82 insertions(+), 85 deletions(-) rename Docs/source/install/hpc/{quartz.rst => dane.rst} (52%) rename Tools/machines/{quartz-llnl/quartz.sbatch => dane-llnl/dane.sbatch} (78%) rename Tools/machines/{quartz-llnl/quartz_warpx.profile.example => dane-llnl/dane_warpx.profile.example} (62%) rename Tools/machines/{quartz-llnl => dane-llnl}/install_dependencies.sh (58%) diff --git a/Docs/source/install/hpc.rst b/Docs/source/install/hpc.rst index 35884050a59..61e60359e59 100644 --- a/Docs/source/install/hpc.rst +++ b/Docs/source/install/hpc.rst @@ -50,7 +50,7 @@ This section documents quick-start guides for a selection of supercomputers that hpc/perlmutter hpc/pitzer hpc/polaris - hpc/quartz + hpc/dane hpc/summit hpc/taurus hpc/tioga diff --git a/Docs/source/install/hpc/quartz.rst b/Docs/source/install/hpc/dane.rst similarity index 52% rename from Docs/source/install/hpc/quartz.rst rename to Docs/source/install/hpc/dane.rst index a49327e8613..e9af32130f5 100644 --- a/Docs/source/install/hpc/quartz.rst +++ b/Docs/source/install/hpc/dane.rst @@ -1,9 +1,9 @@ -.. _building-quartz: +.. _building-dane: -Quartz (LLNL) +Dane (LLNL) ============= -The `Quartz Intel CPU cluster `_ is located at LLNL. +The `Dane Intel CPU cluster `_ is located at LLNL. Introduction @@ -11,9 +11,7 @@ Introduction If you are new to this system, **please see the following resources**: -* `LLNL user account `__ (login required) -* `Quartz user guide `_ -* Batch system: `Slurm `_ +* `LLNL user account `__ (`documentation `__, login required) * `Production directories `_: @@ -21,7 +19,7 @@ If you are new to this system, **please see the following resources**: * Note that the ``$HOME`` directory and the ``/usr/workspace/$(whoami)`` space are NFS mounted and *not* suitable for production quality data generation. -.. _building-quartz-preparation: +.. _building-dane-preparation: Preparation ----------- @@ -32,23 +30,23 @@ Use the following commands to download the WarpX source code: git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx -We use system software modules, add environment hints and further dependencies via the file ``$HOME/quartz_warpx.profile``. +We use system software modules, add environment hints and further dependencies via the file ``$HOME/dane_warpx.profile``. Create it now: .. code-block:: bash - cp $HOME/src/warpx/Tools/machines/quartz-llnl/quartz_warpx.profile.example $HOME/quartz_warpx.profile + cp $HOME/src/warpx/Tools/machines/dane-llnl/dane_warpx.profile.example $HOME/dane_warpx.profile .. dropdown:: Script Details :color: light :icon: info :animate: fade-in-slide-down - .. literalinclude:: ../../../../Tools/machines/quartz-llnl/quartz_warpx.profile.example + .. literalinclude:: ../../../../Tools/machines/dane-llnl/dane_warpx.profile.example :language: bash Edit the 2nd line of this script, which sets the ``export proj=""`` variable. -For example, if you are member of the project ``tps``, then run ``vi $HOME/quartz_warpx.profile``. +For example, if you are member of the project ``tps``, then run ``vi $HOME/dane_warpx.profile``. Enter the edit mode by typing ``i`` and edit line 2 to read: .. code-block:: bash @@ -59,29 +57,29 @@ Exit the ``vi`` editor with ``Esc`` and then type ``:wq`` (write & quit). .. important:: - Now, and as the first step on future logins to Quartz, activate these environment settings: + Now, and as the first step on future logins to Dane, activate these environment settings: .. code-block:: bash - source $HOME/quartz_warpx.profile + source $HOME/dane_warpx.profile -Finally, since Quartz does not yet provide software modules for some of our dependencies, install them once: +Finally, since Dane does not yet provide software modules for some of our dependencies, install them once: .. code-block:: bash - bash $HOME/src/warpx/Tools/machines/quartz-llnl/install_dependencies.sh - source /usr/workspace/${USER}/quartz/venvs/warpx-quartz/bin/activate + bash $HOME/src/warpx/Tools/machines/dane-llnl/install_dependencies.sh + source /usr/workspace/${USER}/dane/venvs/warpx-dane/bin/activate .. dropdown:: Script Details :color: light :icon: info :animate: fade-in-slide-down - .. literalinclude:: ../../../../Tools/machines/quartz-llnl/install_dependencies.sh + .. literalinclude:: ../../../../Tools/machines/dane-llnl/install_dependencies.sh :language: bash -.. _building-quartz-compilation: +.. _building-dane-compilation: Compilation ----------- @@ -91,27 +89,27 @@ Use the following :ref:`cmake commands ` to compile the applicat .. code-block:: bash cd $HOME/src/warpx - rm -rf build_quartz + rm -rf build_dane - cmake -S . -B build_quartz -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3" - cmake --build build_quartz -j 6 + cmake -S . -B build_dane -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3" + cmake --build build_dane -j 6 -The WarpX application executables are now in ``$HOME/src/warpx/build_quartz/bin/``. +The WarpX application executables are now in ``$HOME/src/warpx/build_dane/bin/``. Additionally, the following commands will install WarpX as a Python module: .. code-block:: bash - rm -rf build_quartz_py + rm -rf build_dane_py - cmake -S . -B build_quartz_py -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3" - cmake --build build_quartz_py -j 6 --target pip_install + cmake -S . -B build_dane_py -DWarpX_FFT=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3" + cmake --build build_dane_py -j 6 --target pip_install -Now, you can :ref:`submit Quartz compute jobs ` for WarpX :ref:`Python (PICMI) scripts ` (:ref:`example scripts `). -Or, you can use the WarpX executables to submit Quartz jobs (:ref:`example inputs `). -For executables, you can reference their location in your :ref:`job script ` or copy them to a location in ``$PROJWORK/$proj/``. +Now, you can :ref:`submit Dane compute jobs ` for WarpX :ref:`Python (PICMI) scripts ` (:ref:`example scripts `). +Or, you can use the WarpX executables to submit Dane jobs (:ref:`example inputs `). +For executables, you can reference their location in your :ref:`job script ` or copy them to a location in ``$PROJWORK/$proj/``. -.. _building-quartz-update: +.. _building-dane-update: Update WarpX & Dependencies --------------------------- @@ -135,34 +133,34 @@ If you already installed WarpX in the past and want to update it, start by getti And, if needed, -- :ref:`update the quartz_warpx.profile file `, +- :ref:`update the dane_warpx.profile file `, - log out and into the system, activate the now updated environment profile as usual, -- :ref:`execute the dependency install scripts `. +- :ref:`execute the dependency install scripts `. -As a last step, clean the build directory ``rm -rf $HOME/src/warpx/build_quartz`` and rebuild WarpX. +As a last step, clean the build directory ``rm -rf $HOME/src/warpx/build_dane`` and rebuild WarpX. -.. _running-cpp-quartz: +.. _running-cpp-dane: Running ------- -.. _running-cpp-quartz-CPUs: +.. _running-cpp-dane-CPUs: -Intel Xeon E5-2695 v4 CPUs +Intel Sapphire Rapids CPUs ^^^^^^^^^^^^^^^^^^^^^^^^^^ -The batch script below can be used to run a WarpX simulation on 2 nodes on the supercomputer Quartz at LLNL. +The batch script below can be used to run a WarpX simulation on 2 nodes on the supercomputer Dane at LLNL. Replace descriptions between chevrons ``<>`` by relevant values, for instance ```` could be ``plasma_mirror_inputs``. -.. literalinclude:: ../../../../Tools/machines/quartz-llnl/quartz.sbatch +.. literalinclude:: ../../../../Tools/machines/dane-llnl/dane.sbatch :language: bash - :caption: You can copy this file from ``Tools/machines/quartz-llnl/quartz.sbatch``. + :caption: You can copy this file from ``Tools/machines/dane-llnl/dane.sbatch``. -To run a simulation, copy the lines above to a file ``quartz.sbatch`` and run +To run a simulation, copy the lines above to a file ``dane.sbatch`` and run .. code-block:: bash - sbatch quartz.sbatch + sbatch dane.sbatch to submit the job. diff --git a/Tools/machines/quartz-llnl/quartz.sbatch b/Tools/machines/dane-llnl/dane.sbatch similarity index 78% rename from Tools/machines/quartz-llnl/quartz.sbatch rename to Tools/machines/dane-llnl/dane.sbatch index 4c1a82ff8e9..b2a114b3f1b 100644 --- a/Tools/machines/quartz-llnl/quartz.sbatch +++ b/Tools/machines/dane-llnl/dane.sbatch @@ -15,15 +15,14 @@ # one MPI rank per half-socket (see below) #SBATCH --tasks-per-node=2 # request all logical (virtual) cores per half-socket -#SBATCH --cpus-per-task=18 +#SBATCH --cpus-per-task=112 -# each Quartz node has 1 socket of Intel Xeon E5-2695 v4 -# each Xeon CPU is divided into 2 bus rings that each have direct L3 access +# each Dane node has 2 sockets of Intel Sapphire Rapids with 56 cores each export WARPX_NMPI_PER_NODE=2 -# each MPI rank per half-socket has 9 physical cores -# or 18 logical (virtual) cores +# each MPI rank per half-socket has 56 physical cores +# or 112 logical (virtual) cores # over-subscribing each physical core with 2x # hyperthreading led to a slight (3.5%) speedup on Cori's Intel Xeon E5-2698 v3, # so we do the same here @@ -33,7 +32,7 @@ export WARPX_NMPI_PER_NODE=2 # for N>9, also equally over close-by logical cores export OMP_PROC_BIND=spread export OMP_PLACES=threads -export OMP_NUM_THREADS=18 +export OMP_NUM_THREADS=112 EXE="" # e.g. ./warpx diff --git a/Tools/machines/quartz-llnl/quartz_warpx.profile.example b/Tools/machines/dane-llnl/dane_warpx.profile.example similarity index 62% rename from Tools/machines/quartz-llnl/quartz_warpx.profile.example rename to Tools/machines/dane-llnl/dane_warpx.profile.example index f296a0738ff..1d272979bd1 100644 --- a/Tools/machines/quartz-llnl/quartz_warpx.profile.example +++ b/Tools/machines/dane-llnl/dane_warpx.profile.example @@ -2,7 +2,7 @@ #export proj="" # edit this and comment in # required dependencies -module load cmake/3.23.1 +module load cmake/3.26.3 module load clang/14.0.6-magic module load mvapich2/2.3.7 @@ -15,38 +15,38 @@ module load boost/1.80.0 # optional: for openPMD support module load hdf5-parallel/1.14.0 -SW_DIR="/usr/workspace/${USER}/quartz" -export CMAKE_PREFIX_PATH=${SW_DIR}/c-blosc-1.21.1:$CMAKE_PREFIX_PATH +SW_DIR="/usr/workspace/${USER}/dane" +export CMAKE_PREFIX_PATH=${SW_DIR}/c-blosc-1.21.6:$CMAKE_PREFIX_PATH export CMAKE_PREFIX_PATH=${SW_DIR}/adios2-2.8.3:$CMAKE_PREFIX_PATH export PATH=${SW_DIR}/adios2-2.8.3/bin:${PATH} # optional: for PSATD in RZ geometry support -export CMAKE_PREFIX_PATH=${SW_DIR}/blaspp-2024.05.31:$CMAKE_PREFIX_PATH -export CMAKE_PREFIX_PATH=${SW_DIR}/lapackpp-2024.05.31:$CMAKE_PREFIX_PATH -export LD_LIBRARY_PATH=${SW_DIR}/blaspp-2024.05.31/lib64:$LD_LIBRARY_PATH -export LD_LIBRARY_PATH=${SW_DIR}/lapackpp-2024.05.31/lib64:$LD_LIBRARY_PATH +export CMAKE_PREFIX_PATH=${SW_DIR}/blaspp-2024.10.26:$CMAKE_PREFIX_PATH +export CMAKE_PREFIX_PATH=${SW_DIR}/lapackpp-2024.10.26:$CMAKE_PREFIX_PATH +export LD_LIBRARY_PATH=${SW_DIR}/blaspp-2024.10.26/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=${SW_DIR}/lapackpp-2024.10.26/lib64:$LD_LIBRARY_PATH # optional: for Python bindings -module load python/3.9.12 +module load python/3.12.2 -if [ -d "${SW_DIR}/venvs/warpx-quartz" ] +if [ -d "${SW_DIR}/venvs/warpx-dane" ] then - source ${SW_DIR}/venvs/warpx-quartz/bin/activate + source ${SW_DIR}/venvs/warpx-dane/bin/activate fi # optional: an alias to request an interactive node for two hours -alias getNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task=18 -p pdebug --pty bash" +alias getNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task=56 -p pdebug --pty bash" # an alias to run a command on a batch node for up to 30min # usage: runNode -alias runNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task=18 -p pdebug" +alias runNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task=56 -p pdebug" # fix system defaults: do not escape $ with a \ on tab completion shopt -s direxpand -# optimize CPU microarchitecture for Intel Xeon E5-2695 v4 +# optimize CPU microarchitecture for Intel Sapphire Rapids # note: the cc/CC/ftn wrappers below add those -export CXXFLAGS="-march=broadwell" -export CFLAGS="-march=broadwell" +export CXXFLAGS="-march=sapphirerapids" +export CFLAGS="-march=sapphirerapids" # compiler environment hints export CC=$(which clang) diff --git a/Tools/machines/quartz-llnl/install_dependencies.sh b/Tools/machines/dane-llnl/install_dependencies.sh similarity index 58% rename from Tools/machines/quartz-llnl/install_dependencies.sh rename to Tools/machines/dane-llnl/install_dependencies.sh index cfb01769384..0415d7fa8cc 100755 --- a/Tools/machines/quartz-llnl/install_dependencies.sh +++ b/Tools/machines/dane-llnl/install_dependencies.sh @@ -1,10 +1,10 @@ #!/bin/bash # -# Copyright 2023 The WarpX Community +# Copyright 2024 The WarpX Community # # This file is part of WarpX. # -# Author: Axel Huebl +# Author: Axel Huebl, David Grote # License: BSD-3-Clause-LBNL # Exit on first error encountered ############################################# @@ -14,13 +14,13 @@ set -eu -o pipefail # Check: ###################################################################### # -# Was quartz_warpx.profile sourced and configured correctly? -if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your quartz_warpx.profile file! Please edit its line 2 to continue!"; exit 1; fi +# Was dane_warpx.profile sourced and configured correctly? +if [ -z ${proj-} ]; then echo "WARNING: The 'proj' variable is not yet set in your dane_warpx.profile file! Please edit its line 2 to continue!"; exit 1; fi # Remove old dependencies ##################################################### # -SW_DIR="/usr/workspace/${USER}/quartz" +SW_DIR="/usr/workspace/${USER}/dane" rm -rf ${SW_DIR} mkdir -p ${SW_DIR} @@ -41,13 +41,13 @@ if [ -d ${HOME}/src/c-blosc ] then cd ${HOME}/src/c-blosc git fetch --prune - git checkout v1.21.1 + git checkout v1.21.6 cd - else - git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git ${HOME}/src/c-blosc + git clone -b v1.21.6 https://github.com/Blosc/c-blosc.git ${HOME}/src/c-blosc fi -cmake -S ${HOME}/src/c-blosc -B ${build_dir}/c-blosc-quartz-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.1 -cmake --build ${build_dir}/c-blosc-quartz-build --target install --parallel 6 +cmake -S ${HOME}/src/c-blosc -B ${build_dir}/c-blosc-dane-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/c-blosc-1.21.6 +cmake --build ${build_dir}/c-blosc-dane-build --target install --parallel 6 # ADIOS2 if [ -d ${HOME}/src/adios2 ] @@ -59,44 +59,44 @@ then else git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git ${HOME}/src/adios2 fi -cmake -S ${HOME}/src/adios2 -B ${build_dir}/adios2-quartz-build -DBUILD_TESTING=OFF -DADIOS2_BUILD_EXAMPLES=OFF -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_SST=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3 -cmake --build ${build_dir}/adios2-quartz-build --target install -j 6 +cmake -S ${HOME}/src/adios2 -B ${build_dir}/adios2-dane-build -DBUILD_TESTING=OFF -DADIOS2_BUILD_EXAMPLES=OFF -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_SST=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${SW_DIR}/adios2-2.8.3 +cmake --build ${build_dir}/adios2-dane-build --target install -j 6 # BLAS++ (for PSATD+RZ) if [ -d ${HOME}/src/blaspp ] then cd ${HOME}/src/blaspp git fetch --prune - git checkout v2024.05.31 + git checkout v2024.10.26 cd - else - git clone -b v2024.05.31 https://github.com/icl-utk-edu/blaspp.git ${HOME}/src/blaspp + git clone -b v2024.10.26 https://github.com/icl-utk-edu/blaspp.git ${HOME}/src/blaspp fi -cmake -S ${HOME}/src/blaspp -B ${build_dir}/blaspp-quartz-build -Duse_openmp=ON -Duse_cmake_find_blas=ON -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-2024.05.31 -cmake --build ${build_dir}/blaspp-quartz-build --target install --parallel 6 +cmake -S ${HOME}/src/blaspp -B ${build_dir}/blaspp-dane-build -Duse_openmp=ON -Duse_cmake_find_blas=ON -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=${SW_DIR}/blaspp-2024.10.26 +cmake --build ${build_dir}/blaspp-dane-build --target install --parallel 6 # LAPACK++ (for PSATD+RZ) if [ -d ${HOME}/src/lapackpp ] then cd ${HOME}/src/lapackpp git fetch --prune - git checkout v2024.05.31 + git checkout v2024.10.26 cd - else - git clone -b v2024.05.31 https://github.com/icl-utk-edu/lapackpp.git ${HOME}/src/lapackpp + git clone -b v2024.10.26 https://github.com/icl-utk-edu/lapackpp.git ${HOME}/src/lapackpp fi -CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S ${HOME}/src/lapackpp -B ${build_dir}/lapackpp-quartz-build -Duse_cmake_find_lapack=ON -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-2024.05.31 -cmake --build ${build_dir}/lapackpp-quartz-build --target install --parallel 6 +CXXFLAGS="-DLAPACK_FORTRAN_ADD_" cmake -S ${HOME}/src/lapackpp -B ${build_dir}/lapackpp-dane-build -Duse_cmake_find_lapack=ON -DCMAKE_CXX_STANDARD=17 -Dbuild_tests=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_INSTALL_PREFIX=${SW_DIR}/lapackpp-2024.10.26 +cmake --build ${build_dir}/lapackpp-dane-build --target install --parallel 6 # Python ###################################################################### # python3 -m pip install --upgrade --user virtualenv -rm -rf ${SW_DIR}/venvs/warpx-quartz -python3 -m venv ${SW_DIR}/venvs/warpx-quartz -source ${SW_DIR}/venvs/warpx-quartz/bin/activate +rm -rf ${SW_DIR}/venvs/warpx-dane +python3 -m venv ${SW_DIR}/venvs/warpx-dane +source ${SW_DIR}/venvs/warpx-dane/bin/activate python3 -m pip install --upgrade pip -python3 -m pip cache purge +#python3 -m pip cache purge python3 -m pip install --upgrade build python3 -m pip install --upgrade packaging python3 -m pip install --upgrade wheel