From 3045c8fdca377f72d9dc0a18a4c011e440f6ca8e Mon Sep 17 00:00:00 2001 From: Alexander Smith Date: Fri, 14 Jun 2024 13:43:02 -0400 Subject: [PATCH 1/7] Add new updates to the ALCF workflow --- sphinx_docs/source/alcf.rst | 51 ++++++++++++++++++++++++++++++++++--- 1 file changed, 47 insertions(+), 4 deletions(-) diff --git a/sphinx_docs/source/alcf.rst b/sphinx_docs/source/alcf.rst index ff2d07a..e356fa9 100644 --- a/sphinx_docs/source/alcf.rst +++ b/sphinx_docs/source/alcf.rst @@ -27,16 +27,15 @@ Load the modules: .. prompt:: bash - module swap PrgEnv-nvhpc PrgEnv-gnu - # load gcc/11.2.0 version since CUDA doesn't support gcc 12 yet - module load gcc/11.2.0 + module use /soft/modulefiles + module load PrgEnv-gnu module load nvhpc-mixed Then you can compile via: .. prompt:: bash - make COMP=gnu USE_CUDA=TRUE + make COMP=cray USE_CUDA=TRUE Disks @@ -52,6 +51,10 @@ https://www.alcf.anl.gov/support/user-guides/polaris/queueing-and-running-jobs/j For production jobs, you submit to the ``prod`` queue. +For debugging jobs, there are two options: the ``debug`` queue for only one node use jobs, and +``debug-scaling`` for jobs that require between two nodes and 10-nodes. The former is used mainly for +tunning the optimal parameters and compare the scalability of the allocated resources. + .. note:: The smallest node count that seems to be allowed in production is 10 nodes. @@ -108,3 +111,43 @@ A script that can be used to chain jobs with PBS is: .. literalinclude:: ../../job_scripts/polaris/chainqsub.sh :caption: ``chainqsub.sh`` + +Installing Python +================= + +The most recommended way to install python is to create a virtual environment +on the top of the conda-based environment provided by the module conda, and install all the extra +required modules on this virtual environment. Although is very tempting +to clone the whole base environment and fully customize the installed conda packages, some +modules like ``mpi4py`` may require access to the MPICH libraries that are tailored to be +used only within the conda-base environment provided by the conda module. + +To create the virtual environment: + +.. prompt:: bash + + module use /soft/modulefiles + module load conda + conda activate + VENV_DIR="venvs/polaris" + mkdir -p "${VENV_DIR}" + python -m venv "${VENV_DIR}" --system-site-packages + source "${VENV_DIR}/bin/activate" + +To activate it in a new terminal, if the module path ``/soft/modules`` +is loaded: + +.. prompt:: bash + + module load conda + conda activate + VENV_DIR="venvs/polaris" + source "${VENV_DIR}/bin/activate" + + +Once the virtual environment is active, any extra package can be installed with +the use of ``pip``: + +.. prompt:: bash + + python pip -m install From 90fabfd09834f837d94287bb648bdf6961c4172a Mon Sep 17 00:00:00 2001 From: Alexander Smith Date: Fri, 14 Jun 2024 14:06:38 -0400 Subject: [PATCH 2/7] Erasing an extra line --- sphinx_docs/source/alcf.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/sphinx_docs/source/alcf.rst b/sphinx_docs/source/alcf.rst index e356fa9..02ffbfa 100644 --- a/sphinx_docs/source/alcf.rst +++ b/sphinx_docs/source/alcf.rst @@ -144,7 +144,6 @@ is loaded: VENV_DIR="venvs/polaris" source "${VENV_DIR}/bin/activate" - Once the virtual environment is active, any extra package can be installed with the use of ``pip``: From fbf7c90f8c549e3a3ccb95241220a905659a19cb Mon Sep 17 00:00:00 2001 From: Alexander Smith <72695548+aisclark91@users.noreply.github.com> Date: Fri, 14 Jun 2024 16:55:23 -0400 Subject: [PATCH 3/7] Update sphinx_docs/source/alcf.rst Co-authored-by: Eric T. Johnson --- sphinx_docs/source/alcf.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sphinx_docs/source/alcf.rst b/sphinx_docs/source/alcf.rst index 02ffbfa..a56a875 100644 --- a/sphinx_docs/source/alcf.rst +++ b/sphinx_docs/source/alcf.rst @@ -149,4 +149,4 @@ the use of ``pip``: .. prompt:: bash - python pip -m install + python -m pip install From 7c32414371c23f71435eb6a78cacb20412bdc13c Mon Sep 17 00:00:00 2001 From: Alexander Smith Date: Fri, 14 Jun 2024 19:45:11 -0400 Subject: [PATCH 4/7] Adding webpages --- sphinx_docs/source/alcf.rst | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/sphinx_docs/source/alcf.rst b/sphinx_docs/source/alcf.rst index a56a875..ebece83 100644 --- a/sphinx_docs/source/alcf.rst +++ b/sphinx_docs/source/alcf.rst @@ -47,13 +47,10 @@ Project workspace is at: ``/lus/grand/projects/AstroExplosions/`` Queues ====== -https://www.alcf.anl.gov/support/user-guides/polaris/queueing-and-running-jobs/job-and-queue-scheduling/index.html - For production jobs, you submit to the ``prod`` queue. -For debugging jobs, there are two options: the ``debug`` queue for only one node use jobs, and -``debug-scaling`` for jobs that require between two nodes and 10-nodes. The former is used mainly for -tunning the optimal parameters and compare the scalability of the allocated resources. +For debugging jobs, there are two options: the ``debug`` and ``debug-scaling`` options. More information can be found in: +https://docs.alcf.anl.gov/polaris/running-jobs/#queues .. note:: @@ -120,7 +117,8 @@ on the top of the conda-based environment provided by the module conda, and inst required modules on this virtual environment. Although is very tempting to clone the whole base environment and fully customize the installed conda packages, some modules like ``mpi4py`` may require access to the MPICH libraries that are tailored to be -used only within the conda-base environment provided by the conda module. +used only within the conda-base environment provided by the conda module. All these instructions +follow the guidelines publish here: https://docs.alcf.anl.gov/polaris/data-science-workflows/python/ To create the virtual environment: From f24b56daa980b5cfadbd368533f2f7eb2a30f8b5 Mon Sep 17 00:00:00 2001 From: Alexander Smith Date: Fri, 14 Jun 2024 22:59:25 -0400 Subject: [PATCH 5/7] ammending typos --- sphinx_docs/source/alcf.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/sphinx_docs/source/alcf.rst b/sphinx_docs/source/alcf.rst index ebece83..ddc3504 100644 --- a/sphinx_docs/source/alcf.rst +++ b/sphinx_docs/source/alcf.rst @@ -112,13 +112,13 @@ A script that can be used to chain jobs with PBS is: Installing Python ================= -The most recommended way to install python is to create a virtual environment -on the top of the conda-based environment provided by the module conda, and install all the extra -required modules on this virtual environment. Although is very tempting +The most recommended way to install python in polaris is to create a virtual environment +on the top of the conda-based environment, provided by the module conda, and install all the extra +required packages on this virtual environment by using ``pip``. Although is very tempting to clone the whole base environment and fully customize the installed conda packages, some -modules like ``mpi4py`` may require access to the MPICH libraries that are tailored to be +modules like ``mpi4py`` may require access to the MPICH libraries that are only tailored to be used only within the conda-base environment provided by the conda module. All these instructions -follow the guidelines publish here: https://docs.alcf.anl.gov/polaris/data-science-workflows/python/ +follow the guidelines published here: https://docs.alcf.anl.gov/polaris/data-science-workflows/python/ To create the virtual environment: From 413bab988fb9dea1191fb014946adff6dc1c4de7 Mon Sep 17 00:00:00 2001 From: Alexander Smith Date: Fri, 14 Jun 2024 23:01:46 -0400 Subject: [PATCH 6/7] ammending typos --- sphinx_docs/source/alcf.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sphinx_docs/source/alcf.rst b/sphinx_docs/source/alcf.rst index ddc3504..f4ebc68 100644 --- a/sphinx_docs/source/alcf.rst +++ b/sphinx_docs/source/alcf.rst @@ -117,7 +117,7 @@ on the top of the conda-based environment, provided by the module conda, and ins required packages on this virtual environment by using ``pip``. Although is very tempting to clone the whole base environment and fully customize the installed conda packages, some modules like ``mpi4py`` may require access to the MPICH libraries that are only tailored to be -used only within the conda-base environment provided by the conda module. All these instructions +used by the conda-based environment provided by the conda module. All these instructions follow the guidelines published here: https://docs.alcf.anl.gov/polaris/data-science-workflows/python/ To create the virtual environment: From 818b8048371af0e5582f3437d5bbe6c6b69aa072 Mon Sep 17 00:00:00 2001 From: Alexander Smith Date: Fri, 14 Jun 2024 23:09:05 -0400 Subject: [PATCH 7/7] ammending typos --- sphinx_docs/source/alcf.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sphinx_docs/source/alcf.rst b/sphinx_docs/source/alcf.rst index f4ebc68..6c75ff0 100644 --- a/sphinx_docs/source/alcf.rst +++ b/sphinx_docs/source/alcf.rst @@ -132,7 +132,7 @@ To create the virtual environment: python -m venv "${VENV_DIR}" --system-site-packages source "${VENV_DIR}/bin/activate" -To activate it in a new terminal, if the module path ``/soft/modules`` +To activate it in a new terminal, if the module path ``/soft/modulefiles`` is loaded: .. prompt:: bash