Compiling and Running miniWeather

Software Dependencies

• Parallel-netcdf: https://trac.mcs.anl.gov/projects/parallel-netcdf
  • This is a dependency for two reasons: (1) NetCDF files are easy to visualize and convenient to work with; (2) The users of this code shouldn't have to write their own parallel I/O.
• Ncview: http://meteora.ucsd.edu/~pierce/ncview_home_page.html
  • This is the easiest way to visualize NetCDF files.
• MPI (OpenMPI and MPICH are the most common implementations)
• For OpenACC: An OpenACC-capable compiler (PGI / nvhpc, Cray, GNU)
  • A free version of the PGI / nvhpc compiler can be obtained here: https://developer.nvidia.com/hpc-sdk
• For OpenMP: An OpenMP offload capable compiler (Cray, X)
• For C++ portability, Nvidia's CUB and AMD's hipCUB and rocPRIM are already included as submodules
• CMake: https://cmake.org

On Ubuntu, the pnetcdf, ncview, mpi, and cmake dependencies can be installed with:

sudo apt-get install cmake libopenmpi-dev libpnetcdf-dev ncview

Basic Setup

git clone git@github.com:mrnorman/miniWeather.git
cd miniWeather
git submodule update --init --recursive

Directory structure

• miniWeather
  • c: C code source files
    • c/build: Where to build the C code. The C code is technically C++ code, but the style is all C except with references instead of pointers and some other conveniences like better timers.
  • fortran: Fortran code source files
    • fortran/build: Where to build the Fortran code
  • cpp: C++ code source files
    • cpp/build: Where to build the C++ code

Building and Testing Workflow

Note that you must source the cmake scripts in the build/ directores because they do module loading and set a TEST_MPI_COMMAND environment variable because it will differ from machine to machine.

cd miniWeather/[language]/build
source cmake_[machine]_[compiler].sh
make
make test

To run the code after confirming the tests pass, you can launch the executable of your choice with mpirun -n [# tasks] ./executable_name on most machines. You'll see a number of executables, e.g., serial, mpi, openmp. You'll also see the same executables with _test at the end. Those are for testing purposes. Please run the non-test executables for the test case and grid size you passed through CMake.

On Summit, it gets more complicated, unfortunately. You need to deal with so-called ``resource sets''. The following are single-node options:

• MPI-only: jsrun -n 6 -a 7 -c 7 ./executable_name
• MPI + GPU: jsrun -n 6 -a 1 -c 1 -g 1 ./executable_name

For more nodes on Summit, just multiply the -n parameter by the number of nodes.

Summit: On OLCF's Summit computer (and several other computers), you need to have an allocation loaded in order to run the code. You cannot run the code on the login nodes because the code is compiled for spectrum-mpi, which will give a segmentation fault unless you run the executable with jsrun. It's easiest to grab an interactive allocation on one node for two hours. Each Summit node has six GPUs and 42 CPU cores.

Also, on Summit, it's best to clone the entire repo in /gpfs space because the unit tests require writing files, which can only be done in GPFS space.

Changing the test case, grid size, output frequency, and total simulation time

You can set the test case, grid size, and / or output frequency through the CMake script you use to build.

• -DDATA_SPEC=` changes the test case. Choices include:
  • DATA_SPEC_THERMAL: A rising thermal (kind of like a mushroom cloud)
  • DATA_SPEC_COLLISION: A cold thermal and warm thermal colliding into one another to create some fun looking turbulence
  • DATA_SPEC_GRAVITY_WAVES: A vertical wind perturbation in a stable atmosphere, leading to gravity waves kind of like you get past the leeward side of a mountain
  • DATA_SPEC_DENSITY_CURRENT: A cold bubble crashing into the ground and spinning up eddies as it propagates outward at the surface
  • DATA_SPEC_INJECTION: Injecting cold fast air at the top left side of the domain purely to make fun turbulence.
• -DNX=: Number of cells in x-direction (should be 2x that of NZ)
• -DNZ=: Number of cells in z-direction (should be half that of NX)
• -DSIM_TIME=: Amount of time to simulate in seconds
• -DOUT_FREQ=: Number of seconds between outputting the miniWeather model state to NetCDF file.