Compiling Chombo

You should checkout the adapted Chombo 3.2 version from this git repository (currently GRChombo will not compile with the standard 3.2 version of Chombo).

The variables used by the Chombo build system can be set by creating a file Make.defs.local in the directory

Chombo/lib/mk/

This file does not exist so you need to create it either by copying the file Make.defs.local.template:

cp Make.defs.local.template Make.defs.local

or (which may be easier) by copying one of the example Make.defs.local files for a cluster similar to your own in the InstallNotes/MakeDefsExamples folder of GRChombo. This folder contains examples for Make.defs.local files for machines we have already run GRChombo on.

As noted in Prerequisites, there are several module files that you may need to load first, depending on your cluster. For example, the Intel and GCC compilers, a parallel HDF5 library and Intel MKL.

The main variables you need to set in Make.defs.local are

• DIM = 3
  • GRChombo only works in 3 spatial dimensions.
• DEBUG = TRUE/FALSE
  • This enables/disables debugging flags and also some more stringent checks at points.
  • Note that this will probably slow your code down so you will want to set it to FALSE when compiling for production scale simulations.
• OPT = FALSE/TRUE/HIGH
  • TRUE/HIGH enables some optimization flags that will speed up the code.
  • HIGH disables Chombo asserts (which do lots of little sanity checks so will make it harder to debug) and initialises uninitialised values to 0 rather than NaN.
  • You will want to use HIGH for production scale simulations.
• PRECISION = DOUBLE
  • These days you are unlikely to want to use float precision.
• CXX = <C++ compiler>
• FC = <Fortran compiler>
• OPENMPCC = TRUE/FALSE
  • This enables/disables OpenMP which is the shared memory parallelisation that [GR]Chombo uses.
• MPI = TRUE/FALSE
  • This enables/disables compilation with MPI.
• MPICXX = <MPI C++ Compiler>

Examples for the above variables:

DIM               = 3
DEBUG             = FALSE
OPT               = TRUE
PRECISION         = DOUBLE
CXX               = g++
FC                = gfortran
OPENMPCC          = TRUE
MPI               = TRUE
MPICXX            = mpicxx

If you want to use HDF5 (you probably do) then you will also need to set the following variables:

USE_HDF          = TRUE
HDFINCFLAGS      = -I<path to hdf5-serial>/include
HDFLIBFLAGS      = -L<path to hdf5-serial>/lib -lhdf5 -lz
HDFMPIINCFLAGS   = -I<path to hdf5-parallel>/include
HDFMPILIBFLAGS   = -L<path to hdf5-parallel>/lib -lhdf5 -lz

You can find the relevant HDF5 paths by displaying the details of the hdf5 module you have loaded (using e.g. module display hdf5). If you only plan to run the code in parallel, you can set the paths to the same parallel library for serial and parallel.

Note that you need to remove the # (uncomment) at the beginning of lines you want to make active.

For a more comprehensive list of makefile variables, have a look at the comments in Make.defs.local.template. There are also some notes on some useful ones at the bottom of this page.

Once the Make.defs.local file has been created go to Chombo/lib/ and run

make all -j 8

This compiles in parallel on 8 ranks, and should compile and run without errors. See Compilation FAQ for advice on common problems. The Chombo README file also provides some hints for compilation.

This will also compile the tests which is a good thing to do once but subsequently it is faster to only compile the library by invoking

make lib -j 8

One can also run the tests and make and run the examples, using grep to test they run ok (although sometimes this gives errors, in which case don't worry - the main thing is that make all works ok, and GRChombo tests run ok):

make run
make run | grep "finished with"
cd ../releasedExamples
make all
make run | grep "finished with"

If you subsequently change compiler options, use the command

make clean

or

make realclean

in the lib directory, to start the compilation from a clean slate or change the XTRACONFIG variable (see below).

Other Make.defs.local variables

There are some other useful variables:

• [cxx/f/cpp][dbg/opt]flags
  • These are flags passed to the C++/Fortran compilers or C preprocessor when DEBUG or OPT is TRUE (as well as HIGH in the case of OPT). For example, one might want to set (if using GCC)

cxxoptflags = -march=native -O3

where the first flag -march=native tells the compiler to target the current architecture including all its supported instruction sets and the second is an optimization level flag.

• CXXFLAGS, CPPFLAGS, FFLAGS, LDFLAGS
  • These are flags that are passed to all C++ compilations (both MPI and serial), C preprocessings, Fortran compilations and linking respectively.
• syslibflags
  • Add any flags here to link with system libraries, for example to link BLAS and LAPACK

syslibflags = -lblas -llapack

or to link with the sequential MKL libraries (we don't want the parallel ones as Chombo does its own parallelisation) using the Intel compiler

syslibflags = -mkl=sequential

• XTRACONFIG
  • This is a string which you can use to label a particular build of the Chombo library. If you wish to change compiler options but keep your existing build, you just need to change this variable.