Debugging tools

This page outlines some of the tools present in GRChombo and Chombo to simplify the debugging of equations.

Adding to the information below, a useful pdf guide on this topic from the latest GRChombo training day can be found in Useful resources.

When debugging equations, it is often helpful to print out variables. To this end it is usually best to use Chombo's pout() function, which writes the output for each rank to a file pout.# (with # the number of the rank). If MPI isn't used, the output is simply written to the standard output.

A usage example:

pout () << "Value of var: " << var << std::endl;
Example output: Value of var: 1;

Since GRChombo uses explicit vectorisation "var" may be a vector. GRChombo implements the necessary operator for the above call to work for a vectorised variable. Example output: Value of var: ( 1 2 3 4 ).

Since the above calls are very commonly used, GRChombo implements the following macros to speed up debugging:

DEBUG_SHOW(var): expands to pout () << "var: " << var
DEBUG_FILE: prints the current filename (convenient if one is outputting at many locations)
DEBUG_END: expands to pout () << std::endl
DEBUG_OUT(var): expands to pout () << __FILENAME__ << ": " << "var: " << var << std::endl
DEBUG_OUT2(var1,var2): like DEBUG_OUT but prints both var1 and var2.
DEBUG_OUT3(..), DEBUG_OUT4(...) are also implemented like DEBUG_OUT2.

To use these macros one must include the debugging tools (#include "DebuggingTools.hpp").

When debugging equations it is often helpful to output variables only for a specific cell at arbitrary points in the code. This is not entirely trivial since the coordinate of the current cell is not usually passed around to all functions (as it isn't usually necessary). Furthermore, due to the vectorisation statements we always deal with several cells at the same time. The variable 'current_cell' that is available in all compute classes only points to the first cells in a vector. This means that while if (current_cell == IntVect(int1,int2,int3)) is allowed, the condition will never become true if int1 is not a multiple of the vector width. In general, statements like if (var > 1e10) will not compile.

To solve all these problems GRChombo allows to switch off vectorisation temporarily for any compute function by passing disable_simd() to BoxLoops. For example to switch off vectorisation in the constraint calculation:

BoxLoops(Constraints(m_dx), m_state_new, m_state_new, SKIP_GHOST_CELLS, disable_simd());

Most importantly, however, GRChombo has an equation debug mode in which:

1. Vectorisation is switched off everywhere in the code.
2. The coordinates of the current cell become a global variable: s_current_integer_coords.
3. Since the above does not (currently) work with threading, the code now checks that OMP_NUM_THREADS = 1.

To use the equation debug mode, add -DEQUATION_DEBUG_MODE to the cxxcppflags in $CHOMBO_HOME/mk/Make.defs.local; then make sure to run make clean and compile again. When you run in equation debug mode, the executable might complain about multiple threads. If this happens, make sure that you are only using one thread by typing export OMP_NUM_THREADS=1.

In equation debug mode, while BoxLoops is iterating through the grid, one can query the current coordinates anywhere in the code (not just in the compute class) and output variables as demonstrated in the example below:

#include "DebuggingTools.hpp"
...
if (s_current_integer_coords == IntVect(1,2,3)) //Output only at the cell with integer coordinates (1,2,3)
{
   DEBUG_HEADER; //Writes a header line with filename and position. Note: only available in equation debug mode.
   DEBUG_SHOW(var1); //Output your favourite variable
   DEBUG_SHOW(var2);
   DEBUG_END; //Start a new line
}