Guide to parameters

Although many of the parameters in the params.txt file are self explanatory and commented, this page aims to provide a comprehensive description of all the non-problem-specific parameters available in the existing examples. A useful pdf guide on this topic from the latest GRChombo training day can be found in Useful resources.

Checking parameters are sensible

A useful feature in GRChombo is the ability to run a quick check on the validity of the parameters before a job is submitted to the batch system. Such a check is small enough to run on the login nodes of most systems without creating too much of a load, and ensures the job will not fail immediately due to parameter related errors.

This is done by passing the -check_params flag i.e. by running something like the following command:

mpiexec /path/to/exec.ex /path/to/params.txt -check_params

Note that the checks mostly just ensure that the simulation will run and in particular don't tell you if you are doing something very inefficient with regard to the parallelisation of the run. More detailed information about load balancing using the grid parameters is provided on the performance optimisation page. These parameters are essential for using computing time efficiently, so this is highly recommended reading!

Updating parameters during a run

Most parameters can in principle be updated on restart from checkpoint, except those related to the Grid Setup and any initial conditions (which obviously require a restart from t=0).

However, one should take care when updating block_factor and max_grid_size as this can cause issues with regridding at later times (usually seen as the boxes becoming "stuck" at certain positions). Also changing the gauge or dissipation parameters will generally cause unphysical transients and so is not recommended (although we do not prohibit it).

Guide to specific parameters

For the purposes of this page, we will refer to the params.txt in the current BinaryBH example. See also Running the BinaryBH example.

Note that a hash # denotes a comment and the rest of the line will not be read, so to activate a parameter this should be removed at the start of the line. Parameters that are unlikely to change are set to defaults, and the corresponding values are commented out in the params.txt file so they are easy to identify.

Filesystem parameters

• verbosity can be set to 0, 1, 2,... . Generally, the higher it is set, the more information is output to the pout file from each MPI rank (or just stdout if running in serial). GRChombo currently does not do anything different above 2 but the underlying Chombo library may output more up to 5.
• chk_prefix and plot_prefix are the directory and filename prefixes for the checkpoint and plot hdf5 files respectively, which will be output during the simulation.
• checkpoint_interval is the number of coarse time steps to wait before outputting a checkpoint file. Since checkpoint files are bigger than plot files, this should be greater than plot_interval.
• plot_interval is the number of coarse time steps to wait before outputting a plot file.
• Checkpoint files contain all grid values for all evolution variables on the grid, hence simulations can be restarted from them. Plot files contain all grid values only for evolution and/or diagnostic variables specified in advance. For the BinaryBH example, this is just num_plot_vars=3 - the conformal factor chi and the real and imaginary parts of the Weyl Scalar, as specified by plot_vars. As a result, plot files tend to be significantly smaller than checkpoint files and can be output more frequently. One can choose to write_plot_ghosts but this is usually not necessary unless one plans to use them as checkpoints for another run.
• Each checkpoint/plot filename prefix will be appended with a 6 digit number corresponding to the time step on the coarsest level, the number of spatial dimensions (which should be 3 for these examples) and the hdf5 file extension. For example if the prefix is set to BinaryBH_, then a checkpoint file from the 10th coarse time step will have name BinaryBH_000010.3d.hdf5.
• If a directory is not explicitly specified, then files will be output to the directory in which the binary has been executed. The can be overwritten but providing a base directory in output_path, which needs to exist at runtime. Generally, when running on a cluster, you'll want to change it to some local scratch storage or a space with fast I/O. Refer to your cluster documentation/administrator for help with this.
• Note that Unix style relative paths, for example ../filename-prefix, will probably not work and you should use absolute paths.
• If you need to restart a run from a checkpoint file for whatever reason (usually because you ran out of time or the system crashed), uncomment (remove the #) before restart_file and set it to the full checkpoint filename (including the absolute path if different from where the binary is executed).
• By default 3 folders will be created in output_path to store the hdf5 files for checkpoints and plot files hdf5, the output from processors pout and any data files generated data. These names can be amended by passing values for hdf5_subpath, pout_subpath and data_subpath. These subfolders do not need to exist at runtime and will be created.
• During the simulation each MPI process outputs data on what it is doing to the pout folder. Setting print_progress_only_to_rank_0 to 1 is usually a sensible thing to do as a lot of the output is duplicated between processors. Note that since each processor acts on a different part of the grid, different processors may encounter different issues - e.g. in detecting Nans - so you should check all pout files if your job stops suddenly (a useful command is grep -r 'Nan' pout/). This command just stops the output of data which indicates the current speed and time step on each level, which in general is redundant between processes.

Grid parameters

• N1, N2 and N3 are the number of cells on the coarsest level along each dimension. Note that this needs to be a multiple of the block_factor (minimum box size). For general 3-dimensional problems, you'll probably want to set them to the same value but for problems that are essentially 2 or 1-dimensional dynamically, you can set different values appropriately.
• L is the coordinate length of the longest side of the grid. The spatial step size dx on the coarsest level is L/Nmax where Nmax is the maximum of N1, N2 and N3. Note that dx is the same along each dimension, even if the N values differ.
• There is an option to specify alternatively N_full and L_full which provide the values ignoring the imposition of symmetries. So for example, for a single black hole in a domain of length L=128 with the black hole at the centre, but where symmetries are used to reduce the actual width of the simulation domain to L=64 with the black hole at the edge (ie, only evolving half the grid), L_full would remain set to 128, but L would need to be set to 64. A similar logic applies to N_full and N. These symmetries are also taken into account by the center parameter, which can usually be left to default.

For more information on GRChombo's implementation of the Berger-Rigoutsos AMR algorithm, see §3.1.2 of Katy Clough's PhD thesis. The following bullet points will assume some familiarity with it.

• max_level is the maximum number of refinement levels above the coarsest level, so there are max_level + 1 levels in total.
• regrid_interval is the number of time steps to wait before tagging/regridding occurs on each level. On coarser levels, you don't need to regrid as frequently and for almost static problems, you may not even need to regrid at all (set regrid_interval to 0 0 0 ...), since the mesh is always refined initially anyway. Note that max_level values are needed (as no regrinding occurs on the top level).
• regrid_threshold is the overall threshold which is used to decide whether or not to tag the cell. For more information on tagging criteria see AMR and tagging.
• ref_ratio is the ratio of the spatial step size between levels. This should just be left as 2 2 2 ... as it's possible there are parts of the code which won't work with other values. Note that max_level + 1 values are needed.
• max_grid_size, alias 'max_box_size' is the maximum number of cells along each dimension of a box (the boxes are the subsets of the grid which are shared between processors). This should definitely be greater than or equal to 8 and probably around 16 or 32 but no larger.
• block_factor, alias 'min_box_size' is the minimum number of cells along each dimension of a box. This should ideally be an integer multiple of the vector width for doubles e.g. 8 for AVX-512 (assuming you've compiled with the appropriate flags) and probably shouldn't be smaller as then the high number of ghost cells can cause memory issues. It must be a power of 2 (for a ref_ratio of 2).
• You may want to set max_grid_size and block_factor to the same value (e.g. 16) so that you know exactly how large each box is going to be. This will make it easier to optimise load balancing.
• tag_buffer_size is the number of additional cells that are added to tagged regions to smooth them out, and defaults to 3.
• grid_buffer_size controls the number of cells between the boundaries of the AMR grids at different resolutions. It defaults to 8. This can be set higher to "space out" the grid hierarchy more - it will tend to increase the size of coarser levels to achieve this, rather than reducing the size of finer ones.
• fill_ratio is the minimum proportion of cells in a box that need to be tagged for regridding to occur.
• num_ghosts is the number of ghost cells in each direction for each level. Three are required to enable the 6th order spatial derivatives used for Kreiss Oliger and the 4th order finite difference stencils. (If 6th order stencils are selected it should default to 4 automatically.)
• center should default to the centre of the grid, taking into account any symmetries, but can be overwritten.

Boundary Conditions parameters

• isPeriodic is a vector of boolean values corresponding to whether the grid boundaries along each dimension are periodic or not. If one wants to have periodic boundary conditions it should be set to 1 1 1. However, GRChombo has also implemented fixed, sommerfeld, reflective and extrapolating boundary conditions in which case isPeriodic should be be 0 0 0 and the precise condition must be chosen by using the additional parameters below.
• hi_boundary and lo_boundary are two 3d vectors that each component carries information about the boundary conditions of the coordinate at the high and low part of the simulation box. They should be set to 0 for static, 1 for sommerfeld, 2 for reflective, 3 for extrapolating and 4 for mixed. For example, if one wanted to have reflective boundary condition in $z=0$ with sommerfeld at $x=y=0$ and all fixed for $x=y=z=L$, the vectors should be set to lo_boundary = 1 1 2 and hi_boundary = 0 0 0.
  • vars_asymptotic_values: If Sommerfeld or extrapolating boundary conditions are chosen the user must specify the num_nonzero_asymptotic_vars and their names nonzero_asymptotic_vars and values nonzero_asymptotic_values in the corresponding order for those using Sommerfeld conditions.
  • vars_parity: If reflective boundary conditions are chosen the user must specify the parity of each variable at each boundary in vars_parity. 1,2,3 = odd $x$, $y$, $z$ and 4,5,6 = odd $xy$, $yz$, $xz$. If a variable is even under all xyz directions it is set to vars_parity = 0. Each component of the vector corresponds to a variable in UserVariables.hpp. Note that the order should be the same as the one in UserVariables.hpp. The same should be provided for DiagnosticVariables.
  • If mixed boundaries are selected, the number of extrapolating variables num_extrapolating_vars and their names extrapolating_vars should be specified along with the order of extrapolation extrapolation_order. The remaining variables will be assumed to be Sommerfeld and so any asymptotic values for these must also be provided as above.

Evolution parameters

• dt_multiplier is the factor you need to multiply the level spatial step size dx to get the time step size dt. This is sometimes known as the CFL (Courant-Friedrichs-Lewy) or just Courant factor. For most GR problems, this will probably need to be less than 0.5 for stability and usually 0.25 is a reasonable choice.
• stop_time is the coordinate time up to which the simulation is run.
• max_steps is a cut-off for the maximum number of coarse time steps so that the program does not run out of control.
• max_spatial_derivative_order defines the order of the spatial finite differencing stencils, and can be 4 or 6. The latter is probably required for obtaining high quality waveforms. (Note that the time evolution is always 4th order Kreiss-Oliger, and that the 6th order stencils will only be used for the evolution, and not diagnostics.)
• nan_check is a bool, and usually set to 1 to make sure the code checks for instabilities and exits normally. You may wish to turn this off (ie, set it to 0) for debugging or for final (well-tested) production runs, but it is strongly advised during development.

Gauge conditions for CCZ4/BSSN

• The included CCZ4 class uses the following slicing condition:

$$ \partial_t \alpha = a_1 \beta^k \partial_k \alpha - a_2 \alpha^{a_3}(K - 2\Theta) $$

• lapse_power is the power $a_3$. $a_3$ = 1.0 corresponds to 1+log slicing.
• lapse_coeff is the coefficient $a_2$ on the right hand side. $a_2$ = 2.0 corresponds to 1+log slicing.
• lapse_advec_coeff is the coefficient of advection terms $a_1$ in the lapse condition. To use the "advective" version of 1+log slicing (which is covariant), set this to 1.0 and to use the more conventional version set it to 0.0.
• The included CCZ4 class uses the following shift condition:

$$ \partial_t\beta^i = b_1\beta^k\partial_k\beta^i + b_2 B^i,\quad \partial_tB^i = b_1(\beta^k\partial_kB^i - \beta^k\partial_k\hat{\Gamma}^i) + \partial_t\hat{\Gamma}^i - \eta B^i, $$

• shift_advec_coeff is the coefficient of advection terms b₁ in the shift condition. You'll usually want to set this to 0.0 or 1.0.
• shift_Gamma_coeff is the coefficient b₂ on the right hand side of the first equation. The default is 0.75 which sets the speeds of the gauge modes to 1.0 asymptotically far away.
• eta is the coefficient $\eta$ of the damping term in the second equation. For the hyperbolic Gamma-driver shift condition, set this to something on the order of $1/(2M)$ where $M$ is the ADM mass of the spacetime.

CCZ4 and matter specific params

For more information about CCZ4 and the conventions used, see arXiv:1106.2254 and arXiv:1307.7391.

• formulation = 1 corresponds to BSSN and formulation = 0 corresponds to CCZ4.
• kappa1, kappa2 and kappa3 correspond to the 3 kappa values in arXiv:1106.2254 for CCZ4. To run BSSN they must be set to zero (GRChombo enforces this at runtime if formulation = 1).
  • Constraint violating modes are damped for kappa1 > 0 and kappa2 > -1.
  • kappa3 = 1 corresponds to a fully covariant formulation but, in general, is not stable for spacetimes with black holes or ones where black holes form dynamically in the evolution (although see below).
• covariantZ4 = 0 corresponds to the original CCZ4 system in arXiv:1106.2254. Setting it to 1, replaces $\kappa_1$ with $\kappa_1/\alpha$ as described in equation 27 of arXiv:1307.7391. This allows the stable evolution of black hole spacetimes with $\kappa_3 = 1$ which means you can keep spatial covariance.
• sigma is the coefficient for Kreiss-Oliger numerical dissipation and should be $\mathcal{O}(\Delta x/\Delta t)$. For details on GRChombo's implementation, see §3.2.3 of Katy Clough's PhD thesis. It cannot be made arbitrarily large as above a certain value it affects the solution. In order to guarantee stability, it must not exceed $2\Delta x/\Delta t$ (see Eq. (9.9.7) in Alcubierre) but even close to that value it can cause instability. (PRO TIP: In general more dissipation is always helpful in reducing boundary errors, but if you see the "checkerboard of death" - an instability in which alternate cells are growing in opposite directions - you probably have it set too high.)
• min_chi and min_lapse set floors for these components of the CCZ4 evolution. In general these quantities should be positive definite. Note that for cosmological spacetimes, where the value of the lapse and shift are related to the scale factor and may change exponentially, these limits may need to be disabled or lowered.
• G_Newton is the Gravitational Constant, $G$, in geometric units. The default is 1.0, but if you have a field with a very small energy density, you may wish to keep the field values of order 1 and absorb the smallness into $G$.

Extraction params

See the guidance at Extraction and integration.