Guide to parameters

Although many of the parameters in the params.txt file are self explanatory and, particularly in the ScalarField example, are commented, this page will aim to provide a comprehensive description of all the non-problem-specific parameters available in the existing examples. More detailed information about load balancing is provided on the performance optimisation page.

Additionally, a useful pdf guide on this topic from the latest GRChombo training day can be found in Useful resources.

For the purposes of this page, we will refer to the params.txt file from the ScalarField example at commit 8e59d2c.

Output

• verbosity can be set to 0, 1, 2,... . Generally, the higher it is set, the more information is output to the pout file from each MPI rank (or just stdout if running in serial). GRChombo currently does not do anything different above 2 but the underlying Chombo library may output more up to 5.
• chk_prefix and plot_prefix are the directory and filename prefixes for the checkpoint and plot hdf5 files respectively, which will be output during the simulation.
  • Checkpoint files contain all grid values for all variables on the grid, hence simulations can be restarted from them. Plot files contain all grid values only for variables specified in advance. For the ScalarField example, this is just the scalar field, phi, and the trace of the extrinsic curvature, K, and is specified in the file ScalarFieldLevel.cpp. As a result, plot files tend to be significantly smaller than checkpoint files.
  • Each checkpoint/plot filename prefix will be appended with a 6 digit number corresponding to the time step on the coarsest level, the number of spatial dimensions (which should be 3 for these examples) and the hdf5 file extension. For example if the prefix is set to ScalarField_, then a checkpoint file from the 10th coarse time step will have name ScalarField_000010.3d.hdf5.
  • If a directory is not explicitly specified, then files will be output to the directory in which the binary has been executed. Generally, when running on a cluster, you'll want to change it to some local scratch storage or a space with fast I/O. Refer to your cluster documentation/administrator for help with this.
  • Note that Unix style relative paths, for example ../filename-prefix, will probably not work and you should use absolute paths.
• If you need to restart a run from a checkpoint file for whatever reason (usually because you ran out of time or the system crashed), uncomment (remove the #) before restart_file and set it to the full checkpoint filename (including the absolute path if different from where the binary is executed).

Grid setup

• N1, N2 and N3 are the number of cells on the coarsest level along each dimension. Note that this needs to be a multiple of the block_factor (minimum box size). For general 3-dimensional problems, you'll probably want to set them to the same value but for problems that are essentially 2 or 1-dimensional dynamically, you can set different values appropriately.
• L is the coordinate length of the longest side of the grid. The spatial step size dx on the coarsest level is L/Nmax where Nmax is the maximum of N1, N2 and N3. Note that dx is the same along each dimension, even if the N values differ.

Regridding and Levels

For more information on GRChombo's implementation of the Berger-Rigoutsos AMR algorithm, see §3.1.2 of Katy Clough's PhD thesis. The following bullet points will assume some familiarity with it.

• regrid_threshold is the overall threshold which is used to decide whether or not to tag the cell.
• In this example, there are other tagging criteria: regrid_threshold_K and regrid_threshold_phi. These are used to scale the changes of the trace of the extrinsic curvature, K, and the scalar field, phi, respectively, across the cell, such that they can be combined into a single measure of order 1, which is compared to the regrid threshold.
• max_level is the maximum number of refinement levels above the coarsest level, so there are max_level + 1 levels in total.
• regrid_interval is the number of time steps to wait before tagging/regridding occurs on each level. On coarser levels, you don't need to regrid as frequently and for almost static problems, you may not even need to regrid at all (set regrid_interval to 0 0 0 ...), since the mesh is always refined initially anyway. Note that max_level + 1 values are needed.
• ref_ratio is the ratio of the spatial step size between levels. This should just be left as 2 2 2 ... as it's possible there are parts of the code which won't work with other values. Note that max_level + 1 values are needed.
• max_grid_size is the maximum number of cells along each dimension of a box (the boxes are the subsets of the grid which are shared between processors). This should definitely be greater than or equal to 8 and probably around 16 or 32 but no larger.
• block_factor is the minimum number of cells along each dimension of a box. This should ideally be an integer multiple of the vector width for doubles e.g. 8 for AVX-512 (assuming you've compiled with the appropriate flags) and probably shouldn't be smaller as then the high number of ghost cells can cause memory issues. It must be a power of 2 (for a ref_ratio of 2).
• You may want to set max_grid_size and block_factor to the same value (e.g. 16) so that you know exactly how large each box is going to be. This will make it easier to optimise load balancing.
• fill_ratio is the minimum proportion of cells in a box that need to be tagged for regridding to occur.

Boundaries

• isPeriodic is a vector of boolean values corresponding to whether the grid boundaries along each dimension are periodic or not. If one wants to have periodic boundary conditions it should be set to 1 1 1. However, GRChombo has als implemented fixed, sommerfeld and reflective boundary conditions in which case isPeriodic should be be 0 0 0 and the precise condition must be chosen by using the parameters below:
• hi_boundary and lo_boundary are two 3d vectors that each component carries information about the boundary conditions of the coordinate at the high and low part of the simulation box. They should be set to 0 for static, 1 for sommerfeld or 2 for reflective. For example, if one wanted to have reflective boundary condition in z=0 with sommerfeld at x=y=0 and all fixed for x=y=z=L, the vectors should be set to lo_boundary = 1 1 2 and hi_boundary = 0 0 0.
  • vars_asymptotic_values: If sommerfeld boundary conditions are chosen the user must specify the asymptotic values for each variable in the vector vars_asymptotic_values. Note that the order should be the same as the one in UserVariables.hpp.
  • vars_parity: If reflective boundary conditions are chosen the user must specify the parity of each variable at each boundary in vars_parity. 1,2,3 = odd x, y, z and 4,5,6 = odd xy, yz, xz. If a variable is even vars_parity = 0. Each component of the vector corresponds to a variable in UserVariables.hpp.
• num_ghosts is the number of ghost cells in each direction for each level (3 is required to enable the 6th order spatial derivatives used for Kreiss Oliger).

Time steps

• checkpoint_interval is the number of coarse time steps to wait before outputting a checkpoint file. Since checkpoint files are bigger than plot files, this should be greater than plot_interval.
• plot_interval is the number of coarse time steps to wait before outputting a plot file.
• dt_multiplier is the factor you need to multiply the level spatial step size dx to get the time step size dt. This is sometimes known as the CFL (Courant-Friedrichs-Lewy) or just Courant factor. For most GR problems, this will probably need to be less than 0.5 for stability and usually 0.25 is a reasonable choice.
• stop_time is the coordinate time up to which the simulation is run.
• max_steps is a cut-off for the maximum number of coarse time steps so that the program does not run out of control.

Gauge conditions for CCZ4/BSSN

• The included CCZ4 class uses the following slicing condition:

• lapse_power is the power, p. p = 1.0 corresponds to 1+log slicing.
• lapse_coeff is the coefficient, c, on the right hand side. c = 2.0 corresponds to 1+log slicing.
• lapse_advec_coeff is the coefficient of advection terms (i.e. those involving the shift vector and spatial derivatives) in the lapse condition. To use the "advective" version of 1+log slicing (which is covariant), set this to 1.0 and to use the more conventional version set it to 0.0.
• The included CCZ4 class uses the following shift condition:

• shift_advec_coeff is the coefficient of advection terms in the shift condition. You'll usually want to leave this as 0.0.
• shift_Gamma_coeff is the coefficient, F, on the right hand side of the first equation. The default is 0.75 which, according to Baumgarte and Shapiro, is somewhat arbitrary but leads to good numerical results.
• eta is the coefficient, eta, of the damping term in the second equation. For the hyperbolic Gamma-driver shift condition, set this to something on the order of 1/(2M) where M is the ADM mass of the spacetime.
• SpatialBetaDriverRadius is deprecated and will be removed in a future update.

CCZ4 specific

For more information about CCZ4 and the conventions used, see arXiv:1106.2254 and arXiv:1307.7391.

• formulation = 1 corresponds to BSSN and formulation = 0 corresponds to CCZ4.
• kappa1, kappa2 and kappa3 correspond to the 3 kappa values in arXiv:1106.2254 for CCZ4. To run BSSN they must be set to zero (GRChombo enforces this at runtime if formulation = 1).
  • Constraint violating modes are damped for kappa1 > 0 and kappa2 > -1.
  • kappa3 = 1 corresponds to a fully covariant formulation but, in general, is not stable for spacetimes with black holes or ones where black holes form dynamically in the evolution (although see below).
• covariantZ4 = 0 corresponds to the original CCZ4 system in arXiv:1106.2254. Setting it to 1, replaces kappa1 with kappa1/lapse as described in equation 27 of arXiv:1307.7391. This allows the stable evolution of black hole spacetimes with kappa3 = 1 which means you can keep spatial covariance.

Other

• sigma is the coefficient for Kreiss-Oliger numerical dissipation and should be O(Δx). For details on GRChombo's implementation, see §3.2.3 of Katy Clough's PhD thesis. It cannot be made arbitrarily large as above a certain value it affects the solution. In order to guarantee stability, it does not have to exceed 2Δx (see Eq. (9.9.7) in Alcubierre)
• G_Newton is the Gravitational Constant, G, in geometric units. The default is 1.0.