Configuration Options

The configuration options for ngen-cal are separated into main sections described blow. A complete example configuration file with commented options is available on the repository.

General

The general block contains model agnostic configuration of the software. The following is a complete general configuration block

#An exampmle ngen-calibration configuration file
general:
  # Strategy configuration
  strategy: 
      # Type of strategey, currently supported is estimation
      type: estimation
      #defaults to dds
      algorithm: "dds"
      # To adjuts the neighborhood size parameter of the dds algorithm, uncomment the following two lines
      #parameters:
      #  neighborhood: 0.5
      # To use PSO optmization, select the pso algorithm and configure its parameters as follows
      #algorithm: "pso"
      #parameters:
      #    pool: 4 #number of processors to use (by default, uses 1)
      #    particles: 8 #number of particles to use (by default, uses 4)
      #    options: #the PSO parameters (defaults to c1: 0.5, c2: 0.3, w:0.9)
      #        c1: 0.1
      #        c2: 0.1
      #        w: 0.42

  #iteration control
  # In general, the start iteration should probably always be 0
  # if `restart: true`, this is overridden by the detected existing iterations saved
  # otherwise, this can have some odd consequences, and will likely be removed from the
  # configuration support in the near future
  start_iteration: 0
  # The total number of search iterations to run
  # Note that if using `restart: true`, you shouldn't change the number of total iterations
  # though it may be tempting to use this to extend a previous run, some algorithms are sentstive
  # to the total number of iterations, and changing it unexpectedly across restarts may lead
  # to undesirable results
  iterations: 100

Optional keys in the general configuration include

• random_seed: <number>

  Optional value to seed the random generators, useful for reproducing results

  Defaults to None, which will not use a seed

• name: <string>

  Optional custom name for this run, can be used to generate unique output files

  Defaults to "ngen-calibration"

• restart: <boolean>

  Attempt to restart a previous calibration. Will look for log and parameter information in an existing worker dir to restart If the required restart information cannot be found, will start back from the 0 iteration

  Defaults to false

• workdir: <path>

  working directory for the calibration runs note that all relative paths in this configuration file as well as any relative paths in any model input files will be relative to this working directory

  defaults to ./

• log: <boolean>

  Enable model runtime logging (captures standard out and error and writes to file) logs will be written to <model.type>.log when enabled

  defaults to False, which sends all output to /dev/null

• parameter_log_file: <string>

  Name of the best parameter log file

  defaults to <name>_best_params.txt

• objective_log_file: <string>

  Name of the objective function log file

  defaults to <name>_objective.txt

Model

#Model specifc configuration
model:
    # Which model to execute for the search optimization
    # Currently only support `ngen` and `none` (for testing purposes)
    type: ngen
    # A binary in $PATH or a qualified path to the binary to run
    binary: "ngen"
    
    # Required path to ngen realization config (with calibration info included)
    realization: ../cfe-cal-2/realization_config.json
    # Required path to catchment hydrofabirc file
    catchments: ../cfe-cal-2/hydrofabric/catchment_data.geojson
    # Required path to nexus hydrofabric file
    nexus: ../cfe-cal-2/hydrofabric/nexus_data.geojson
    # Required path to hydrofabric crosswalk file
    crosswalk: ../cfe-cal-2/hydrofabric/crosswalk.json
    #ngen calibration strategies include
    #uniform: Each catchment shares the same parameter space, evaluates at one observable nexus
    #independet: Each catchment upstream of observable nexus gets its own permuated parameter space, evalutates at one observable nexus 
    #explicit: only calibrates basins in the realization_config with a "calibration" definition and an observable nexus
    strategy: independent
    params: 
        CFE: *cfe_params
    
    eval_params:
      # This is the range of the hydrograph dates to run the objective function over
      # To evaluate the entire period, you can comment these lines out
      #evaluation_start: '2015-12-15 12:00:00'
      #evaluation_stop: '2015-12-30 23:00:00'
      # choices are "kling_gupta", "nnse", "custom", "single_peak", "volume"
      objective: "kling_gupta"
      # one can also provide a module path to any function that takes
      # obs, sim array-like arguments and produces a single value float
      # for example, nnse above could be called this way
      #objective: "ngen_cal.objectives.normalized_nash_sutcliffe"
      # Can choose to minimize the objective function or maximixe it (only when using the DDS algorithm)
      # choices are 'min', 'max'.
      # An an explicit floating point value can be supplied instead, and the
      # optmization will attempt to converge on that value
      # Default: min
      #target: 0.0

Optional keys in the model configuration block include

• parallel: <int>

  If running ngen in parallel, provide the number of requested parallel processes. The binary will get prepended with mpirun -n parallel automatically

  Must be >= 2

• partitions: <path>

  If a parallel run is requested, ngen requires a static partitions.json input file this is the path to that file

• args: By default, ngen args will be <catchments> "all" <nexus> "all" <realization> and if running in parallel <catchments> "all" <nexus> "all" <realization> <partitions> If you provide a custom arg string here, these are passed directly to the binary and no adjustments are made

  Defaults to null/None

Parameter references

The above model definition references the calibration parameters to use for CFE. This utilizes yaml references, so these parameters can be an independently defined block.

#Describe the model parameters you want to use, valid for independent and uniform
#ngen strategies
cfe_params: &cfe_params
  - 
      name: maxsmc
      min: 0.2
      max: 1.0
      init: 0.439
  - 
      name: satdk
      min: 0.0
      max: 0.000726
      init: 3.38e-06
  - 
      name: slope
      min: 0.0
      max: 1.0
      init: 0.01
#   - 
#       name: bb
#       min: 0.0
#       max: 21.9
#       init: 4.05
  - 
      name: expon
      min: 1.0
      max: 8.0
      init: 6.0