Investigate new instability in mid-resolution AMIP benchmark

[LenkaNovak]

Unconverged surface fluxes arise after the last series of upgrades, after ~120 sim days. This needs thorough investigation.

Builds

• Benchmark longrun fails after 120d with this configuration, though this used to run 140d+

• August benchmark w/o CO2 with this config

  • NB Atmos internal flags may have changed
  • w/o CO2 - fails after 60d build
  • running prior

• Emulating atmos aquaplanet in coupled and not coupled scenario - atmos yml config

  • w CO2 - crash 5d
  • w/o CO2 - crash 10d
  • uncoupled - both w CO2 & without still running

• long runs with different UFs ... coming up

• spin up of new_target dominated by tropical convection... this was resolved by using less abrupt initial conditions in the surface.

Findings

• maxiter may be an issue
• Better stability may be achievable with other UFs (Businger rn) - use FDScheme if so
• coarse benchmarks expected to be inaccurate, since dz_1 = 500m
• LSM - ppt fluxes opposite to convention; possibly an extra E term in melt
• W and sigma S can become negative : e.g. here the evaporative flux and snow fraction are based on the previous timestep
  • proposed fix for LSM negative W and sigmaS #472
• race condition in land albedo read #470
• may need to dss P_liq and P_snow
• even when calculating flux at atmos substeps e_tot is not conserved.
• atmos calls this ts with maxiter 3 and rtol 0.003. Seems overly restrictive and fails with seemingly ok states.
• current atmos T_init_sfc seems to be quite extreme and crashes most coarse runs
• using the same T_init as atmos + constant SST in the slab ocean breaks
  • T is set properly - but still crashes (build 275)
  • try update_surface_conditions! from the coupler in the model update
    • prescribing T_sfc etc crashes (still conserving) build
    • update_surface_conditions! without changing the initial ts , but still calculating fluxes outside at dt_cpl build
    • stepping from within atmos build
  • check all other MOST parameters are consistent - z0m,b
  • local runs
    • standalone atmos - nocouple
    • standalone atmos - 400s couple; fluxes during Atmos stepping, no update sim -> identical to the above
    • standalone atmos - 400s couple; fluxes during Atmos stepping, with update sim T -> identical to the above
    • standalone atmos - 400s couple; fluxes during Atmos stepping, with update sim T, alpha -> identical to the above (with alpha = 0.38)
    • coupled atmos - dt_cpl=400s; fluxes during dt_cpl, with update sim T, alpha, constant ocean -> small qualitative differences (e.g. F_turb_energy at 10d spin up):
      
    • coupled atmos - dt_cpl=400s; fluxes during dt_cpl, with update sim T, alpha, sluggish bucket
      
    • increasing the thermal heat capacity of the bucket brings it closer to the aquaplanet
  • initialization stage checked for proper field exchange

Components

• general callbacks - this allows easier checkpointing at an arbitrary interval. Generalize callbacks #482
• revamp longruns buildkite pipeline #477
  • integration hierarchy
  • longrun hierarchy, re-add snow, fix ice mask update, add topo, check insolation inconsistency

Next steps

• topo runs seem to break at ~130days, thermal structure is not quite correct, also seen in atandalone atmos runs. More physical investigation needed: Phase 2: Stabilizing long runs #485
• add better diagnostics to CI pipeline

[akshaysridhar]

See also buildkite #1906

• Compare Builds 1902 v 1908

[LenkaNovak]

Closes Phase 1 Milestone #389