Add `restart_file` configuration variable, remove ENV variable, enable restarted progress logging #2777

nefrathenrici · 2024-03-11T20:40:49Z

This PR improves restarted simulations:

Content

Move the restart file from ENV["RESTART_FILE"] to parsed_args["restart_file"]. This will make it simpler to set the restart file for calibration experiments.
Enable progress logging in restarted simulations. The progress callback now takes the start time into account instead of assuming that a sim starts from 0.0

Sbozzolo · 2024-03-12T17:09:30Z

Since you are looking at this, maybe it's a good time to tackle the following.

The accumulated diagnostics cannot go across restart files. E.g., if you ask for a 30days mean, you cannot have a checkpoint in between (and restart from the checkpoint). Could you please add a check to enforce this? (Maybe in another PR)

nefrathenrici · 2024-03-12T19:13:39Z

Since you are looking at this, maybe it's a good time to tackle the following.

The accumulated diagnostics cannot go across restart files. E.g., if you ask for a 30days mean, you cannot have a checkpoint in between (and restart from the checkpoint). Could you please add a check to enforce this? (Maybe in another PR)

@Sbozzolo I am happy to take that on in a separate PR!

charleskawczynski · 2024-03-20T17:15:15Z

src/callbacks/callbacks.jl

@@ -259,10 +260,10 @@ function print_walltime_estimate(integrator)
    if wte.n_calls == wte.n_next && ready_to_report
        t = integrator.t
        n_steps_total = ceil(Int, t_end / dt)
-        n_steps = ceil(Int, t / dt)
+        n_steps = ceil(Int, (t - t_start) / dt)


I think above we need n_steps_total = ceil(Int, (t_end - t_start) / dt)

But I think that might be all we need

charleskawczynski

Looks good once this comment is resolved

nefrathenrici · 2024-03-20T19:21:49Z

After testing locally, this seems to work well. The walltime estimate, SYPD, and wall_time_per_timestep are roughly the same for an initial simulation and a restarted simulation starting from midway to t_end.

Local test:

import ClimaAtmos as CA

config_dict = Dict("t_end" => "20days", "dt_save_state_to_disk" => "10days", "job_id" => "test_restart")
config = CA.AtmosConfig(config_dict)
include("examples/hybrid/driver.jl")

config_dict = Dict("t_end" => "20days", "restart_file" => "output/test_restart/hdf5_files/day10.0.hdf5")
config = CA.AtmosConfig(config_dict)
include("examples/hybrid/driver.jl")

Initial simulation log:

julia> include("examples/hybrid/driver.jl")
┌ Info: numerics 
│            `energy_upwinding`::`Val{:none}`
│            `tracer_upwinding`::`Val{:none}`
│             `edmfx_upwinding`::`Val{:none}`
│     `edmfx_sgsflux_upwinding`::`Val{:none}`
│     `test_dycore_consistency`::`Nothing`
│                     `limiter`::`Nothing`
└         `use_reference_state`::`Bool`
┌ Info: AtmosModel: 
│ 
│                    `model_config`::`ClimaAtmos.SphericalModel`
│                       `perf_mode`::`ClimaAtmos.PerfStandard`
│                  `moisture_model`::`ClimaAtmos.DryModel`
│                    `precip_model`::`ClimaAtmos.NoPrecipitation`
│                     `cloud_model`::`ClimaAtmos.QuadratureCloud`
│                    `forcing_type`::`Nothing`
│                      `subsidence`::`Nothing`
│                  `radiation_mode`::`Nothing`
│                          `ls_adv`::`Nothing`
│                   `edmf_coriolis`::`Nothing`
│                `edmfx_entr_model`::`ClimaAtmos.NoEntrainment`
│                `edmfx_detr_model`::`ClimaAtmos.NoDetrainment`
│                  `turbconv_model`::`Nothing`
│     `non_orographic_gravity_wave`::`Nothing`
│         `orographic_gravity_wave`::`Nothing`
│                       `hyperdiff`::`ClimaAtmos.ClimaHyperdiffusion{Float32}`
│                       `vert_diff`::`Nothing`
│                       `diff_mode`::`ClimaAtmos.Explicit`
│                  `viscous_sponge`::`Nothing`
│                 `rayleigh_sponge`::`Nothing`
│                 `sfc_temperature`::`ClimaAtmos.ZonallySymmetricSST`
│                   `surface_model`::`ClimaAtmos.PrescribedSurfaceTemperature`
│                  `surface_albedo`::`ClimaAtmos.ConstantAlbedo{Float32}`
└                        `numerics`::`ClimaAtmos.AtmosNumerics{Val{:none}, Val{:none}, Val{:none}, Val{:none}, Nothing, Nothing}`
┌ Info: Time info:
│   dt = "600secs"
│   t_end = "20days"
└   floor_n_steps = 2880
┌ Info: Topography
└   topography = "NoWarp"
┌ Info: z heights
│   z_mesh.faces =
│    11-element Vector{ClimaCore.Geometry.ZPoint{Float32}}:
│     ZPoint(0.0)
│     ZPoint(499.99905)
│     ZPoint(1648.3628)
│     ZPoint(3446.1985)
│     ⋮
│     ZPoint(20350.482)
│     ZPoint(25000.0)
└     ZPoint(30000.0)
┌ Info: Resolution stats: 
│   Nq = 4
│   h_elem = 6
│   z_elem = 10
│   ncols = 3456
└   ndofs_total = 34560
[ Info: Allocating Y: 1.066 s (7574495 allocations: 498.40 MiB)
[ Info: Allocating cache (p): 7.086 s (60232789 allocations: 3.27 GiB)
[ Info: Using ODE config: `ClimaTimeSteppers.ARS343`
[ Info: ode_configuration: 357.141 ms (1315666 allocations: 82.12 MiB)
[ Info: Progress logging enabled.
[ Info: get_callbacks: 90.463 ms (145125 allocations: 9.89 MiB)
[ Info: initializing diagnostics: 1.638 s (6495816 allocations: 454.64 MiB)
[ Info: Computing diagnostics:
[ Info: HDF5Writer: Any[]
[ Info: NetCDFWriter: ["orog_inst", "ts_1d_average", "ta_1d_average", "thetaa_1d_average", "ha_1d_average", "pfull_1d_average", "rhoa_1d_average", "ua_1d_average", "va_1d_average", "wa_1d_average", "hfes_1d_average", "ts_1d_min", "ts_1d_max"]
[ Info: Prepared diagnostic callbacks: 185.852 ms (846478 allocations: 60.06 MiB)
[ Info: Prepared SciMLBase.CallbackSet callbacks: 6.160 ms (411 allocations: 29.67 KiB)
[ Info: n_steps_per_cycle_per_cb (non diagnostics): [1, 1, 1440, 18, 1]
[ Info: n_steps_per_cycle_per_cb_diagnostic: [1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144]
[ Info: n_steps_per_cycle (non diagnostics): 1440
[ Info: Define ode function: 384.913 ms (581421 allocations: 30.84 MiB)
[ Info: dt_save_to_sol: 86400.0, length(saveat): 1
[ Info: Saving state to HDF5 file on day 0 second 0
[ Info: init integrator: 4.984 s (30438281 allocations: 1.80 GiB)
[ Info: Init diagnostics: 3.186 s (18054388 allocations: 1.17 GiB)
┌ Info: Running
│   job_id = "test_restart"
│   output_dir = "output/test_restart"
└   tspan = (0.0f0, 1.728f6)
[ Info: Progress: Completed first step
┌ Info: Progress
│   simulation_time = "30 minutes"
│   n_steps_completed = 3
│   wall_time_per_step = "55 milliseconds, 787 microseconds"
│   wall_time_total = "2 minutes, 40 seconds"
│   wall_time_remaining = "2 minutes, 40 seconds"
│   wall_time_spent = "167 milliseconds, 363 microseconds"
│   percent_complete = "0.1%"
│   sypd = 29.466
│   date_now = 2024-03-20T11:55:45.398
└   estimated_finish_date = 2024-03-20T11:58:25.891
┌ Info: Progress
│   simulation_time = "50 minutes"
│   n_steps_completed = 5
│   wall_time_per_step = "93 milliseconds, 163 microseconds"
│   wall_time_total = "4 minutes, 28 seconds"
│   wall_time_remaining = "4 minutes, 27 seconds"
│   wall_time_spent = "465 milliseconds, 819 microseconds"
│   percent_complete = "0.2%"
│   sypd = 17.645
│   date_now = 2024-03-20T11:55:45.614
└   estimated_finish_date = 2024-03-20T12:00:13.460
┌ Info: Progress
│   simulation_time = "1 hour, 30 minutes"
│   n_steps_completed = 9
│   wall_time_per_step = "89 milliseconds, 250 microseconds"
│   wall_time_total = "4 minutes, 17 seconds"
│   wall_time_remaining = "4 minutes, 16 seconds"
│   wall_time_spent = "803 milliseconds, 252 microseconds"
│   percent_complete = "0.3%"
│   sypd = 18.418
│   date_now = 2024-03-20T11:55:45.965
└   estimated_finish_date = 2024-03-20T12:00:02.202
┌ Info: Progress
│   simulation_time = "2 hours, 50 minutes"
│   n_steps_completed = 17
│   wall_time_per_step = "87 milliseconds, 425 microseconds"
│   wall_time_total = "4 minutes, 11 seconds"
│   wall_time_remaining = "4 minutes, 10 seconds"
│   wall_time_spent = "1 second, 486 milliseconds"
│   percent_complete = "0.6%"
│   sypd = 18.803
│   date_now = 2024-03-20T11:55:46.634
└   estimated_finish_date = 2024-03-20T11:59:56.934
┌ Info: Progress
│   simulation_time = "5 hours, 30 minutes"
│   n_steps_completed = 33
│   wall_time_per_step = "85 milliseconds, 559 microseconds"
│   wall_time_total = "4 minutes, 6 seconds"
│   wall_time_remaining = "4 minutes, 3 seconds"
│   wall_time_spent = "2 seconds, 823 milliseconds"
│   percent_complete = "1.1%"
│   sypd = 19.213
│   date_now = 2024-03-20T11:55:47.971
└   estimated_finish_date = 2024-03-20T11:59:51.559
┌ Info: Progress
│   simulation_time = "10 hours, 50 minutes"
│   n_steps_completed = 65
│   wall_time_per_step = "84 milliseconds, 480 microseconds"
│   wall_time_total = "4 minutes, 3 seconds"
│   wall_time_remaining = "3 minutes, 57 seconds"
│   wall_time_spent = "5 seconds, 491 milliseconds"
│   percent_complete = "2.3%"
│   sypd = 19.458
│   date_now = 2024-03-20T11:55:50.639
└   estimated_finish_date = 2024-03-20T11:59:48.451
┌ Info: Progress
│   simulation_time = "21 hours, 30 minutes"
│   n_steps_completed = 129
│   wall_time_per_step = "83 milliseconds, 577 microseconds"
│   wall_time_total = "4 minutes, 704 milliseconds"
│   wall_time_remaining = "3 minutes, 49 seconds"
│   wall_time_spent = "10 seconds, 781 milliseconds"
│   percent_complete = "4.5%"
│   sypd = 19.668
│   date_now = 2024-03-20T11:55:55.930
└   estimated_finish_date = 2024-03-20T11:59:45.852
┌ Info: Progress
│   simulation_time = "1 day, 18 hours"
│   n_steps_completed = 257
│   wall_time_per_step = "108 milliseconds, 754 microseconds"
│   wall_time_total = "5 minutes, 13 seconds"
│   wall_time_remaining = "4 minutes, 45 seconds"
│   wall_time_spent = "27 seconds, 949 milliseconds"
│   percent_complete = "8.9%"
│   sypd = 15.115
│   date_now = 2024-03-20T11:56:13.113
└   estimated_finish_date = 2024-03-20T12:00:58.375
┌ Info: Progress
│   simulation_time = "3 days, 13 hours"
│   n_steps_completed = 513
│   wall_time_per_step = "110 milliseconds, 830 microseconds"
│   wall_time_total = "5 minutes, 19 seconds"
│   wall_time_remaining = "4 minutes, 22 seconds"
│   wall_time_spent = "56 seconds, 856 milliseconds"
│   percent_complete = "17.8%"
│   sypd = 14.832
│   date_now = 2024-03-20T11:56:42.004
└   estimated_finish_date = 2024-03-20T12:01:04.340
┌ Info: Progress
│   simulation_time = "5 days, 8 hours"
│   n_steps_completed = 769
│   wall_time_per_step = "110 milliseconds, 765 microseconds"
│   wall_time_total = "5 minutes, 19 seconds"
│   wall_time_remaining = "3 minutes, 53 seconds"
│   wall_time_spent = "1 minute, 25 seconds"
│   percent_complete = "26.7%"
│   sypd = 14.841
│   date_now = 2024-03-20T11:57:10.327
└   estimated_finish_date = 2024-03-20T12:01:04.152
┌ Info: Progress
│   simulation_time = "1 week, 2 hours"
│   n_steps_completed = 1025
│   wall_time_per_step = "110 milliseconds, 497 microseconds"
│   wall_time_total = "5 minutes, 18 seconds"
│   wall_time_remaining = "3 minutes, 24 seconds"
│   wall_time_spent = "1 minute, 53 seconds"
│   percent_complete = "35.6%"
│   sypd = 14.877
│   date_now = 2024-03-20T11:57:38.430
└   estimated_finish_date = 2024-03-20T12:01:03.403
┌ Info: Progress
│   simulation_time = "1 week, 1 day"
│   n_steps_completed = 1281
│   wall_time_per_step = "108 milliseconds, 65 microseconds"
│   wall_time_total = "5 minutes, 11 seconds"
│   wall_time_remaining = "2 minutes, 52 seconds"
│   wall_time_spent = "2 minutes, 18 seconds"
│   percent_complete = "44.5%"
│   sypd = 15.211
│   date_now = 2024-03-20T11:58:03.580
└   estimated_finish_date = 2024-03-20T12:00:56.376
[ Info: Saving state to HDF5 file on day 10 second 0
┌ Info: Progress
│   simulation_time = "1 week, 3 days"
│   n_steps_completed = 1537
│   wall_time_per_step = "108 milliseconds, 117 microseconds"
│   wall_time_total = "5 minutes, 11 seconds"
│   wall_time_remaining = "2 minutes, 25 seconds"
│   wall_time_spent = "2 minutes, 46 seconds"
│   percent_complete = "53.4%"
│   sypd = 15.204
│   date_now = 2024-03-20T11:58:31.324
└   estimated_finish_date = 2024-03-20T12:00:56.525
┌ Info: Progress
│   simulation_time = "1 week, 5 days"
│   n_steps_completed = 1793
│   wall_time_per_step = "108 milliseconds, 447 microseconds"
│   wall_time_total = "5 minutes, 12 seconds"
│   wall_time_remaining = "1 minute, 57 seconds"
│   wall_time_spent = "3 minutes, 14 seconds"
│   percent_complete = "62.3%"
│   sypd = 15.158
│   date_now = 2024-03-20T11:58:59.594
└   estimated_finish_date = 2024-03-20T12:00:57.476
┌ Info: Progress
│   simulation_time = "2 weeks, 5 hours"
│   n_steps_completed = 2049
│   wall_time_per_step = "108 milliseconds, 662 microseconds"
│   wall_time_total = "5 minutes, 12 seconds"
│   wall_time_remaining = "1 minute, 30 seconds"
│   wall_time_spent = "3 minutes, 42 seconds"
│   percent_complete = "71.1%"
│   sypd = 15.128
│   date_now = 2024-03-20T11:59:27.797
└   estimated_finish_date = 2024-03-20T12:00:58.095
┌ Info: Progress
│   simulation_time = "2 weeks, 2 days"
│   n_steps_completed = 2305
│   wall_time_per_step = "108 milliseconds, 715 microseconds"
│   wall_time_total = "5 minutes, 13 seconds"
│   wall_time_remaining = "1 minute, 2 seconds"
│   wall_time_spent = "4 minutes, 10 seconds"
│   percent_complete = "80.0%"
│   sypd = 15.12
│   date_now = 2024-03-20T11:59:55.738
└   estimated_finish_date = 2024-03-20T12:00:58.249
┌ Info: Progress
│   simulation_time = "2 weeks, 3 days"
│   n_steps_completed = 2561
│   wall_time_per_step = "107 milliseconds, 803 microseconds"
│   wall_time_total = "5 minutes, 10 seconds"
│   wall_time_remaining = "34 seconds, 389 milliseconds"
│   wall_time_spent = "4 minutes, 36 seconds"
│   percent_complete = "88.9%"
│   sypd = 15.249
│   date_now = 2024-03-20T12:00:21.232
└   estimated_finish_date = 2024-03-20T12:00:55.621
┌ Info: Progress
│   simulation_time = "2 weeks, 5 days"
│   n_steps_completed = 2817
│   wall_time_per_step = "107 milliseconds, 870 microseconds"
│   wall_time_total = "5 minutes, 10 seconds"
│   wall_time_remaining = "6 seconds, 795 milliseconds"
│   wall_time_spent = "5 minutes, 3 seconds"
│   percent_complete = "97.8%"
│   sypd = 15.239
│   date_now = 2024-03-20T12:00:49.020
└   estimated_finish_date = 2024-03-20T12:00:55.815
[ Info: Saving state to HDF5 file on day 20 second 0
[ Info: solve!: 312.104 s (5415880 allocations: 1.10 GiB)
[ Info: sypd: 15.168801875889496
[ Info: wall_time_per_timestep: 108 milliseconds, 369 microseconds

Restarted simulation log:

julia> include("examples/hybrid/driver.jl")
┌ Info: numerics 
│            `energy_upwinding`::`Val{:none}`
│            `tracer_upwinding`::`Val{:none}`
│             `edmfx_upwinding`::`Val{:none}`
│     `edmfx_sgsflux_upwinding`::`Val{:none}`
│     `test_dycore_consistency`::`Nothing`
│                     `limiter`::`Nothing`
└         `use_reference_state`::`Bool`
┌ Info: AtmosModel: 
│ 
│                    `model_config`::`ClimaAtmos.SphericalModel`
│                       `perf_mode`::`ClimaAtmos.PerfStandard`
│                  `moisture_model`::`ClimaAtmos.DryModel`
│                    `precip_model`::`ClimaAtmos.NoPrecipitation`
│                     `cloud_model`::`ClimaAtmos.QuadratureCloud`
│                    `forcing_type`::`Nothing`
│                      `subsidence`::`Nothing`
│                  `radiation_mode`::`Nothing`
│                          `ls_adv`::`Nothing`
│                   `edmf_coriolis`::`Nothing`
│                `edmfx_entr_model`::`ClimaAtmos.NoEntrainment`
│                `edmfx_detr_model`::`ClimaAtmos.NoDetrainment`
│                  `turbconv_model`::`Nothing`
│     `non_orographic_gravity_wave`::`Nothing`
│         `orographic_gravity_wave`::`Nothing`
│                       `hyperdiff`::`ClimaAtmos.ClimaHyperdiffusion{Float32}`
│                       `vert_diff`::`Nothing`
│                       `diff_mode`::`ClimaAtmos.Explicit`
│                  `viscous_sponge`::`Nothing`
│                 `rayleigh_sponge`::`Nothing`
│                 `sfc_temperature`::`ClimaAtmos.ZonallySymmetricSST`
│                   `surface_model`::`ClimaAtmos.PrescribedSurfaceTemperature`
│                  `surface_albedo`::`ClimaAtmos.ConstantAlbedo{Float32}`
└                        `numerics`::`ClimaAtmos.AtmosNumerics{Val{:none}, Val{:none}, Val{:none}, Val{:none}, Nothing, Nothing}`
┌ Info: Time info:
│   dt = "600secs"
│   t_end = "20days"
└   floor_n_steps = 2880
[ Info: Allocating Y: 1.479 s (3296235 allocations: 221.41 MiB)
[ Info: Allocating cache (p): 333.303 ms (362562 allocations: 36.77 MiB)
[ Info: Using ODE config: `ARS343`
[ Info: ode_configuration: 274.959 μs (168 allocations: 17.41 KiB)
[ Info: Progress logging enabled.
[ Info: get_callbacks: 22.687 ms (48099 allocations: 3.34 MiB)
[ Info: initializing diagnostics: 252.105 ms (689287 allocations: 61.86 MiB)
[ Info: Computing diagnostics:
[ Info: HDF5Writer: Any[]
[ Info: NetCDFWriter: ["orog_inst", "ts_1d_average", "ta_1d_average", "thetaa_1d_average", "ha_1d_average", "pfull_1d_average", "rhoa_1d_average", "ua_1d_average", "va_1d_average", "wa_1d_average", "hfes_1d_average", "ts_1d_min", "ts_1d_max"]
[ Info: Prepared diagnostic callbacks: 129.792 μs (91 allocations: 9.17 KiB)
[ Info: Prepared SciMLBase.CallbackSet callbacks: 5.844 ms (408 allocations: 29.30 KiB)
[ Info: n_steps_per_cycle_per_cb (non diagnostics): [1, 1, 18, 1]
[ Info: n_steps_per_cycle_per_cb_diagnostic: [1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144, 1, 144]
[ Info: n_steps_per_cycle (non diagnostics): 144
[ Info: Define ode function: 1.389 ms (183 allocations: 4.76 MiB)
[ Info: dt_save_to_sol: 86400.0, length(saveat): 1
[ Info: init integrator: 894.061 ms (418693 allocations: 40.93 MiB)
[ Info: Init diagnostics: 54.920 ms (27923 allocations: 7.13 MiB)
┌ Info: Running
│   job_id = "20days_output_test_restart_hdf5_files_day10.0.hdf5"
│   output_dir = "output/20days_output_test_restart_hdf5_files_day10.0.hdf5"
└   tspan = (864000.0f0, 1.728f6)
[ Info: Progress: Completed first step
┌ Info: Progress
│   simulation_time = "1 week, 3 days"
│   n_steps_completed = 3
│   wall_time_per_step = "53 milliseconds, 797 microseconds"
│   wall_time_total = "1 minute, 17 seconds"
│   wall_time_remaining = "1 minute, 17 seconds"
│   wall_time_spent = "161 milliseconds, 393 microseconds"
│   percent_complete = "0.1%"
│   sypd = 30.556
│   date_now = 2024-03-20T12:04:51.903
└   estimated_finish_date = 2024-03-20T12:06:09.209
┌ Info: Progress
│   simulation_time = "1 week, 3 days"
│   n_steps_completed = 5
│   wall_time_per_step = "64 milliseconds, 366 microseconds"
│   wall_time_total = "1 minute, 32 seconds"
│   wall_time_remaining = "1 minute, 32 seconds"
│   wall_time_spent = "321 milliseconds, 832 microseconds"
│   percent_complete = "0.2%"
│   sypd = 25.539
│   date_now = 2024-03-20T12:04:52.063
└   estimated_finish_date = 2024-03-20T12:06:24.428
┌ Info: Progress
│   simulation_time = "1 week, 3 days"
│   n_steps_completed = 9
│   wall_time_per_step = "72 milliseconds, 897 microseconds"
│   wall_time_total = "1 minute, 44 seconds"
│   wall_time_remaining = "1 minute, 44 seconds"
│   wall_time_spent = "656 milliseconds, 76 microseconds"
│   percent_complete = "0.3%"
│   sypd = 22.55
│   date_now = 2024-03-20T12:04:52.397
└   estimated_finish_date = 2024-03-20T12:06:36.713
┌ Info: Progress
│   simulation_time = "1 week, 3 days"
│   n_steps_completed = 17
│   wall_time_per_step = "77 milliseconds, 836 microseconds"
│   wall_time_total = "1 minute, 52 seconds"
│   wall_time_remaining = "1 minute, 50 seconds"
│   wall_time_spent = "1 second, 323 milliseconds"
│   percent_complete = "0.6%"
│   sypd = 21.119
│   date_now = 2024-03-20T12:04:53.064
└   estimated_finish_date = 2024-03-20T12:06:43.824
┌ Info: Progress
│   simulation_time = "1 week, 3 days"
│   n_steps_completed = 33
│   wall_time_per_step = "81 milliseconds, 85 microseconds"
│   wall_time_total = "1 minute, 56 seconds"
│   wall_time_remaining = "1 minute, 54 seconds"
│   wall_time_spent = "2 seconds, 675 milliseconds"
│   percent_complete = "1.1%"
│   sypd = 20.273
│   date_now = 2024-03-20T12:04:54.417
└   estimated_finish_date = 2024-03-20T12:06:48.504
┌ Info: Progress
│   simulation_time = "1 week, 3 days"
│   n_steps_completed = 65
│   wall_time_per_step = "80 milliseconds, 828 microseconds"
│   wall_time_total = "1 minute, 56 seconds"
│   wall_time_remaining = "1 minute, 51 seconds"
│   wall_time_spent = "5 seconds, 253 milliseconds"
│   percent_complete = "2.3%"
│   sypd = 20.337
│   date_now = 2024-03-20T12:04:56.995
└   estimated_finish_date = 2024-03-20T12:06:48.133
┌ Info: Progress
│   simulation_time = "1 week, 3 days"
│   n_steps_completed = 129
│   wall_time_per_step = "81 milliseconds, 600 microseconds"
│   wall_time_total = "1 minute, 57 seconds"
│   wall_time_remaining = "1 minute, 46 seconds"
│   wall_time_spent = "10 seconds, 526 milliseconds"
│   percent_complete = "4.5%"
│   sypd = 20.145
│   date_now = 2024-03-20T12:05:02.267
└   estimated_finish_date = 2024-03-20T12:06:49.245
┌ Info: Progress
│   simulation_time = "1 week, 4 days"
│   n_steps_completed = 257
│   wall_time_per_step = "101 milliseconds, 436 microseconds"
│   wall_time_total = "2 minutes, 26 seconds"
│   wall_time_remaining = "1 minute, 59 seconds"
│   wall_time_spent = "26 seconds, 69 milliseconds"
│   percent_complete = "8.9%"
│   sypd = 16.205
│   date_now = 2024-03-20T12:05:17.810
└   estimated_finish_date = 2024-03-20T12:07:17.809
┌ Info: Progress
│   simulation_time = "1 week, 6 days"
│   n_steps_completed = 513
│   wall_time_per_step = "106 milliseconds, 50 microseconds"
│   wall_time_total = "2 minutes, 32 seconds"
│   wall_time_remaining = "1 minute, 38 seconds"
│   wall_time_spent = "54 seconds, 403 milliseconds"
│   percent_complete = "17.8%"
│   sypd = 15.501
│   date_now = 2024-03-20T12:05:46.145
└   estimated_finish_date = 2024-03-20T12:07:24.453
┌ Info: Progress
│   simulation_time = "2 weeks, 1 day"
│   n_steps_completed = 769
│   wall_time_per_step = "107 milliseconds, 244 microseconds"
│   wall_time_total = "2 minutes, 34 seconds"
│   wall_time_remaining = "1 minute, 11 seconds"
│   wall_time_spent = "1 minute, 22 seconds"
│   percent_complete = "26.7%"
│   sypd = 15.328
│   date_now = 2024-03-20T12:06:14.212
└   estimated_finish_date = 2024-03-20T12:07:26.173
┌ Info: Progress
│   simulation_time = "2 weeks, 3 days"
│   n_steps_completed = 1025
│   wall_time_per_step = "108 milliseconds, 38 microseconds"
│   wall_time_total = "2 minutes, 35 seconds"
│   wall_time_remaining = "44 seconds, 836 milliseconds"
│   wall_time_spent = "1 minute, 50 seconds"
│   percent_complete = "35.6%"
│   sypd = 15.215
│   date_now = 2024-03-20T12:06:42.481
└   estimated_finish_date = 2024-03-20T12:07:27.317
┌ Info: Progress
│   simulation_time = "2 weeks, 4 days"
│   n_steps_completed = 1281
│   wall_time_per_step = "106 milliseconds, 487 microseconds"
│   wall_time_total = "2 minutes, 33 seconds"
│   wall_time_remaining = "16 seconds, 931 milliseconds"
│   wall_time_spent = "2 minutes, 16 seconds"
│   percent_complete = "44.5%"
│   sypd = 15.437
│   date_now = 2024-03-20T12:07:08.151
└   estimated_finish_date = 2024-03-20T12:07:25.082
[ Info: solve!: 155.960 s (1332648 allocations: 526.98 MiB)
[ Info: sypd: 15.177748321636802
[ Info: wall_time_per_timestep: 108 milliseconds, 305 microseconds

nefrathenrici changed the title ~~Add restart_file configuration variable, remove ENV variable~~ Add restart_file configuration variable, remove ENV variable, enable restarted progress logging Mar 11, 2024

nefrathenrici requested a review from charleskawczynski March 11, 2024 21:19

nefrathenrici force-pushed the ne/restart branch from b60c304 to 4c2d616 Compare March 11, 2024 23:36

charleskawczynski reviewed Mar 20, 2024

View reviewed changes

charleskawczynski approved these changes Mar 20, 2024

View reviewed changes

nefrathenrici force-pushed the ne/restart branch from 5b74027 to 635c5b8 Compare March 20, 2024 19:18

nefrathenrici added 2 commits March 20, 2024 12:22

Add restart_file configuration variable, remove ENV variable

bbb65cb

Enable progress estimate for restarted simulations

3b135c1

nefrathenrici force-pushed the ne/restart branch from 635c5b8 to 3b135c1 Compare March 20, 2024 19:22

nefrathenrici enabled auto-merge March 20, 2024 19:23

nefrathenrici added this pull request to the merge queue Mar 20, 2024

Merged via the queue into main with commit b184b62 Mar 20, 2024
9 of 11 checks passed

nefrathenrici deleted the ne/restart branch March 20, 2024 20:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `restart_file` configuration variable, remove ENV variable, enable restarted progress logging #2777

Add `restart_file` configuration variable, remove ENV variable, enable restarted progress logging #2777

nefrathenrici commented Mar 11, 2024 •

edited

Loading

Sbozzolo commented Mar 12, 2024

nefrathenrici commented Mar 12, 2024

charleskawczynski Mar 20, 2024

charleskawczynski Mar 20, 2024

charleskawczynski left a comment

nefrathenrici commented Mar 20, 2024 •

edited

Loading

Add restart_file configuration variable, remove ENV variable, enable restarted progress logging #2777

Add restart_file configuration variable, remove ENV variable, enable restarted progress logging #2777

Conversation

nefrathenrici commented Mar 11, 2024 • edited Loading

Sbozzolo commented Mar 12, 2024

nefrathenrici commented Mar 12, 2024

charleskawczynski Mar 20, 2024

Choose a reason for hiding this comment

charleskawczynski Mar 20, 2024

Choose a reason for hiding this comment

charleskawczynski left a comment

Choose a reason for hiding this comment

nefrathenrici commented Mar 20, 2024 • edited Loading

Add `restart_file` configuration variable, remove ENV variable, enable restarted progress logging #2777

Add `restart_file` configuration variable, remove ENV variable, enable restarted progress logging #2777

nefrathenrici commented Mar 11, 2024 •

edited

Loading

nefrathenrici commented Mar 20, 2024 •

edited

Loading