Support reproducible restarts

[Sbozzolo]

RRTMGP uses a random number generator. If we want to restart a run and debug it in a deterministic way, we need to control how random numbers are generated.

The simplest way I can think of is to force the seed to something deterministic before writing the restart file. The seed could be anything (even time) and could be stored to the restart file, so that we can reproduce the same stream of random numbers.

[sriharshakandala]

One simple workaround, for now, could be to set the seed just before the solve_lw and solve_sw calls in ClimaAtmos.jl. This could be a bit restrictive in terms of random number quality, but should work! cc: @szy21 , @akshaysridhar

[Sbozzolo]

One simple workaround, for now, could be to set the seed just before the solve_lw and solve_sw calls in ClimaAtmos.jl. This could be a bit restrictive in terms of random number quality, but should work! cc: @szy21 , @akshaysridhar

Right, that's what I suggested, but only for when we save a checkpoint, so that it doesn't impact much the generation of random numbers. We can still make the seed "random" by taking it to be the current time. What's important is that it is saved.

[sriharshakandala]

One simple workaround, for now, could be to set the seed just before the solve_lw and solve_sw calls in ClimaAtmos.jl. This could be a bit restrictive in terms of random number quality, but should work! cc: @szy21 , @akshaysridhar

Right, that's what I suggested, but only for when we save a checkpoint, so that it doesn't impact much the generation of random numbers. We can still make the seed "random" by taking it to be the current time. What's important is that it is saved.

Yes. Seeding to current "simulation" time (converted to UInt32) in ClimaAtmos.jl just before the longwave and shortwave calls would be a good option.

[sriharshakandala]

I believe this issue needs to be moved to ClimaAtmos.jl!