sphere_aquaplanet_rhoe_equilmoist_allsky_gw_res is nondeterministic

[szy21]

See the latest build of main branch here: https://buildkite.com/clima/climaatmos-ci/builds/19580#01907559-2d36-4711-856d-c68813e7b7e9. It fails immediately but works when rerunning. We saw something similar in the coupler that 1M jobs may be nondeterministic. @trontrytel Any idea if there could be something not initialized correctly for 1M?

[trontrytel]

No idea right now. But I'll look into that when I start working on making 1M scheme more stable in Atmos

[szy21]

Sounds good, thanks! It could be radiation too, or some combination of radiation and microphysics, so let me know if you can't find anything, and I'll look into radiation.

[szy21]

Actually I think it might be aerosols (although it doesn't explain the coupler case, which doesn't have aerosols)! I'll add a fix.

[szy21]

Nevermind the aerosol fix didn't fix this. When it breaks both e_int and q_tot are NaNs so maybe it's not radiation, as radiation shouldn't affect q_tot.

[szy21]

I found one more thing. Let's see if #3178 helps. I am still not sure if it's from radiation though.

[szy21]

I don't see it anymore. I will close it for now and we can reopen if if we see it again.