Job has non-deterministic results due to RRTMGP

[nefrathenrici]

Job ID: sphere_aquaplanet_rhoe_equilmoist_allsky_gw_raw_zonallyasymmetric
This is likely caused by cloud-radiation interactions in the simulation added after #2704 .

[szy21]

I did some test by looking at the norm of the solution. This may not be as accurate as MSE, but it shows:
FT32 simulation with FT32 radiation: non-determinisitic
FT64 simulation with FT64 radiation: non-deterministic
FT32 simulation with FT64 radiation (which is the same as what we have before): deterministic

@dennisYatunin suggested that maybe radiation is always not deterministic, but in the FT32 simulation with FT64 radiation case, we lose the last few non-deterministic digits when converting from FT64 to FT32. That makes sense to me. I don't think the issue is related to the PR.

[szy21]

I think this may have been fixed by the recent fix in RRTMGP. I run the simulation a couple of times on ci and the norm of the resolution is the same. I can add back the regression test. @charleskawczynski is there an easy way to make sure it is reproducible before adding the regression test?

[charleskawczynski]

I think opting into the regression test is the simplest way

[szy21]

Ok, I’ll add it in and keep an eye on the builds.

[szy21]

The regression test has been added back in #2798, and the most recent two builds passed, so I think we can close this issue.