Ability for globally averaged output

[bogensch]

It would be nice to have the capability to produce domain/globally averaged output. My real motivation for this is for DP simulations. This would allow for high frequency output of profiles while saving substantially on storage requirements and post-processing times. Currently this is difficult to do in high resolution (or large domain and/or long duration) DP runs because you need to frequently output 3D snapshots and the storage demand can quickly mount. EXAMPLE: Preparing to submit DP simulations to RCEMIP2 each simulation (of which there are five) produces ~10 TB of data. More than 95% of this is because of the need to output 3D snapshots to make the domain averaged profiles needed for submission. Had we this capability, each simulation would produce less than 0.5 TB of data.

Most CRMs (i.e. SAM) have this capability and would be a large quality of life boost for DP users. I imagine this could also be useful for global simulations as well, but probably less so.

[PeterCaldwell]

Can't we do this now by writing the output on a grid which just has a single location and uses mapping weights = 1/cell area? Maybe it is a bit of a hack, but it is something.

[AaronDonahue]

I was going to say the same thing.

[bogensch]

@PeterCaldwell thanks; once I am able to get a DPv1 case up and running I will try this out and report back here whether or not it was successful.

[bartgol]

Yeah, sounds like it could be a useful feature. Is the avg supposed to be only in the horiz direction?

[bogensch]

Yes, my desire would be for horizontal average only.

[mahf708]

@bartgol, if we end up doing this, we should implement it in a composable fashion so that it can easily be applied to all sorts of stats (min, max, weighted mean, unweighted mean, median, etc.) at different stages of outputs (e.g., derived_value_at_850Pa_weighted_mean) ... which reminds of a convo we had about "mag(u,v) at 10m" 😄

[bartgol]

Yeah, this would be of course something we can use in combination with other options (avg type, v/h remap, diags...). That said, there is one scenario that prevents us from using this in a meaningful way: masking. If output is masked, then it makes no sense to compute a global avg. I suppose we could check the input fields, to see if they have a mask field associated to them, or if they have a fillValue associated to them, or... I don't recall all the ways we can get a masked/filled value in a field...

[bogensch]

Now that I have prototype DPxx to play with; I went back to attempt to employ the methods suggested by @PeterCaldwell and @AaronDonahue (near the top of the issue). I now understand that to do this you would need a remapping file. This is not desirable for DP simulations since domain size / resolution are always different based on user preference and the case selected; thus not desirable to rely on the user to generate this for each instance.

Thus, for a more elegant and flexible solution, I still advocate my wishlist for the functionality to allow for the option of globally (horizontally) averaged output. Again, while my motivation is for DP simulations, this could also be beneficial for the global model as well.

(changed the wording of this issue from "domain/globally averaged output" to simply "globally averaged output" as I understand some were confused that I wanted averages of subdomains, which is not the case... and a more complicated issue).

[bartgol]

@bogensch Are you ok having all fields in an nc file being globally averaged, or would you expect to have a mix of averaged and spatially varying fields?

Also, I mentioned above masked fields, for which global horiz average makes no sense. I am assuming DP won't generate masked fields, since it has a planar geometry, right? If that's the case, we can simpy error out if we see one (just in case), but assume otherwise that all values are fine.

[bogensch]

@bartgol yes absolutely fine with all fields being globally averaged in a particular nc file (that was what I was expecting the final product to be anyway).

And yes, DP won't generate masked fields.