Group NCDatasets.sync calls

[charleskawczynski]

In light of #2827, this PR hoists the NCDatasets.sync to a separate callback so that we can (eventually) and use Threads.@spawn to leverage threads (if we have them).

I've also named/annotated compute_callback and output_callback, so that they show up in the NVTX reports.

[Sbozzolo]

I think we should also avoid calling sync for every diagnostic. Instead, let's flush the output once per diagnostic cycle.

As in #2623

[charleskawczynski]

I think we should also avoid calling sync for every diagnostic. Instead, let's flush the output once per diagnostic cycle.

As in #2623

Yeah, I agree, but first I'd like to see how this performs, since it's functionally equivalent. Updating it to be called every diagnostics lcd cycle is a much simpler change than this is turning out to be.

[Sbozzolo]

I think we should also avoid calling sync for every diagnostic. Instead, let's flush the output once per diagnostic cycle.
As in #2623

Yeah, I agree, but first I'd like to see how this performs, since it's functionally equivalent. Updating it to be called every diagnostics lcd cycle is a much simpler change than this is turning out to be.

Do you mean functionally equivalent to main?

[charleskawczynski]

Do you mean functionally equivalent to main?

Yeah. It's technically different in terms of order of compute / io, but it should at least be calling sync the same number of times.

[Sbozzolo]

Something we might want to keep in mind is this:

https://juliaparallel.org/MPI.jl/stable/reference/environment/#MPI.ThreadLevel

[Sbozzolo]

(It should not be a problem at the moment because sync does not contain an MPI call.)

[charleskawczynski]

Yeah, I thought that only the root process does this, but it's a good point.