Assorted small improvements #2609
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sbozzolo
force-pushed
the
gb/test_ci
branch
5 times, most recently
from
bc0cf37 to
ccab9c0
Compare
Sbozzolo
force-pushed
the
gb/test_ci
branch
2 times, most recently
from
540ac34 to
2da4805
Compare
|
I think we should enable a way to statically elide the diagnostics (#2598). I like a handful of changes in this PR (using ClimaCore API for getproperty calls, GHA timeout times), but the diagnostic pieces are pretty different, and I think need more though, since the flame graph is showing a pretty big perf hit. Can you split the PR up? I'm happy to get non-controversial parts in asap. |
This value is not important for the test, but it seems to cause problems for GitHub actions on Ubuntu (it's extremely slow).
If these jobs run for more than 30 minutes, there's something wrong
charleskawczynski
approved these changes
Feb 2, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I was trying to debug the issue with a test taking forever to run on ubuntu on GitHub and I ended up cleaning a few small things. I identified the problem with the interpolating the diagnostics. I don't still don't know why this takes so long on GitHub: it is approximately 1000x slower than everywhere else. I verified that the package versions are identical (between a "good" run and a "bad" one). In any case, we don't need any diagnostic for the specific test, so I disabled them.
The main real change in this PR is to reduce the number of syncs that are forced for NetCDF files.
sync(= flush) was introduced because when running on a GPU, the NetCDF file will be buffered until the end of the run. This was problematic for simulations that are interrupted for any reason (all the data is lost).Now,
syncis called once per cycle, ie, once all the diagnostics are called, accumulated, and reduced.Working (ie, fast) build: https://github.com/CliMA/ClimaAtmos.jl/actions/runs/7759147548/job/21162571999