ClimaUtilities.jl

ClimaUtilities is collection of general-purpose tools for CliMA packages.

ClimaUtilities.jl contains:

• ClimaArtifacts, a module that provides an MPI-safe (with ClimaComms) way to lazily download artifacts and to tag artifacts that are being accessed in a given simulation.
• SpaceVaryingInputs and TimeVaryingInputs to work with external input data.
• DataStructures with useful general purpose data structures.
• FileReaders, DataHandling, and Regridders to process input data and remap it onto the simulation grid.
• OnlineLogging to add output information while a simulation is running.
• OutputPathGenerator to prepare the output directory structure of a simulation.
• TimeManager to handle dates.

ClimaUtilities.jl Developer Guidelines

These guidelines aim to ensure consistent code quality, maintainability, and a smooth collaborative workflow for ClimaUtilities.jl. Please, read these guidelines even if you are familiar with other CliMA packages as there may be some differences.

A key design principle

ClimaUtilities.jl is meant to be a foundational package, so it should have near-zero-cost for features that are not explicitly requested. This means that it should not depend on any package outside of the standard library and ClimaComms and should have negligible import costs. To accomplish this, everything is implemented in Julia extensions.

Extensions are organized in the following way. The extensions defined in the Project.toml are defined in terms of the packages they require to be loaded. This avoids circular dependencies among extensions. Each of these extensions includes modules that match what is defined in src.

For example, ClimaUtilitiesClimaCoreNCDatasetsExt is loaded when ClimaCore and NCDatasets are loaded. Internally, ClimaUtilitiesClimaCoreNCDatasetsExt loads DataHandlingExt.jl, SpaceVaryingInputsExt.jl, and TimeVaryingInputsExt.jl. Each of them implements methods defined in src/DataHandling.jl, src/SpaceVaryingInputs.jl, and src/TimeVaryingInputs.jl respectively

Tests and environments

We prioritize well-tested code to guarantee ClimaUtilities.jl functions reliably. Here are some principles we follow:

Tests are collected in the test folder and are exclusively there

This means that:

• All the tests can be run with Pkg.test() or julia --project=test tests/runtests.jl;
• There are no special tests in the Buildkite pipeline;
• In fact, we use Buildkite only to run the tests with MPI/GPUs.

There are no checked-in Manifest.toml files

While checking in Manifest.toml files ensures reproducibility, it also introduces some nuisance, including:

• lot of git/repository noise just for "up deps";
• multiple environments that have to be managed;
• busywork to keep the manifests updated.

In this repository, we have four environments:

• project,
• documentation,
• test,
• buildkite.

The buildkite environment, in .buildkite, contains everything that is required to test ClimaUtilities.jl in its most general configuration (ie, including CUDA.jl and MPI.jl). As the name suggests, this is the (only) environment used by our Buildkite pipeline.

:note: Please, open an issue if you find workflow problems/friction with this system.

Running tests

There are two equivalent ways to run tests.

First, Start a Julia session in the ClimaUtilities directory:

julia --project

Enter Pkg mode by typing ]. This will change the prompt. Run test.

Equivalently, you can run

julia --project=test tests/runtests.jl

:note: Please, open an issue if you find workflow problems/friction with this system.

Code Formatting with JuliaFormatter.jl

One of the tests consists in checking that the code is uniformly formatted. We use JuliaFormatter.jl to achieve consistent formatting. Here's how to use it:

You can either install in your base environment with

julia -e 'using Pkg; Pkg.add("JuliaFormatter")'

or use it from within the TestEnv or the .buildkite environments (see previous section).

Then, you can format the package running:

using JuliaFormatter; format(".")

or just with format(".") if the package is already imported.

The rules for formatting are defined in the .JuliaFormatter.toml.

If you are used to formatting from the command line instead of the REPL, you can install JuliaFormatter in your base environment and call

julia -e 'using JuliaFormatter; format(".")'

You could also define a shell alias

alias julia_format_here="julia -e 'using JuliaFormatter; format(\".\")'"

:note: Please, open an issue if you find workflow problems/friction with this system.

Documentation

Documentation is generated with Documenter.jl. We strive to have complete and up-to-date information.

To generate documentation, run

julia --project=docs docs/make.jl

Please, update the documentation if you add new features or change the behavior of existing ones.

Pull Request (PR) and commits

Here's how to structure your contributions effectively:

• Descriptive Title: Briefly summarize the changes your PR introduces. Commit titles should preferably be under 50 characters, start with a capital latter, and use imperative verbs (e.g., "Remove superfluous function call").
• Detailed Description: Explain the purpose of your changes. Focus on the intent.
• Breaking Changes: If your PR introduces breaking changes, highlight them clearly in the description.

Your pull request can contain one or multiple commits. In either cases, it is important that each commit is atomic (meaning that each commit represents a single logical change).

Please, squash commits that represent a single logical change (e.g., do not have two commits when the second just fixes the first).

Credits

The Space and TimeVaryingInputs modules were initially developed in the context of ClimaLand, the TempestRegridder and TimeManager ones were initially developed in ClimaCoupler.