upgrade to Julia 1.9

[simonbyrne]

Purpose

Upgrade our examples Manifest and pipeline to Julia 1.9.

To-do

• tag a new release of ClimaCore

Content

---

• I have read and checked the items on the review checklist.

[simonbyrne]

This seems to speed up initialization in some cases

• gravity wave: 4min => 2min
• density current 17 min => 12 min

[simonbyrne]

created longrun build https://buildkite.com/clima/climaatmos-longruns/builds/1149

[simonbyrne]

Latest long runs: https://buildkite.com/clima/climaatmos-longruns/builds/1151

[simonbyrne]

bors r+

[bors]

Build failed:

• buildkite/climaatmos-ci

[szy21]

bors r+

[bors]

Build failed:

• buildkite/climaatmos-ci

[simonbyrne]

bors r+

[bors]

Canceled.

[simonbyrne]

bors r+

[bors]

Build failed:

• buildkite/climaatmos-ci

[szy21]

Two performance job failed: https://buildkite.com/clima/climaatmos-ci/builds/12348#018a28ac-e081-4eaf-9b3c-fb6e13140f7e

[simonbyrne]

bors r+

[bors]

Build failed (retrying...):

• buildkite/climaatmos-ci

[Sbozzolo]

@Sbozzolo This is the PR

Yes, I am looking into it.

[Sbozzolo]

I tracked it down to the new flag we added slurm_mem_per_cpu: 8G.

This minimal build starts.

I also verified that manually submitting a script with that option works (after you fiddle around with MPI/HDF5).

#!/bin/bash

#SBATCH --ntasks=2
#SBATCH --mem-per-cpu=8G
#SBATCH --job-name=MyTest
#SBATCH --time=0:23:00

module purge
module load julia/1.9.2 cuda/11.8 ucx/1.14.1_cuda-11.8 openmpi/4.1.5_cuda-11.8 hdf5/1.12.2-ompi415 nsight-systems/2023.2.1

export CLIMACORE_DISTRIBUTED="MPI"
export CLIMACOMMS_CONTEXT="MPI"

cd ClimaAtmos.jl

julia -e 'using Pkg; Pkg.add("MPIPreferences"); using MPIPreferences; use_system_binary()'
julia --project=examples -e 'using Pkg; Pkg.build("MPI")'
julia --project=examples -e 'using Pkg; Pkg.build("HDF5")'
julia --project=examples -e 'using Pkg; Pkg.API.precompile()'
srun julia --color=yes --project=examples examples/hybrid/driver.jl --config_file config/mpi_configs/mpi_make_restart.yml

So, I am guessing that buildkite-slurm bridge is not properly translating the option.

EDIT: I also tested that the job starts ~immediately when I submit it with walltime of 6 hours.

[szy21]

Great, thanks! If all other ci jobs work, I am happy to get this in and remove the slurm_mem_per_cpu flag. We can also wait until Simon comes back - up to you.

[simonbyrne]

Ah, I think it's because we set --mem (globally for all jobs), and that must be incompatible with --mem-per-cpu

[Sbozzolo]

Good point, I was running into the same problem when writing my own SLURM script. It has to be it!

…

On Tue, Aug 29, 2023 at 9:56 AM Simon Byrne ***@***.***> wrote: Ah, I think it's because we set --mem (globally for all jobs), and that must be incompatible with --mem-per-cpu — Reply to this email directly, view it on GitHub <#1998 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACF6E7IKHGLHMI7LC46XMJTXXYNL5ANCNFSM6AAAAAA3UTNPAI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[simonbyrne]

bors r+

[bors]

Build failed:

• buildkite/climaatmos-ci

[simonbyrne]

bors r+

[simonbyrne]

bors r-

[bors]

Canceled.

[simonbyrne]

bors r+

[bors]

Build failed (retrying...):

• buildkite/climaatmos-ci

[bors]

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

• buildkite/climaatmos-ci
• docbuild
• format

[simonbyrne]

Fixes #2001