End-to-end calibration test occasionally times out

[charleskawczynski]

Sometimes our end-to-end calibration test is timing out, preventing PRs from being merged.

Some examples:

• https://buildkite.com/clima/climaatmos-ci/builds/20742
• https://buildkite.com/clima/climaatmos-ci/builds/20760

@costachris, any idea why this might be timing out? The SYPD is pretty huge for that job.

[szy21]

I think it times out when the cluster is busy, because it is waiting for submitting new jobs for each iteration.

[Sbozzolo]

I think it times out when the cluster is busy, because it is waiting for submitting new jobs for each iteration.

Yes, that's the problem.

We can decrease the number of ensable members and iterations.

[sriharshakandala]

I am experiencing this problem here: https://buildkite.com/clima/climaatmos-ci/builds/21125#0192969c-8c24-4494-8607-7ff128db03d5

[nefrathenrici]

Closed by #3386, since the job has been set to soft-fail and the slurm timeout limit has been lowered.

[nefrathenrici]

I also opened an issue in ClimaCalibrate to remove the dependency on the slurm database
CliMA/ClimaCalibrate.jl#115