Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Distributed.jl workers #120

Merged
merged 3 commits into from
Dec 17, 2024
Merged

Add support for Distributed.jl workers #120

merged 3 commits into from
Dec 17, 2024

Conversation

nefrathenrici
Copy link
Member

@nefrathenrici nefrathenrici commented Dec 11, 2024

This PR adds basic support for calibrations parallelized with Distributed.jl's remote workers. This mostly copies SlurmManager functionality from ClusterManagers.jl with some simplifications. ClusterManagers has been unmaintained for a while now and this PR allows us to control and test the features we need for parallelizing calibration.

Core changes:

  • set_up_forward_model and run_forward_model are now combined into one function, forward_model
  • SlurmManager: subtype of ClusterManager
  • worker_calibrate and run_iteration: actual calibration functions
  • cleaned up methods for initialize, calibrate, added ekp_constructor

@nefrathenrici nefrathenrici force-pushed the ne/workers branch 2 times, most recently from 4318988 to d4dceb4 Compare December 16, 2024 18:38
Copy link

codecov bot commented Dec 16, 2024

Codecov Report

Attention: Patch coverage is 11.93182% with 155 lines in your changes missing coverage. Please review.

Project coverage is 17.19%. Comparing base (60be971) to head (0b0ef01).
Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
src/workers.jl 0.00% 110 Missing ⚠️
src/backends.jl 9.67% 28 Missing ⚠️
src/slurm.jl 0.00% 14 Missing ⚠️
src/ekp_interface.jl 88.88% 2 Missing ⚠️
src/ClimaCalibrate.jl 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##            main     #120       +/-   ##
==========================================
+ Coverage   0.00%   17.19%   +17.19%     
==========================================
  Files          7        9        +2     
  Lines        370      506      +136     
==========================================
+ Hits           0       87       +87     
- Misses       370      419       +49     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@nefrathenrici nefrathenrici force-pushed the ne/workers branch 2 times, most recently from 031ca24 to c3f6707 Compare December 17, 2024 00:53
@nefrathenrici nefrathenrici merged commit 9490d61 into main Dec 17, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant