Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch Aerosol DA to use JCB and Jedi class #3125

Merged
merged 134 commits into from
Dec 14, 2024

Conversation

CoryMartin-NOAA
Copy link
Contributor

@CoryMartin-NOAA CoryMartin-NOAA commented Nov 26, 2024

Description

This PR switches to using JCB for aerosol DA YAML generation + makes use of the Jedi class refactoring already done by @DavidNew-NOAA for the atmospheric JEDI-based DA.

Resolves #3035

Note this PR is in draft for 2 reasons. 1) Need associated GDASApp PR to be opened and merged first and 2) requires some fix file updates. This will be coordinated with an issue and linked to this PR once @andytangborn is satisfied with a first-attempt at the fix files.

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? YES

How has this been tested?

  • Clone and build on Hera
  • Cycled test on Hera

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

RussTreadon-NOAA and others added 30 commits August 28, 2024 17:01
@CoryMartin-NOAA CoryMartin-NOAA marked this pull request as ready for review December 11, 2024 21:40
@CoryMartin-NOAA
Copy link
Contributor Author

marking this as ready to review as it will be ready once #3160 is resolved

aerorahul
aerorahul previously approved these changes Dec 11, 2024
DavidNew-NOAA
DavidNew-NOAA previously approved these changes Dec 11, 2024
@RussTreadon-NOAA
Copy link
Contributor

Given approvals from @DavidNew-NOAA and @aerorahul , I will set up and run g-w CI for this PR on various machines once new fix files have been staged.

@WalterKolczynski-NOAA and @KateFriedman-NOAA : g-w CI for this PR can not begin until the required fix files have been added to EIB staged fix file directories. See g-w issue #3160 for details.

Note: We have no WCOSS2 dev access 12/16 through 12/20. If we are unable to run g-w CI for this PR before 12/16, we will need to wait until 12/21 or later for WCOSS2 testing.

@WalterKolczynski-NOAA WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Dec 13, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Dec 13, 2024
@RussTreadon-NOAA
Copy link
Contributor

Hera C96C48_hybatmaerosnowDA failure

g-w stream C96C48_hybatmaerosnowDA job enkfgdas_earc00 fails for 20211220 18Z due to

FileNotFoundError: FATAL ERROR: Required file, directory, or glob logs/2021122018/enkfgdas_echgres.log not found!
+ JGDAS_ENKF_ARCHIVE[1]: postamble JGDAS_ENKF_ARCHIVE 1734113838 1

File $HOMEgfs/parm/archive/enkf.yaml.j2 lists _echgres.log as a required archive file. However, the xml does not list echgres as a job dependency for metatask enkfgdas_eamn. The only dependency is metatask enkfgdas_epmn.

Since earc00 is required to archive echgres.log, we need to add task echgres as a job dependency for metatask enkfgdas_eamn

@RussTreadon-NOAA
Copy link
Contributor

RussTreadon-NOAA commented Dec 13, 2024

Hera C96C48_hybatmaerosnowDA failure follow up

The following change to workflow/rocoto/gfs_tasks.py adds enkfgdas_echgres as a dependency to metatask enkfgdas_eamn

@@ -2898,6 +2898,8 @@ class GFSTasks(Tasks):
         deps = []
         dep_dict = {'type': 'metatask', 'name': f'{self.run}_epmn'}
         deps.append(rocoto.add_dependency(dep_dict))
+        dep_dict = {'type': 'task', 'name': 'enkfgdas_echgres'}
+        deps.append(rocoto.add_dependency(dep_dict))
         dependencies = rocoto.create_dependency(dep=deps)

         earcenvars = self.envars.copy()

When the revised gfs_tasks.py is used to generate the xml, we get the following dependencies for metatask enkfgdas_eamn

                <dependency>
                        <metataskdep metatask="enkfgdas_epmn"/>
                        <taskdep task="enkfgdas_echgres"/>
                </dependency>

g-w issue #3165 open to report this bug.

@emcbot emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Dec 13, 2024
@emcbot
Copy link

emcbot commented Dec 13, 2024

CI Passed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/3125


Experiment C48_ATM_963ba4f9 Completed 2 Cycles: *SUCCESS* at Fri Dec 13 20:48:48 UTC 2024
Experiment C48mx500_3DVarAOWCDA_963ba4f9 Completed 2 Cycles: *SUCCESS* at Fri Dec 13 20:48:50 UTC 2024
Experiment C48mx500_hybAOWCDA_963ba4f9 Completed 2 Cycles: *SUCCESS* at Fri Dec 13 21:01:06 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_963ba4f9 Completed 1 Cycles: *SUCCESS* at Fri Dec 13 21:01:31 UTC 2024
Experiment C96_atm3DVar_963ba4f9 Completed 3 Cycles: *SUCCESS* at Fri Dec 13 22:07:59 UTC 2024
Experiment C96C48_hybatmDA_963ba4f9 Completed 3 Cycles: *SUCCESS* at Fri Dec 13 22:08:14 UTC 2024
Experiment C96C48_hybatmaerosnowDA_963ba4f9 Completed 3 Cycles: *SUCCESS* at Fri Dec 13 22:20:22 UTC 2024
Experiment C96C48_ufs_hybatmDA_963ba4f9 Completed 3 Cycles: *SUCCESS* at Fri Dec 13 22:56:58 UTC 2024
Experiment C48_S2SW_963ba4f9 Completed 2 Cycles: *SUCCESS* at Fri Dec 13 23:09:53 UTC 2024
Experiment C48_S2SWA_gefs_963ba4f9 Completed 1 Cycles: *SUCCESS* at Fri Dec 13 23:34:20 UTC 2024

@RussTreadon-NOAA
Copy link
Contributor

g-w CI

Install CoryMartin-NOAA:feature/aero-jcb at caa4500. Run g-w CI on WCOSS2 (Dogwood), Hera, and Orion

Dogwood

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_ATM_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Dec 13 2024 15:37:02    Dec 13 2024 16:50:11
202103231800        Done    Dec 13 2024 15:37:02    Dec 13 2024 17:00:10

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_3DVarAOWCDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Dec 13 2024 15:37:04    Dec 13 2024 16:00:16
202103250000        Done    Dec 13 2024 15:37:04    Dec 14 2024 00:39:57

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_hybAOWCDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Dec 13 2024 15:37:06    Dec 13 2024 16:05:30
202103250000        Done    Dec 13 2024 15:37:06    Dec 14 2024 00:40:00   

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SW_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Dec 13 2024 15:37:12    Dec 13 2024 17:25:11
202103231800        Done    Dec 13 2024 15:37:12    Dec 13 2024 17:40:19

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Dec 13 2024 15:37:14    Dec 13 2024 16:05:36
202112210000        Done    Dec 13 2024 15:37:14    Dec 13 2024 18:00:18
202112210600        Done    Dec 13 2024 15:37:14    Dec 13 2024 17:42:33

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Dec 13 2024 15:37:16    Dec 13 2024 18:22:54
202112201800        Done    Dec 13 2024 15:37:16    Dec 13 2024 20:35:31
202112210000        Done    Dec 13 2024 15:37:16    Dec 13 2024 20:40:18

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Dec 13 2024 15:37:19    Dec 13 2024 16:05:44
202112210000        Done    Dec 13 2024 15:37:19    Dec 13 2024 18:00:23
202112210600        Done    Dec 13 2024 15:37:19    Dec 13 2024 17:44:29

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_ufs_hybatmDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Dec 13 2024 15:37:21    Dec 13 2024 16:05:47
202402240000        Done    Dec 13 2024 15:37:21    Dec 13 2024 18:48:31
202402240600        Done    Dec 13 2024 15:37:21    Dec 13 2024 18:44:52

Hera

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48_ATM_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    Dec 13 2024 12:39:40    Dec 13 2024 13:55:13
202103231800        Done    Dec 13 2024 12:39:40    Dec 13 2024 15:45:16

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48_S2SW_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    Dec 13 2024 12:39:42    Dec 13 2024 16:10:22
202103231800        Done    Dec 13 2024 12:39:42    Dec 13 2024 16:33:08

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48mx500_3DVarAOWCDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    Dec 13 2024 12:39:48    Dec 13 2024 12:53:55
202103250000        Done    Dec 13 2024 12:39:48    Dec 13 2024 13:55:22

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48mx500_hybAOWCDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    Dec 13 2024 12:39:49    Dec 13 2024 16:33:16
202103250000        Done    Dec 13 2024 12:39:49    Dec 13 2024 18:26:21

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96C48_hybatmDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Dec 13 2024 12:39:51    Dec 13 2024 12:53:58
202112210000        Done    Dec 13 2024 12:39:51    Dec 13 2024 16:35:24
202112210600        Done    Dec 13 2024 12:39:51    Dec 13 2024 16:33:18

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96C48_hybatmaerosnowDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201200        Done    Dec 13 2024 12:39:53    Dec 13 2024 18:16:36
202112201800        Done    Dec 13 2024 12:39:53    Dec 13 2024 23:29:20
202112210000        Done    Dec 13 2024 12:39:53    Dec 13 2024 21:00:22

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96C48_ufs_hybatmDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Dec 13 2024 12:39:54    Dec 13 2024 12:54:01
202402240000        Done    Dec 13 2024 12:39:54    Dec 13 2024 16:40:32
202402240600        Done    Dec 13 2024 12:39:54    Dec 13 2024 17:50:30

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96_atm3DVar_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Dec 13 2024 12:39:58    Dec 13 2024 12:54:05
202112210000        Done    Dec 13 2024 12:39:58    Dec 13 2024 16:35:30
202112210600        Done    Dec 13 2024 12:39:58    Dec 13 2024 16:15:36

Orion

/work2/noaa/stmp/rtreadon/EXPDIR/C48_ATM_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    Dec 13 2024 12:39:30    Dec 13 2024 13:50:05
202103231800        Done    Dec 13 2024 12:39:30    Dec 13 2024 14:10:05

/work2/noaa/stmp/rtreadon/EXPDIR/C48mx500_3DVarAOWCDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    Dec 13 2024 12:39:31    Dec 13 2024 13:00:10
202103250000        Done    Dec 13 2024 12:39:31    Dec 13 2024 14:15:06

/work2/noaa/stmp/rtreadon/EXPDIR/C48mx500_hybAOWCDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    Dec 13 2024 12:39:33    Dec 13 2024 13:00:13
202103250000        Done    Dec 13 2024 12:39:33    Dec 13 2024 14:20:09

/work2/noaa/stmp/rtreadon/EXPDIR/C48_S2SW_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    Dec 13 2024 12:39:45    Dec 13 2024 16:20:11
202103231800        Done    Dec 13 2024 12:39:45    Dec 13 2024 16:40:11

/work2/noaa/stmp/rtreadon/EXPDIR/C96_atm3DVar_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Dec 13 2024 12:39:47    Dec 13 2024 13:00:23
202112210000        Done    Dec 13 2024 12:39:47    Dec 13 2024 15:35:12
202112210600        Done    Dec 13 2024 12:39:47    Dec 13 2024 15:15:15

/work2/noaa/stmp/rtreadon/EXPDIR/C96C48_hybatmaerosnowDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201200        Done    Dec 13 2024 12:39:49    Dec 13 2024 18:15:25
202112201800        Done    Dec 13 2024 12:39:49    Dec 13 2024 20:19:16
202112210000        Done    Dec 13 2024 12:39:49    Dec 13 2024 20:19:16

/work2/noaa/stmp/rtreadon/EXPDIR/C96C48_hybatmDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    Dec 13 2024 12:39:51    Dec 13 2024 13:00:28
202112210000        Done    Dec 13 2024 12:39:51    Dec 13 2024 15:30:17
202112210600        Done    Dec 13 2024 12:39:51    Dec 13 2024 15:30:17

/work2/noaa/stmp/rtreadon/EXPDIR/C96C48_ufs_hybatmDA_pr3125
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Dec 13 2024 12:39:53    Dec 13 2024 13:00:31
202402240000        Done    Dec 13 2024 12:39:53    Dec 13 2024 16:30:19
202402240600        Done    Dec 13 2024 12:39:53    Dec 13 2024 17:05:18

All CI streams run on Dogwood, Hera, and Orion successfully completed all jobs with the following caveats

  1. build Dogwood GDASApp using spack-stack/1.6.0 - see GDASApp issues #1331 and #1336
  2. locally fix WCOSS2.env bug reported in g-w issue #3158
  3. populate Dogwood GDA with `gdas.20210325/00 dump files - see g-w issue #3159
  4. rerun failed enkfgdas_earc00 after enkfgdas_echgres completed - see g-w issue #3165

@RussTreadon-NOAA RussTreadon-NOAA self-requested a review December 14, 2024 00:55
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

g-w CI run on Dogwood, Hera, and Orion with acceptable behavior.

Approve.

@RussTreadon-NOAA
Copy link
Contributor

@WalterKolczynski-NOAA : You do not need to set up g-w CI on WCOSS2. I ran on Dogwood (see results above). g-w CI has also been run on Orion.

Do we need to run g-w CI on Hercules?

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit bef3e0b into NOAA-EMC:develop Dec 14, 2024
10 of 11 checks passed
sbanihash pushed a commit to sbanihash/global-workflow that referenced this pull request Dec 14, 2024
This PR switches to using JCB for aerosol DA YAML generation + makes use
of the Jedi class refactoring already done by @DavidNew-NOAA for the
atmospheric JEDI-based DA.

Resolves NOAA-EMC#3035

Note this PR is in draft for 2 reasons. 1) Need associated GDASApp PR to
be opened and merged first and 2) requires some fix file updates. This
will be coordinated with an issue and linked to this PR once
@andytangborn is satisfied with a first-attempt at the fix files.
---------

Co-authored-by: RussTreadon-NOAA <Russ.Treadon@noaa.gov>
Co-authored-by: DavidNew-NOAA <david.new@noaa.gov>
@RussTreadon-NOAA
Copy link
Contributor

Thank you @WalterKolczynski-NOAA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Switch to JCB for aerosol DA jobs
6 participants