-
Notifications
You must be signed in to change notification settings - Fork 119
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[develop] Consolidate verification tasks using retrieve_data.py #864
[develop] Consolidate verification tasks using retrieve_data.py #864
Commits on Aug 14, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 9070063 - Browse repository at this point
Copy the full SHA 9070063View commit details -
Configuration menu - View commit details
-
Copy full SHA for faf5f72 - Browse repository at this point
Copy the full SHA faf5f72View commit details -
Convert exregional_get_obs_ccpa.sh to use retrieve_data.py...this ver…
…sion may have bugs, I am committing so I can test on another platform since my Jet "fairshare" is apparently off-the-charts low
Configuration menu - View commit details
-
Copy full SHA for 1a5da5d - Browse repository at this point
Copy the full SHA 1a5da5dView commit details -
Configuration menu - View commit details
-
Copy full SHA for b383c3f - Browse repository at this point
Copy the full SHA b383c3fView commit details -
Fix bug in retrieve_data.py: when the glob expression for files in th…
…e tarball contains a full path, we need to lstrip the leading "/" before globbing
Configuration menu - View commit details
-
Copy full SHA for 93a6ca4 - Browse repository at this point
Copy the full SHA 93a6ca4View commit details -
Consolidate all "get_obs" tasks for vx into single J-job/exscript.
All tasks now working for simple test case! Next up is to try all WE2E vx tests
Configuration menu - View commit details
-
Copy full SHA for 77fcc4c - Browse repository at this point
Copy the full SHA 77fcc4cView commit details -
Now that retrieve_data.py checks for the existence of the output dire…
…ctory, we need to create it prior to calling the script
Configuration menu - View commit details
-
Copy full SHA for 5230d7a - Browse repository at this point
Copy the full SHA 5230d7aView commit details -
Complete transition to new obs pulling script. Currently works to pull
obs for tests MET_verification, MET_verification_only_vx, and MET_ensemble_verification (modified to pull rather than use staged data) - Remove old j-jobs and exscripts - Cleanup and update mrms_pull_topofhour.py to be more pythonic - Move to obs dir prior to pulling data to avoid potential conflicts from temporarily extracted files from HPSS - Move many comments from config_defaults.yaml to more appropriate location in new data pulling/staging script; add more details about data quirks
Configuration menu - View commit details
-
Copy full SHA for 58ef36f - Browse repository at this point
Copy the full SHA 58ef36fView commit details -
Get rid of unneccessary directory structure requirement for obs. All …
…verification tests now pass when retrieving from HPSS!
Configuration menu - View commit details
-
Copy full SHA for b199379 - Browse repository at this point
Copy the full SHA b199379View commit details -
Configuration menu - View commit details
-
Copy full SHA for 906846b - Browse repository at this point
Copy the full SHA 906846bView commit details -
Updates to exregional_get_verif_obs.sh:
- Remove usage of "*_vrfy" functions due to issues mentioned in Issue 861 (ufs-community#861) - Remove extra "mkdir" statement where it shouldn't be for MRMS - Use correct variable (vhh_noZero) for checking valid hour for NDAS
Configuration menu - View commit details
-
Copy full SHA for e998ded - Browse repository at this point
Copy the full SHA e998dedView commit details -
Updates to verification tests:
- Remove stanzas eliminating the "task_get_obs*" tasks; these should now be run every time (if using staged data, they will check that the data exists) - Modify tasks RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP (MET_verification) and MET_ensemble_verification_only_vx_time_lag to retrieve obs data from HPSS - Fix MET_ensemble_verification_only_vx_time_lag test by correcting variable NET to NET_default (addresses issue 851) - Add new verification task custom_ESGgrid_Eastern_NA_snow_8km that will have winter weather verification tasks (for subsequent PR)
Configuration menu - View commit details
-
Copy full SHA for d4cb266 - Browse repository at this point
Copy the full SHA d4cb266View commit details -
- Don't retrieve obs data from HPSS for grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP; too many platforms use this test - Rename custom_ESGgrid_Eastern_NA_snow_8km to custom_ESGgrid_Great_Lakes_snow_8km; add to comprehensive and Jet coverage test - Add MET_ensemble_verification_only_vx to comprehensive for Cheyenne and Orion, as well as to coverage suite via Hera/GNU - Move MET_verification_only_vx coverage from Hera/GNU to Cheyenne/GNU - Add new mysterious "2020_CAD" test to all comprehensive lists, not just Orion (what's up with that?), sort alphabetically
Configuration menu - View commit details
-
Copy full SHA for b666879 - Browse repository at this point
Copy the full SHA b666879View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2882610 - Browse repository at this point
Copy the full SHA 2882610View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9cfbcd1 - Browse repository at this point
Copy the full SHA 9cfbcd1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 712207e - Browse repository at this point
Copy the full SHA 712207eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 48fb9d3 - Browse repository at this point
Copy the full SHA 48fb9d3View commit details -
Remove immediate quit when specified output path does not exist; this…
… didn't account for templating behavior
Configuration menu - View commit details
-
Copy full SHA for 2dbdf94 - Browse repository at this point
Copy the full SHA 2dbdf94View commit details -
Configuration menu - View commit details
-
Copy full SHA for ef161a2 - Browse repository at this point
Copy the full SHA ef161a2View commit details -
Roll new snowfall/NOHRSC verification into new data retrieval system
- Remove stand-alone j-jobs and exscripts for NOHRSC - Add NOHRSC_obs stanza to parm/data_locations.yml - Update documentation and comments to include NOHRSC obs - Add "ASNOW" (accumulated snow) verification to custom_ESGgrid_Great_Lakes_snow_8km -
Configuration menu - View commit details
-
Copy full SHA for 0fde1cb - Browse repository at this point
Copy the full SHA 0fde1cbView commit details -
Cheyenne can't automatically get AWS data, removing 2020_CAD test fro…
…m comprehensive.cheyenne
Configuration menu - View commit details
-
Copy full SHA for b39decf - Browse repository at this point
Copy the full SHA b39decfView commit details -
Configuration menu - View commit details
-
Copy full SHA for e35d28d - Browse repository at this point
Copy the full SHA e35d28dView commit details -
Move machine file definitions of *_OBS_DIRs to TEST_*_OBS_DIR for WE2…
…E testing purposes, similar to TEST_ALT_EXTRN_MDL_SYSBASEDIR_ICS
Configuration menu - View commit details
-
Copy full SHA for 624084d - Browse repository at this point
Copy the full SHA 624084dView commit details
Commits on Aug 15, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 6b97050 - Browse repository at this point
Copy the full SHA 6b97050View commit details