Skip to content

Commit

Permalink
[develop] Consolidate verification tasks using retrieve_data.py (#864)
Browse files Browse the repository at this point in the history
As described in Issue #850, the scripts for retrieving observation data for verification are quite convoluted and hard-to-maintain due to the constant changing of naming conventions on HPSS. We can leverage the functionality of the retrieve_data.py script to greatly simplify and consolidate this code, and that's what this PR achieves.

In addition, many other bug fixes and improvements are included:
* The j-Jobs and exscripts for the retrieving of CCPA, MRMS, NDAS, and NOHRSC data are consolidated into a single set of scripts.
* Verification tests that use staged data on disk no longer need to manually deactivate "get_obs" tasks; now tasks check if staged data was specified and ensure it is all present
* Obs locations (CCPA_OBS_DIR, MRMS_OBS_DIR, NDAS_OBS_DIR, and NOHRSC_OBS_DIR) no longer need a specific directory structure, though the file format remains the same
* Major improvements to mrms_pull_topofhour.py; now has a main() function for importing, uses proper command-line arguments with argparse, uses f-strings to simplify print statements, and raises an exception if no observation file is found instead of failing silently
* Fixes bug in retrieve_data.py where filenames in tarballs containing full paths are not properly extracted
* Removes "choices" for --data_type argument in retrieve_data.py: the data type is customizable via the config file (which is also customizable), so we shouldn't hard-code arbitrary choices. Missing data types are gracefully handled later anyway.
* Add all verification tests to a coverage suite
* Add new test custom_ESGgrid_Great_Lakes_snow_8km that specifies a custom domain for a snowstorm in February 2023, re-using data that was staged for the AQM test. This is part of an effort to expand testing that will be completed in a subsequent PR, including explicit tests for winter precipitation verification.
* Add cold-air-damming test to all comprehensive test lists, not just orion.
* Test MET_ensemble_verification_only_vx_time_lag now works on most platforms, not just hard-coded to Hera
* Comments related to verification obs in config_defaults.yaml are consolidated and relevant information moved to new consolidated obs-pulling script
  • Loading branch information
mkavulich authored Aug 15, 2023
1 parent d056a84 commit 3dee70b
Show file tree
Hide file tree
Showing 38 changed files with 766 additions and 1,315 deletions.
91 changes: 0 additions & 91 deletions jobs/JREGIONAL_GET_OBS_CCPA

This file was deleted.

92 changes: 0 additions & 92 deletions jobs/JREGIONAL_GET_OBS_MRMS

This file was deleted.

91 changes: 0 additions & 91 deletions jobs/JREGIONAL_GET_OBS_NOHRSC

This file was deleted.

10 changes: 5 additions & 5 deletions jobs/JREGIONAL_GET_OBS_NDAS → jobs/JREGIONAL_GET_VERIF_OBS
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#!/bin/bash
#!/usr/bin/env bash

#
#-----------------------------------------------------------------------
#
# This script pulls NDAS observation data for comparison to the model.
# This script checks, pulls, and stages observation data for model verification.
#
#-----------------------------------------------------------------------
#
Expand Down Expand Up @@ -51,8 +51,8 @@ print_info_msg "
Entering script: \"${scrfunc_fn}\"
In directory: \"${scrfunc_dir}\"
This is the J-job script for the task that pulls NDAS observation data
for verification purposes.
This is the J-job script for the task that checks, pulls, and stages
observation data for verification purposes.
========================================================================"

#
Expand All @@ -64,7 +64,7 @@ for verification purposes.
#
#-----------------------------------------------------------------------
#
$SCRIPTSdir/exregional_get_obs_ndas.sh || \
$SCRIPTSdir/exregional_get_verif_obs.sh || \
print_err_msg_exit "\
Call to ex-script corresponding to J-job \"${scrfunc_fn}\" failed."
#
Expand Down
Loading

0 comments on commit 3dee70b

Please sign in to comment.