Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[develop] Consolidate verification tasks using retrieve_data.py (#864)
As described in Issue #850, the scripts for retrieving observation data for verification are quite convoluted and hard-to-maintain due to the constant changing of naming conventions on HPSS. We can leverage the functionality of the retrieve_data.py script to greatly simplify and consolidate this code, and that's what this PR achieves. In addition, many other bug fixes and improvements are included: * The j-Jobs and exscripts for the retrieving of CCPA, MRMS, NDAS, and NOHRSC data are consolidated into a single set of scripts. * Verification tests that use staged data on disk no longer need to manually deactivate "get_obs" tasks; now tasks check if staged data was specified and ensure it is all present * Obs locations (CCPA_OBS_DIR, MRMS_OBS_DIR, NDAS_OBS_DIR, and NOHRSC_OBS_DIR) no longer need a specific directory structure, though the file format remains the same * Major improvements to mrms_pull_topofhour.py; now has a main() function for importing, uses proper command-line arguments with argparse, uses f-strings to simplify print statements, and raises an exception if no observation file is found instead of failing silently * Fixes bug in retrieve_data.py where filenames in tarballs containing full paths are not properly extracted * Removes "choices" for --data_type argument in retrieve_data.py: the data type is customizable via the config file (which is also customizable), so we shouldn't hard-code arbitrary choices. Missing data types are gracefully handled later anyway. * Add all verification tests to a coverage suite * Add new test custom_ESGgrid_Great_Lakes_snow_8km that specifies a custom domain for a snowstorm in February 2023, re-using data that was staged for the AQM test. This is part of an effort to expand testing that will be completed in a subsequent PR, including explicit tests for winter precipitation verification. * Add cold-air-damming test to all comprehensive test lists, not just orion. * Test MET_ensemble_verification_only_vx_time_lag now works on most platforms, not just hard-coded to Hera * Comments related to verification obs in config_defaults.yaml are consolidated and relevant information moved to new consolidated obs-pulling script
- Loading branch information