Merge pull request #581 from dbekaert/dev

v0.4.3

.github/workflows/build.yml

@@ -12,13 +12,13 @@ on:
 
 jobs:
   call-version-info-workflow:
-    uses: ASFHyP3/actions/.github/workflows/reusable-version-info.yml@v0.7.0
+    uses: ASFHyP3/actions/.github/workflows/reusable-version-info.yml@v0.8.1
     with:
       python_version: '3.10'
 
   call-docker-ghcr-workflow:
     needs: call-version-info-workflow
-    uses: ASFHyP3/actions/.github/workflows/reusable-docker-ghcr.yml@v0.7.0
+    uses: ASFHyP3/actions/.github/workflows/reusable-docker-ghcr.yml@v0.8.1
     with:
       version_tag: ${{ needs.call-version-info-workflow.outputs.version_tag }}
       release_branch: main

.github/workflows/changelog.yml

@@ -13,6 +13,6 @@ on:
 
 jobs:
   call-changelog-check-workflow:
-    uses: ASFHyP3/actions/.github/workflows/reusable-changelog-check.yml@v0.7.0
+    uses: ASFHyP3/actions/.github/workflows/reusable-changelog-check.yml@v0.8.1
     secrets:
       USER_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/deploy-docs.yml

@@ -14,7 +14,7 @@ jobs:
         with:
           fetch-depth: 0
 
-      - uses: mamba-org/provision-with-micromamba@v15
+      - uses: mamba-org/provision-with-micromamba@v16
         with:
           extra-specs: |
             python=3.10

.github/workflows/labeled-pr.yml

@@ -12,4 +12,4 @@ on:
 
 jobs:
   call-labeled-pr-check-workflow:
-    uses: ASFHyP3/actions/.github/workflows/reusable-labeled-pr-check.yml@v0.7.0
+    uses: ASFHyP3/actions/.github/workflows/reusable-labeled-pr-check.yml@v0.8.1

.github/workflows/release.yml

@@ -7,7 +7,7 @@ on:
 
 jobs:
   call-release-workflow:
-    uses: ASFHyP3/actions/.github/workflows/reusable-release.yml@v0.7.0
+    uses: ASFHyP3/actions/.github/workflows/reusable-release.yml@v0.8.1
     with:
       release_prefix: RAiDER
       develop_branch: dev

.github/workflows/tag.yml

@@ -7,7 +7,7 @@ on:
 
 jobs:
   call-bump-version-workflow:
-    uses: ASFHyP3/actions/.github/workflows/reusable-bump-version.yml@v0.7.0
+    uses: ASFHyP3/actions/.github/workflows/reusable-bump-version.yml@v0.8.1
     with:
       user: dbekaert
       email: bekaertdavid@gmail.com

.gitignore

@@ -13,7 +13,6 @@ build
 CMake*
 Makefile
 dummy*
-weather_files
 test/data
 *.log
 *.pdf

CHANGELOG.md

@@ -6,16 +6,90 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
 and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-+ Add assert statement to raise error if the delay cube for each SAR date in a GUNW IFG is not written 
+## [0.4.3]
++ Prevent ray tracing integration from occuring at exactly top of weather model
++ Properly expose z_ref (max integration height) parameter, and dont allow higher than weather model
++ Min version for sentineleof for obtaining restituted orbits.
++ Rename datetime columns and convert from strings for GNSS workflow
++ Use native model levels in HRRR which extend up to 2 hPa as opposed to 50 hPa in pressure levels
++ Update tests to account for different interpolation scheme
++ Dont error out when the weather model contains nan values (HRRR)
++ Fix bug in fillna3D for NaNs at elevations higher than present in the weather model
++ write delays even if they contain nans
++ check that the aoi is contained within HRRR extent
++ streamline some unit tests to remove downloading
++ move the ray building out of the _build_cube_ray and into its own function for cleaner testing
++ update the tests to use the new build_ray function
++ If the processed weather file exists use it; otherwise check if raw exists and covers study area; otherwise download new
++ Update the integration height for raytracing from 50 km to 80 km
++ Reinstate test 3 (slant proj and ray trace), remove unused calls with ZREF
++ Add buffer to W/E for ERA5
++ refactor imports to allow for a cleaner raider-base
++ Add buffer to HRES when downloading as with the other models
++ Refactor to pass a weather file directly to fetch
++ Update staged weather models to reflect update to aligned grid
++ Correctly pass buffered bounds when aligning grid
++ Check the valid bounds prior to starting and use HRRR-AK if its correct so that rounding times to obtain data at are correctly fed to Herbie
++ Update test_intersect to already existing weather model files 
++ Replace the real weather model files used for the synthetic test with the correct ones (bounding box changed slightly)
++ Update test_scenerio_1 to match golden data by selecting a grid by lat/lon rather than indices
++ Adjust the buffering to account for grid spacing
++ Update ERA5 model coordinates to reflect changes in support of HRRR
++ Re-work the HRRR weather model to use herbie (https://github.com/blaylockbk/Herbie) for weather model access. HRRR conus and Alaska validation periods are respectively 2016-7-15 and 2018-7-13 onwards.
++ minor bug fixes and unit test updates
++ add log file write location as a top-level command-line option and within Python as a user-specified option
++ account for grid spacing impact on bounding box before downloading weather model
++ update the GUNW test to account for change in grid spacing on affine transform
++ add CLI for the old processDelayFiles script and rename to raiderCombine
++ Fix gridding bug in accessing HRRR-AK 
++ misc clean-up
++ Specify unbuffered python output in the docker entrypoint script using `python -um RAiDER.cli ...` whose `__main__.py` is the desired entrypoint.
++ For the GUNW workflow uses azimuth time interpolation using ISCE3 geo2rdr (see [here](https://github.com/ACCESS-Cloud-Based-InSAR/s1_azimuth_time_grid)).
+    - Updates `interpolate_time` options to: `'none'` (formerly `False`), `'center_time'` (formerly `True` and `default`), and `azimuth_time_grid` (not implemented previously)
++ Series of bug-fixes/compatibility updates with stats class: 
+    + Inconsistent definition of index IDs, which leads to key errors as so when querying the grid space for valid data points
+    + Turn off default behavior of plotting minor ticks on colorbars, which translates to unreadable plots especially when scientific notation is involved
+    + Assign valid geotrans to output tif files used for replotting/dedup.
+    + Properly load existing grids for replotting runs. Before the program crashed as single bands were incorrectly being read as cubes.
+    + Update in pandas not backwards compatible with original conditional logic. Specifically, conditions like `(not self.df['Date'].dt.is_leap_year)` replaced with `(self.df['Date'].dt.is_leap_year is False)`
++ add unit tests for the hydro and two pieces of wet equation
++ bump  bottom/top height of user requested levels by ~1mm during ray tracing to ensure interpolation works
++ ensure directories for storage are written
++ fix bug in writing delays for station files
++ Force lat/lon/hgt to float32 so that they line up correctly in stitching
++ Add two stage buffer;
+    + first pad user bounding box such that a 3D cube is generated that at min covers user area of interest.
+    + then if ray tracing is used, pad the downloaded model in look direction. Assumes look angle is fixed increases with latitude.
++ Update and convert user given AOI to weather model projection (except for HRRR)
++ Clean up error messagse, skip date if temporal interpolation fails
++ Update valid range for ERA5 (current date - 3 months) & ERA5T
++ Temporal interpolation of delays if the requested datetime is more than _THRESHOLD_SECONDS away from the closest weather model available time and `interpolate_time = True` (default behavior)
++ Add assert statement to raise error if the delay cube for each SAR date in a GUNW IFG is not written
++ Verify some constants / equations and remove the comments questioning them
++ Relocate the time resolution of wmodels to one spot
++ Skip test_scenario_3 until a new golden dataset is created
++ Update environment with scipy version minimum and requiring pybind11 (the latter for Apple ARM64 with rosetta2)
++ For GUNW entrypoint and associated workflow, update json metadata when bucket argument is provided to include `weather_model` key with value being a list.
+   - A GUNW is downloaded from a bucket prefix with netcdf, json, and png whose name is the <GUNW_ID>. We download the json and update it to be consistent with ASF DAAC schema (link: https://github.com/asfadmin/grfn-ingest/blob/test/verify/src/metadata_schema.json)
++ For the GUNW workflow:
+   - Updated GUNW workflow to expose input arguments (usually passed through command line options) within the python function for testing
+   - Include integration test of HRRR for GUNW workflow
+   - Test the json write (do not test s3 upload/download) in that it conforms to the DAAC ingest schema correctly - we add a weather model field to the metadata in this workflow
+   - Removed comments in GUNW test suite that were left during previous development
+   - If a bucket is provided and the GUNWs reference or secondary scenes are not in the valid range, we do nothing - this is to ensure that GUNWs can still be delivered to the DAAC without painful operator (i.e. person submitting to the hyp3 API) book-keeping
 
 ## [0.4.2]
 
 ### New/Updated Features
++ `prepFromGUNW` reads the date/time from the SLCs rather than the GUNW filename
 + `calcDelaysGUNW` allows processing with any supported weather model as listed in [`RAiDER.models.allowed.ALLOWED_MODELS`](https://github.com/dbekaert/RAiDER/blob/dev/tools/RAiDER/models/allowed.py).
-+ Removed NCMR removed from supported model list till re-tested 
++ Removed NCMR removed from supported model list till re-tested
 + `credentials` looks for weather model API credentials RC_file hidden file, and creates it if it does not exists
 + Isolate ISCE3 imports to only those functions that need it.
 + Small bugfixes and updates to docstrings
++ Only orbit file is used (even if multiple specified) to minimize errors and ensure consistency over region
++ GUNW packaging is restructed to store SLC (ref and sec) wet and tropo delays rather than the differential
++ padding made consistent throughout and default arguments reduced (manually update in test_losreader)
 
 ## [0.4.1]
 
@@ -33,34 +107,34 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [0.4.0]
 
-Adding of new GUNW support to RAiDER. This is an interface delivery allowing for subsequent integration into HYP3 (input/output parsing is not expected to change; computed data is not yet verified). 
+Adding of new GUNW support to RAiDER. This is an interface delivery allowing for subsequent integration into HYP3 (input/output parsing is not expected to change; computed data is not yet verified).
 
 ### New/Updated Features
 + Working GUNW entry point in workflow for raider.py
 + Ability to parse a GUNW to workflows from which all required RAiDER information is extracted (e.g. dates, UTC, orbit, bbox, look direction, wavelength) with an option to specify weather model (those already supported by RAiDER) and ability to squeeze in the derived output into the original GUNW product.
-+ Delays for GUNW are calculated in RAiDER using the ray-tracing option specifying bbox (GUNW driven), a hardcoded lateral posting (0.05º for HRRR and 0.1º for others),  fixed vertical height levels, using an different orbit file for  secondary and master. 
++ Delays for GUNW are calculated in RAiDER using the ray-tracing option specifying bbox (GUNW driven), a hardcoded lateral posting (0.05º for HRRR and 0.1º for others),  fixed vertical height levels, using an different orbit file for  secondary and master.
      - The hard-coded heights and posting will be refined per model and to ensure stitching abilities in ARIA-tools.
-     - The orbit should be refined to not change between secondary and reference to avoid issues. See https://github.com/dbekaert/RAiDER/discussions/435#discussioncomment-4392665 
-+ Bug fix for raider.py "date" input argument when multiple dates are requested (i.e. support of requesting two dates or two dates with a sampling).  
+     - The orbit should be refined to not change between secondary and reference to avoid issues. See https://github.com/dbekaert/RAiDER/discussions/435#discussioncomment-4392665
++ Bug fix for raider.py "date" input argument when multiple dates are requested (i.e. support of requesting two dates or two dates with a sampling).
 + Add unit test for date input argument checking (single day, two dates, two dates with samples)
 + Write the diagnostic weather model files to the 'output_directory' rather than PWD
 + Fix for incorrectly written hard-cored projection embedded in the computed output data
 + Allow for multiple orbits files/dates to be used for slant:projection
-+ correctly pass llh to lla_to_ecef function for slant:projection 
++ correctly pass llh to lla_to_ecef function for slant:projection
     ++ verified this doesnt change anything
 + removed deprecated ray projection functionality
 + added 1º buffer for zenith and projected (already done for ray tracing)
 + differential delay is rounded to model-dependent nearest hour
-+ version 1c hardcoded into the updated GUNW 
++ version 1c hardcoded into the updated GUNW
 
 ### Added dependencies for:
-+ sentinelof: used to fetch the orbit for GUNW 
++ sentinelof: used to fetch the orbit for GUNW
 + rioxarray: used for reading rasters with xarray
 
 ### Not implemented / supported in this release### Not implemented / supported in this release
-+ no temporal interpolation 
++ no temporal interpolation
 + no refined model specific hardcoded spacing and heights
-+ no ability for single orbit Interferometric calculation 
++ no ability for single orbit Interferometric calculation
 + no verification of results
 
 ## [0.3.1]

README.md

@@ -115,6 +115,7 @@ For more details on installing from source see [here](https://github.com/dbekaer
 * David Bekaert
 * Jeremy Maurer
 * Raymond Hogenson
+* Brett Buzzanga
 * Piyush Agram _(Descartes Labs)_
 * Yang Lei
 * Rohan Weeden

environment.yml

@@ -11,6 +11,7 @@ dependencies:
  - python>=3.8
  - pip
  # For running
+ - asf_search
  - boto3
  - cdsapi
  - cfgrib
@@ -23,21 +24,24 @@ dependencies:
  - h5py
  - herbie-data
  - h5netcdf
+ - hyp3lib
  - isce3>=0.9.0
+ - jsonschema==3.2.0  # this is for ASF DAAC ingest schema validation
  - lxml
  - matplotlib
  - netcdf4
  - numpy
  - pandas
  - progressbar
+ - pybind11
  - pydap>3.2.2
  - pyproj>=2.2.0
  - pyyaml
  - rasterio>=1.3.0
  - rioxarray
  - requests
  - s3fs
- - scipy
+ - scipy>1.10.0
  - shapely
  - sysroot_linux-64
  - tqdm
@@ -47,6 +51,7 @@ dependencies:
  - pytest
  - pytest-cov
  - pytest-timeout
+ - pytest-mock
  - pytest-console-scripts
  - setuptools_scm >=6.2
  # For docs website
@@ -56,7 +61,7 @@ dependencies:
  - mkdocs-macros-plugin
  - mkdocs-material
  - mkdocs-material-extensions
- - sentineleof
+ - sentineleof>=0.8.1
  # For RAiDER-docs
  - jupyterlab
  - jupyter_contrib_nbextensions

pyproject.toml

@@ -41,13 +41,14 @@ repository = "https://github.com/dbekaert/RAiDER"
 "Bug Tracker" = "https://github.com/dbekaert/RAiDER/issues"
 
 [project.scripts]
-"raider.py"       = "RAiDER.cli.__main__:main"
-"calcDelays.py"   = "RAiDER.cli.__main__:calcDelays"
-"calcDelaysGUNW.py"     = "RAiDER.cli.__main__:calcDelaysGUNW"
-"raiderDownloadGNSS.py" = "RAiDER.cli.__main__:downloadGNSS"
-"downloadGNSS.py" = "RAiDER.cli.__main__:downloadGNSS"
-"raiderStats.py"  = "RAiDER.cli.statsPlot:main"
+"raider.py"             = "RAiDER.cli.__main__:main"
+"calcDelays.py"         = "RAiDER.cli.raider:calcDelays"
+"calcDelaysGUNW.py"     = "RAiDER.cli.raider:calcDelaysGUNW"
+"raiderDownloadGNSS.py" = "RAiDER.cli.raider:downloadGNSS"
+"downloadGNSS.py"       = "RAiDER.cli.raider:downloadGNSS"
+"raiderStats.py"        = "RAiDER.cli.statsPlot:main"
 "generateGACOSVRT.py"   = "RAiDER.models.generateGACOSVRT:main"
+"raiderCombine.py"      = "RAiDER.cli.raider:combineZTDFiles"
 
 [tool.setuptools]
 include-package-data = true

setup.py

@@ -44,4 +44,5 @@
 setup(
     ext_modules=cython_extensions + pybind_extensions,
     cmdclass={"build_ext": build_ext},
+    package_data={'tools': ['RAiDER/models/*.zip']}
 )

test/__init__.py

@@ -1,10 +1,22 @@
 import os
-
+import pytest
+import subprocess
+import shutil
 from contextlib import contextmanager
 from pathlib import Path
 
+import numpy as np
+import xarray as xr
+
 test_dir = Path(__file__).parents[0]
 
+TEST_DIR = test_dir.absolute()
+DATA_DIR = os.path.join(TEST_DIR, 'data')
+GEOM_DIR = os.path.join(TEST_DIR, 'test_geom')
+WM_DIR   = os.path.join(TEST_DIR, 'weather_files')
+ORB_DIR  = os.path.join(TEST_DIR, 'orbit_files')
+
+WM = 'GMAO'
 
 @contextmanager
 def pushd(dir):
@@ -17,7 +29,52 @@ def pushd(dir):
     os.chdir(prevdir)
 
 
-TEST_DIR = test_dir.absolute()
-DATA_DIR = os.path.join(TEST_DIR, "data")
-GEOM_DIR = os.path.join(TEST_DIR, 'test_geom')
-WM = 'GMAO'
+def update_yaml(dct_cfg:dict, dst:str='temp.yaml'):
+    """ Write a new yaml file from a dictionary.
+
+    Updates parameters in the default 'raider.yaml' file.
+    Each key:value pair will in 'dct_cfg' will overwrite that in the default
+    """
+    import RAiDER, yaml
+
+    template_file = os.path.join(
+                    os.path.dirname(RAiDER.__file__), 'cli', 'raider.yaml')
+
+    with open(template_file, 'r') as f:
+        try:
+            params = yaml.safe_load(f)
+        except yaml.YAMLError as exc:
+            print(exc)
+            raise ValueError(f'Something is wrong with the yaml file {template_file}')
+
+    params = {**params, **dct_cfg}
+
+    with open(dst, 'w') as fh:
+        yaml.safe_dump(params, fh,  default_flow_style=False)
+
+    return dst
+
+
+def makeLatLonGrid(bbox, reg, out_dir, spacing=0.1):
+    """ Make lat lons at a specified spacing """
+    S, N, W, E = bbox
+    lat_st, lat_en = S, N
+    lon_st, lon_en = W, E
+
+    lats = np.arange(lat_st, lat_en, spacing)
+    lons = np.arange(lon_st, lon_en, spacing)
+    Lat, Lon = np.meshgrid(lats, lons)
+    da_lat = xr.DataArray(Lat.T, name='data', coords={'lon': lons, 'lat': lats}, dims='lat lon'.split())
+    da_lon = xr.DataArray(Lon.T, name='data', coords={'lon': lons, 'lat': lats}, dims='lat lon'.split())
+
+    dst_lat = os.path.join(out_dir, f'lat_{reg}.nc')
+    dst_lon = os.path.join(out_dir, f'lon_{reg}.nc')
+    da_lat.to_netcdf(dst_lat)
+    da_lon.to_netcdf(dst_lon)
+
+    return dst_lat, dst_lon
+
+
+def make_delay_name(weather_model_name, date, time, kind='ztd'):
+    assert kind in 'ztd std ray'.split(), 'Incorrect type of delays.'
+    return f'{weather_model_name}_tropo_{date}T{time.replace(":", "")}_{kind}.nc'