v0.3.0

The weather splitter has a new API that allows for partitioning weather data by any dimension (we intentionally exclude lat/lngs). weather-dl Now has a simpler, more pythonic interface for expressing target paths. The weather-mv tool now supports dry runs and BigQuery geopoints.

We're happy to welcome @mahrsee1997 and @ksic8 to the weather-tools dev team!

Current Status

weather-dl: Fixes and DSL usability improvements

• Specifying templates is much simpler. Only target_path is needed, and we fully support python string formatting syntax.
• A significant error was fixed, and now downloads have better skipping and retry logic.
• Log ergonomics were improved by adding timestamps and removing needless warnings (thanks, @pbattaglia!).
• Internal code refactors were included to improve maintainability.
• Data source clients (now, only from ECMWF) includes important license information regarding terms of data use.

weather-mv: Schema & usage improvements

• The default schemas were improved to include BigQuery Geography-type columns. Now, lat/lngs will be represented as POINTs.
• The weather mover now has dry runs! Users will be able to preview their data ingestion into BigQuery before making use of infrastructure.

weather-sp: Flexible splits

• A new version of the splitter was introduce to allow for flexible splits of weather data: Now, you can divide Grib and NetCDF data by any dimension except latitude and longitude (great work, @uhager!).

General

• Pip install instructions include debugging advice for long installs.
• We've removed open meetings from our contributing guide due to low attendance

What's Changed

• Fixes issue (127) where append_date_dirs wasn't working properly. by @pbattaglia in #130
• Added support for dry-runs to weather-mv tool by @mahrsee1997 in #132
• CONTRIBUTING guide changes for pip install command by @ksic8 in #137
• Code-changes for #49, added S2_LOCATION column by @ksic8 in #133
• License information added in client and documentation by @ksic8 in #136
• Added logger timestamp. by @pbattaglia in #131
• Robust download from clients to VMs. by @alxmrs in #143
• Suppress urllib3 warning by @pbattaglia in #134
• Unscheduling developer meetings. by @alxmrs in #145
• Flexible splits by @uhager in #125
• Changes to make 'target_path' and 'target_filename' compliant with Python's standard string formatting by @mahrsee1997 in #144
• Converted Config dict into dataclass by @mahrsee1997 in #142
• Netcdf splits by @uhager in #147

New Contributors

• @mahrsee1997 made their first contribution in #132
• @ksic8 made their first contribution in #137

Full Changelog: v0.2.2...v0.3.0