Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing data fetching via pooch #62

Merged
merged 17 commits into from
Jan 9, 2024
Merged

Testing data fetching via pooch #62

merged 17 commits into from
Jan 9, 2024

Conversation

Zeitsperre
Copy link
Collaborator

@Zeitsperre Zeitsperre commented Dec 19, 2023

Pull Request Checklist:

  • This PR addresses an already opened issue (for bug fixes / features)
    • This PR fixes #xyz
  • (If applicable) Documentation has been added / updated (for bug fixes / features).
  • (If applicable) Tests have been added.
  • CHANGES.rst has been updated (with summary of main changes).
    • Link to issue (:issue:number) and pull request (:pull:number) has been added.

What kind of change does this PR introduce?

  • Added pooch as a hard dependency (this can be revisited)
  • Added xhydro.testing.helpers.py and xhydro.testing.registry.txt
  • Added functions:
    • generate_registry: Parses data found in package (xhydro.testing.data), and adds it to the registry.txt
    • load_registry: Loads installed (or custom) registry and returns dictionary
    • populate_testing_data: Fetches the registry and optionally caches files at a different location (helpful for pytest-xdist)
  • Added a pre-commit hook for validating NumPy docstrings, fixed a few docstrings.

Does this PR introduce a breaking change?

Yes. Testing data management is largely handled at the environment variable level:

  • XHYDRO_DATA_DIR can be used to override the default cache location set for pooch (on *nix: $XDG_CACHE_HOME; Must be an absolute path)
  • XHYDRO_TESTDATA_BRANCH can be used to override the branch that the testing data
    should be fetched from (default: main).

Other information:

https://www.fatiando.org/pooch/latest/sample-data.html
Based on a similar approach taken for: https://github.com/Ouranosinc/xclim-testdata

Typically, the loaded pooch registries are given dog names, so I landed on DEVEREAUX as an appropriate name for this project.

@Zeitsperre Zeitsperre added the enhancement New feature or request label Dec 19, 2023
@Zeitsperre Zeitsperre self-assigned this Dec 19, 2023
Copy link
Collaborator

@RondeauG RondeauG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hard to really test or review without data in xhydro-testdata, but overall this seems good!

environment-dev.yml Outdated Show resolved Hide resolved
xhydro/testing/helpers.py Show resolved Hide resolved
Copy link
Collaborator

@RondeauG RondeauG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Zeitsperre Zeitsperre merged commit 643c2eb into main Jan 9, 2024
12 checks passed
@Zeitsperre Zeitsperre deleted the pooch branch January 9, 2024 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants