Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate Proof of Concept Partitions for Behaviour with Dask #444

Open
stephenworsley opened this issue Dec 3, 2024 · 1 comment
Open
Assignees
Labels
New: Issue Highlight a new community raised "generic" issue

Comments

@stephenworsley
Copy link
Contributor

📰 Custom Issue

Subtask of #427.

Before designing the rest of the Partition class, it's worth determining what obstacles there may be to the core idea of loading and applying regridders lazily.

The proof of concept should:

  • Reference a collection of saved regridders and indexing information (probably with some overlaps).
  • Use those regridders to map a dask array to another dask array.
  • Only load the regridders when the result dask array is being realised/computed.

The investigation should check:

  • How flexible is a partition with repsect to different source chunking? Is it necessary to have a specific partition for a specific source chunking?
  • How should the source chunking affect the target chunking?
  • How can vertical chunks be handled efficiently? Is it possible to not have to reload the same regridder multiple times?
  • Are there any blockers to loading during a chunkwise opperation?
@stephenworsley stephenworsley added the New: Issue Highlight a new community raised "generic" issue label Dec 3, 2024
@trexfeathers
Copy link
Contributor

So excited to see this in action (hopefully I can be some use in helping)!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
New: Issue Highlight a new community raised "generic" issue
Projects
Status: No status
Development

No branches or pull requests

3 participants