Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve robustness of metadata results merging #34

Open
michaelwood opened this issue Nov 29, 2021 · 0 comments
Open

Improve robustness of metadata results merging #34

michaelwood opened this issue Nov 29, 2021 · 0 comments
Labels
enhancement New feature or request

Comments

@michaelwood
Copy link
Member

In https://github.com/ThreeSixtyGiving/datagetter/blob/master/getter/get.py#L297 we currently

  1. Take the list of publishers/files from the registry
  2. In each multiprocess take one of the entries from the list and add the metadata information
  3. recreate the list of publishers/files from the registry from the returned results of (2)

If something goes wrong we should return the original entry from the registry to rebuild the list, this is done by catching all the exceptions and returning the original data. This is fine.. but we don't want to rely on that mechanism, especially in the case where one of those multiprocesses gets terminated without an exception but doesn't terminate the parent process (if that is even possible).

@michaelwood michaelwood added the enhancement New feature or request label Nov 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant