divide large shoots by maximum total size #33
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
agnesgaroux
approved these changes
Oct 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this change?
Fixes #31
Previously, because most shoots were relatively small, I wrote all this to download, zip, then upload, in discrete steps. I have now encountered some shoots that were too large for Lambda to cope with in that way.
In order to handle this, I now download only as many files as can fit (alongside the zip they become), then upload that partial shoot, before cleaning up and downloading the next batch.
This could have been optimised further by doing it all in memory, but I have chosen to work with the filesystem partly because it makes it easier to diagnose problems when run locally.
How to test
Push the remaining shoots onto the restore queue. They should all now transfer successfully.
How can we measure success?
This should mark the end of having to make code changes to accommodate the vagaries of this process.
Have we considered potential risks?
It's possible that the size I have set is too big for Archivematica to handle. If that's the case, then I can revisit the numbers.