Better retries

[paul-butcher]

What does this change?

This improves the handling of larger shoots by

1. Being better at understanding what has already been transferred
2. Only transferring sub shoots that are not already in born-digital-accessions.

How to test

To see how point 1, above, behaves:

1. Set SOURCE_PROFILE and TARGET_PROFILE accordingly
2. Populate a file shoots/accept-tab with shoot numbers.
3. make the two incomplete transfer lists

• make shoots/accept-tab.transfer_status.touchable
• make shoots/accept-tab.transfer_status.needs_transfer

You should see only what you expect. You can fiddle with accept-tab.transfer_status to give the impression that some files (most pertinently parts of larger shoots, which were not properly dealt with before) have not been ingested (if they actually have been, they will appear in *.touchable), and you can include shoots that you know have not been transferred in the accept-tab list (they will appear in *.needs_transfer).

Point 2 has been fully exercised by the recent complete transfer.

How can we measure success?

The next time we transfer a batch of shoots, some of them will still fail.

In the case of the larger, broken-up shoots, we reingested some of the subshoots multiple times, even though they were successful. Next time, we will only successfully ingest them all once.

Have we considered potential risks?

This is a mitigation for some problems I encountered during the recent ingest.