filemanager: add way to manually link objects

[mmalenic]

The filemanager API should support a method to manually link objects, in case the ingestId automatic linking fails. This could take a similar form to the manual API calls which allow annotating objects with attributes like the portalRunId.

For example, we could first tag objects on the portalRunId, and then link all objects with the same portalRunId:

/api/v1/s3/link?attributes[portalRunId]=*20220416e17de179*

Care needs to be taken to ensure that the correct object gets linked with the correct id in each directory. How should this be determined? Could objects with the same etag and size be linked? Or the same key suffix? Otherwise objects potentially could be enumerated and linked individually:

/api/v1/s3/link?bucket=pipeline-prod-cache-503977275616-ap-southeast-2&key=byob-icav2/production/analysis/wts/20220416e17de179/MDX220129_somalier/MDX220129.somalier&withBucket=archive-prod-analysis-503977275616-ap-southeast-2&withKey=v1/year=2022/month=04/20220416e17de179/MDX220129_somalier/MDX220129.somalier

[mmalenic]

It should be possible to do this using attributes patching as well, but this would require first finding the original ingestId, and then manually tagging each moved object.

For example, first find the existing ingestId for an object:

/api/v1/s3?bucket=pipeline-prod-cache-503977275616-ap-southeast-2&key=byob-icav2/production/analysis/wts/20220416e17de179/MDX220129_somalier/MDX220129.somalier&currentState=false

Then patch the moved object:

[ {  "op": "replace", "path": "/ingestId", "value": "01938f1e-22f2-7040-af84-2830eea564a7" } ]

At:

/api/v1/s3?bucket=archive-prod-analysis-503977275616-ap-southeast-2&key=v1/year=2022/month=04/20220416e17de179/MDX220129_somalier/MDX220129.somalier

Currently the patch route can only update attributes, so this would still require adding the ability to update ingestId. However, it might be simpler to implement than an entirely new mechanism.

[victorskl]

As discussed, please take time to explore solutions and differing strategy.

It is just another backlog records for us to reprocess. So, all fine.