Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track file replicas separately from files #776

Open
BenGalewsky opened this issue May 17, 2024 · 1 comment
Open

Track file replicas separately from files #776

BenGalewsky opened this issue May 17, 2024 · 1 comment

Comments

@BenGalewsky
Copy link
Contributor

The dataset table represents the list of files connected to a particular DID. Requests are tied to their dataset record. While the list of files is guaranteed to not change, the specific replicas of these files can change all of the time.

We need a way to refresh the list of replicas without introducing referential integrity problems in the transform_results table.

The best solution would be to have a separate table for replicas. The transform results table will reference the file table. When we want to refresh the list of replicas we would delete the replica records for the dataset, but leave the files in place.

The submit_transform workflow will check to see if the dataset record exists or if the list of replicas is empty to decide when to go to Rucio

@ivukotic
Copy link
Member

Dataset table does not have list of files. It has I for on dataset. Info on files is in Files table. While number of files connected to a dataset rarely changes(only during lookup or if files were declared lost), paths variable changes.
While it would work with the new table for replicas, it is really not necessary. One can simply update paths variable in existing file record.
We should probably make a list of all the changes needed on the db and apply them in one go. It would be nice to also drop all the old versions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants