EVA-2178 Fail when DB constraint is violated

[andresfsilva]

based on PR #62

[sundarvenkata-EBI]

I understand that this might be a herculean task due to the concurrency requirements but did you check if it is possible to write a test that fails without the flush statement here but succeeds with it? If this is outside the scope of this PR, suggest creating a new JIRA ticket to check the viability of the timer latch code in PR#62 to test this.

[sundarvenkata-EBI]

In the scope of this PR to honor database constraint violations, we succeed here on the Postgres side by potentially flushing reserved blocks. But on the Mongo side, we still have the insidious save-watermark-after-mongo-save scenario which can lead to overlapping allocations even when we run accession for multiple species sequentially with the same instance ID (see scenario below). I am not sure if recoverState during accession initialization fully fixes this (but this scenario should be relatively easy to simulate).

Scenario:

Step 1: species 1 -> instance-1 -> available block(100-200) -> allocation requested for 10 records -> mongo save for 10 records (partial failure for some reason with 5 written) -xx-> watermark NOT updated for the 5 saved records in the block

Step 2: We fail to check results carefully for the above species (although unlikely if we chain the clustering executions in Nextflow) and proceed with Species 2

species 2 -> instance-1 -> available block(100-200 since no watermark update happened previously) -> allocation requested for 10 records -> mongo save for 10 records (fully succeeds with possible overlap of 5 accessions with species 1) -> watermark updated for 10 saved accessions in the block -> available block (110-200)