[APIPUB-84] [APIPUB-88] - Fix total count timeout and publishing with only 1 change #87
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes issues in two Ed-Fi Success cases, 2795 and 2783. The issue in 2783 appears to have been designated APIPUB-84 internally. These two issues were combined into one PR because they both modify the same lines in the same files, and to avoid merge conflicts.
Case 2795 - API Publisher times out making full min-max CV total count request
The API publisher has functionality to limit the maximum difference between min and max change version. The intent of this functionality is to reduce the load on the source Ed-Fi database, and allow GET requests to return more quickly. This was implemented as a "paging" strategy in API Publisher. In this strategy, the API Publisher correctly limits the change version window size when reading resources, but it still starts each resource with a totalCount request for the entire, unlimited, change version rage. In some deployments this query times out and causes the publisher to skip processing the affected resource. This applies to both the EdFiApiChangeVersionPagingStreamResourcePageMessageProducer and the EdFiApiChangeVersionReversePagingStreamResourcePageMessageProducer.
The code in this PR looks like more changes than it really is, because an
if
statement was removed and code realigned. Separate commits that make it easier to see what the actual changes are can be seen in the links below.edanalytics@46b2549
edanalytics@0ffbd0c
Case 2783 - API Publisher doesn't publish when only 1 change was made
Assume that API publisher is configured for a source and target, and has been publishing successfully. The lastChangeVersionProcessed is stored along with the connection information and was successfully updated the last time API publisher ran. Now, only one change is made in the source Ed-Fi environment and the change version value is incremented by 1. API Publisher correctly identifies that there are changes to process and begins looping through the configured resources, getting totalCount. During this process the value it uses for minChangeVersion in the API request is the value from lastChangeVersionsProcessed + 1. Because only one change was made on the source system, the value for minChangeVersion and maxChangeVersion now match. This is fine. When it reaches the resource where totalCount=1, it notes that in the log, but then never publishes the record. This only occurs when using the change version paging or reverse change version paging strategies, and is due to how the the number of "partitions" is calculated. There may be other ways to solve this issue, but I was able to solve it by ensuring that the number of partitions must always be 1 or more.
A separate commit showing this fix is available here. 6f0f0fd