Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[APIPUB-84] [APIPUB-88] - Fix total count timeout and publishing with only 1 change #87

Merged
merged 2 commits into from
Dec 26, 2024

Conversation

ea-mtenhoor
Copy link
Contributor

@ea-mtenhoor ea-mtenhoor commented Dec 5, 2024

This PR fixes issues in two Ed-Fi Success cases, 2795 and 2783. The issue in 2783 appears to have been designated APIPUB-84 internally. These two issues were combined into one PR because they both modify the same lines in the same files, and to avoid merge conflicts.

Case 2795 - API Publisher times out making full min-max CV total count request

The API publisher has functionality to limit the maximum difference between min and max change version. The intent of this functionality is to reduce the load on the source Ed-Fi database, and allow GET requests to return more quickly. This was implemented as a "paging" strategy in API Publisher. In this strategy, the API Publisher correctly limits the change version window size when reading resources, but it still starts each resource with a totalCount request for the entire, unlimited, change version rage. In some deployments this query times out and causes the publisher to skip processing the affected resource. This applies to both the EdFiApiChangeVersionPagingStreamResourcePageMessageProducer and the EdFiApiChangeVersionReversePagingStreamResourcePageMessageProducer.

The code in this PR looks like more changes than it really is, because an if statement was removed and code realigned. Separate commits that make it easier to see what the actual changes are can be seen in the links below.
edanalytics@46b2549
edanalytics@0ffbd0c

Case 2783 - API Publisher doesn't publish when only 1 change was made

Assume that API publisher is configured for a source and target, and has been publishing successfully. The lastChangeVersionProcessed is stored along with the connection information and was successfully updated the last time API publisher ran. Now, only one change is made in the source Ed-Fi environment and the change version value is incremented by 1. API Publisher correctly identifies that there are changes to process and begins looping through the configured resources, getting totalCount. During this process the value it uses for minChangeVersion in the API request is the value from lastChangeVersionsProcessed + 1. Because only one change was made on the source system, the value for minChangeVersion and maxChangeVersion now match. This is fine. When it reaches the resource where totalCount=1, it notes that in the log, but then never publishes the record. This only occurs when using the change version paging or reverse change version paging strategies, and is due to how the the number of "partitions" is calculated. There may be other ways to solve this issue, but I was able to solve it by ensuring that the number of partitions must always be 1 or more.

A separate commit showing this fix is available here. 6f0f0fd

@ea-mtenhoor ea-mtenhoor changed the title Fix total count timeout and publishing with only 1 change [APIPUB-84] - Fix total count timeout and publishing with only 1 change Dec 5, 2024
@roberthunterjr roberthunterjr changed the title [APIPUB-84] - Fix total count timeout and publishing with only 1 change [APIPUB-84] [APIPUB-88] - Fix total count timeout and publishing with only 1 change Dec 19, 2024
@DavidJGapCR DavidJGapCR merged commit 4be38b6 into Ed-Fi-Alliance-OSS:main Dec 26, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants