source-zendesk-support: checkpoint during AuditLogs
stream
#2070
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
For tasks that have a significant number of audit logs in Zendesk Support, it can take more than 24 hours to backfill. With the current
AuditLogs
stream strategy of reading records in descending order then stopping when we've reached a record we already read, we can't checkpoint until the backfill is completed. This means tasks that take more than 24 hours to backfillAuditLogs
can get stuck restarting & attempting to backfill.This PR updates the
AuditLogs
stream to read records in ascending order. This lets the connector checkpoint after each response & pick up where it left off when it's restarted.Some noteable changes include:
start_time
andend_time
for thefilter[created_at][]
query param to bound the responses' results to the specified timespan.end_time
30 seconds in the past to avoid missing records if we query Zendesk in the middle of creating multiple records with the samecreated_at
.AuditLogs
on the olderget_updated_state
method of updating state instead of migrating to the newerstate
property. This is to avoid rebackfilling all existing tasks.Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
The upper time bound for the
AuditLogs
stream is moved a little in the past to avoid potential issues where we query Zendesk Support when its in the middle of creating multiple records with the samecreated_at
. I don't have evidence that this can actually occur, but it sounds possible to me and a 30 second delay is relatively insignificant when the connector is running on 5 minute intervals.Existing tasks do not need backfilled. The
AuditLogs
cursor value has remained the same (created_at
for the most recent record) and the way the connector manages this state is the same.Tested on a local stack. Confirmed:
AuditLogs
state is used after restarting the connector during a backfill.This change is