source-impact-native: fix Actions backfill cursor and other misc. fixes
#2186
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… filter during backfills Previously, the connector missed large batches of `Actions` records during backfills. This is because it was filtering based on the "LockingDate" of each action. It seems like the "LockingDate" is a fixed date the month after an action is created (ex: all actions made in October have a "LockingDate" of 28NOV). This results in backfills effectively only getting actions in month-sized batches. Meaning, if we didn't start a batch on exactly the right time, the backfill will miss data. To fix this, the connector now uses "ActionDateStart" and "ActionDateEnd" query params for `Actions` backfills. This filters results based on their `EventDate`, which gives us a much finer granularity & captures the records we were previously missing.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
The scope of this PR includes:
stop_datetostart_datein the config.Actions.Actionsbackfills. All actions have a "LockingDate" that's a fixed day the month after the action is created. For example, all actions created in October have a "LockingDate" of November 28th. This effectively batches actions into month-sized chunks, and using "LockingDate" as a cursor field forced us to only get completed month-sized chunks. This caused the connector to miss actions whose "LockingDate" was after the cutoff date (i.e. within in-progress month-sized chunks) during backfills.Actionsnow uses theEventDatefield during backfills (via the "ActionDateStart" and "ActionDateEnd" query params). This field has much finer granularity than "LockingDate", and backfills now capture the data that was previously missed.Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
Documentation does not exist for the
source-impact-nativeconnector, so it should be created.Notes for reviewers:
Tested on a local stack. Confirmed that:
Actionsbackfills now get all data between the config's start date & the present - data is no longer missed during a backfill if it's "LockingDate" is after the cutoff date.After this change is merged:
stop_dateto be renamed tostart_date.Snapshot changes are expected due to the
stop_datetostart_daterename, updating the documentation URL, and increasing the interval toPT5M.This change is