source-stripe-native: fix streams' incremental replications #1976
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ication. Switch to passing an array of event types for most streams' incremental replications. This lets us listen to more than one type of event (like both `.created`, `.updated`. and `.deleted` events) as well as provide more accurate `meta_.op` information based on specific events. As a result of this switch, some child streams are now listening to events that return the child document directly in the event API response. This means they behave like child streams during backfills but like non-child streams during incremental replication. These streams are listed in the `SPLIT_CHILD_STREAM_NAMES` array, which is used to help determine whether this a standard child stream or a split child stream. Some streams' associated events are not generated by Stripe (ex: `Accounts`, `Persons`, `Payouts`), and determining how to address these is left for a later commit. Discover snapshot changes are expected due to adding `object` to each document's schema.
No `payout.*` events are generated by Stripe despite being detailed in their docs, so we can't rely on those events to do incremental replication. Instead, we can treat `Payouts` as a no event stream. It seems like most payouts aren't updated after creation, so this strategy should work fine most of the time.
williamhbaker
approved these changes
Sep 24, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
There were various issues preventing some streams from incrementally replicating:
ApplicationFees)Payouts).The connector has been updated to let us check for multiple different event types with the
EVENT_TYPESclass variable. This variable references a dictionary whose keys are Stripe API event types and whose values are the appropriatemeta_.opvalue.EVENT_TYPEScan be different between parent and child streams, enabling "split" streams - streams that backfill like child streams but incrementally replicate like standard streams with associated events.Notes:
Payoutsstream was switched over to be a no-events stream since the associated events aren't generated by Stripe. It seems like most payout docs aren't updated after they are created, so this should work for most use cases. But if that assumption doesn't hold & we need to capture updated payouts, we'll need to rethink this strategy.Closes #1965.
Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
Tested on a local stack. Confirmed what streams I could access are now performing incremental replication.
Discover snapshot changes are expected due to description updates and the addition of the
objectattribute to all Stripe documents' schemas.This change is