source-braintree-native: new connector #2177
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Alex-Bair
force-pushed
the
bair/source-braintree
branch
from
a943b38 to
b9db8ba
Compare
This is a minimal connector for Braintree. All incremental streams only capture incremental creations (not updates), so regular backfills are required to capture updates. I did not include logic to trigger backfills within the connector since we already have a separate mechanism for this. Braintree provides SDKs and a GraphQL API for retrieving data. Instead of using the GraphQL API and figuring out the quirks of using it to capture data, I decided to use the Python SDK. This circumvents error handling baked into the Estuary CDK's HTTPMixin object, so we may need to improve error handling later. Braintree's SDK returns custom objects instead of dictionaries, and the `braintree_object_to_dict` function converts these custom objects to dictionaries to feed the Pydantic models. This function may need additional handling/improvement once the connector is used in production and we understand better what types users want. The `credit_card_verifications` stream is not included in the capture snapshot because I couldn't figure out how to create these in a sandbox Braintree account.
Alex-Bair
changed the title
Alex-Bair
force-pushed
the
bair/source-braintree
branch
from
b9db8ba to
f24c8e7
Compare
willdonnelly
reviewed
Dec 3, 2024
willdonnelly
approved these changes
Dec 4, 2024
The window size used for incremental streams is now a config setting, and hitting search limits will cause the connector to error out & state that a smaller window size should be used. The various incremental functions in `resources.py` have been refactored. I did not refactor the various `fetch_foobar` functions in `api.py` since we may need to customize the behavior based on user feedback & I anticipate we'll need to add more streams in the near future. I think it'll be easier/more efficient to refactor these functions when we're done adding additional streams.
Alex-Bair
force-pushed
the
bair/source-braintree
branch
from
209c978 to
e8e4afe
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This is a minimal connector for Braintree. All incremental streams only capture incremental creates (not updates) due to limitations with Braintree's APIs, so regular backfills are required to capture updates. I did not include logic to trigger backfills within the connector since we already have a separate mechanism for scheduled backfills.
Braintree provides SDKs and a GraphQL API for retrieving data. Instead of using the GraphQL API and figuring out the quirks of using it for incremental replication, I decided to use the Python SDK to speed up development. This circumvents error handling baked into the Estuary CDK's
HTTPMixinobject, so we may need to improve error handling later.Braintree's SDK returns custom objects instead of dictionaries, and the
braintree_object_to_dictfunction converts these custom objects to dictionaries to feed the Pydantic models. This function may need additional handling/improvement once the connector is used in production and we better understand what types users want.The
credit_card_verificationsstream is not included in the capture snapshot because I couldn't figure out how to create these in a sandbox Braintree account.Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
Documentation will need created for this connector. It should probably clarify that incremental streams only capture creates, not updates, and regular backfills are needed to capture updates. The docs PR will be linked here once it's ready.
Notes for reviewers:
Tested on a local stack. Confirmed:
I could not figure out how to create a
credit_card_verificationobject within Braintree, so I could not confirm that stream captures data.Supabase, Strapi, and Short.io will need updated to list this new connector in our catalog.
This change is