Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

source-pendo: separate backfill from incremental #2023

Merged
merged 2 commits into from
Oct 7, 2024

Conversation

Alex-Bair
Copy link
Contributor

@Alex-Bair Alex-Bair commented Oct 7, 2024

Description:

For Pendo accounts with a significant amount of data, the strategy of asking for all data up until the present does not work. Pendo's response time is long enough to cause TimeoutErrors, and responses that are received end up OOM-ing the connector.

To fix this, there is now a distinct backfill process for events and aggregated events.
Notable changes include:

  • Separating backfills from ongoing incremental replication.
    • Backfills try to get at max a day's worth of data. Incremental replication continues to get all data up to the present.
  • The cutoff between backfills and incremental replication is shifted backwards 12 hours due to delays between when an event occurred and when it is available in the API. This is to prevent backfills from mistaking this delay as a series of empty date windows.
  • The Pendo API response limit is increased to 50k. This might be able to be increased, but I've done limited testing to confirm 50k won't OOM the connector.
  • API responses are now sorted first by timestamp and next by resource ID. This lets us more effectively "paginate" through documents if multiple occur at the same timestamp.
  • If all the documents in a given API response have the same timestamp, the connector will fetch the remaining documents with that exact timestamp before incrementing the connector by a millisecond.

Discover snapshot change is expected due to the increased interval for event streams.

Workflow steps:

(How does one use this feature, and how has it changed)

Documentation links affected:

(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)

Notes for reviewers:

Tested on a local stack. Confirmed:

  • Events and aggregated events are backfilled a day at a time.
  • Backfills are completed once the lower time bound reaches the cutoff date.
  • In a small timeframe, all streams make progress with a data-heavy Pendo account & the connector does not crash.

Sometimes, Pendo's API takes too long to response to a given query, even a simple/small query. These cause intermittent TimeoutErrors, but the connector makes progress after restarting & re-querying the API.

All existing tasks will need their event & aggregated events streams backfilled after merging.


This change is Reviewable

For Pendo accounts with a significant amount of data, the strategy of
asking for all data up until the present does not work. Pendo's response
time is long enough to cause TimeoutErrors, and responses that are
received end up OOM-ing the connector.

To fix this, there is now a distinct backfill process for events and
aggregated events. Notable changes include:
- Separating backfills from ongoing incremental replication.
  - Backfills try to get at max a day's worth of data. Incremental
    replication continues to get all data up to the present.
- The cutoff between backfills and incremental replication is shifted
  backwards 12 hours due to delays between when an event occurred and
  when it is available in the API.
- The Pendo API response limit is increased to 50k. This might be able
  to be increased, but I've done limited testing to confirm 50k won't
  OOM the connector.
- API responses are now sorted first by timestamp and next by resource
  ID. This lets us more effectively "paginate" through documents if
  multiple occur at the same timestamp.
- If all the documents in a given API response have the same timestamp,
  the connector will fetch the remaining documents with that exact
  timestamp before incrementing the connector by a millisecond.
Events don't show up in the Pendo API until 45+ minutes after they
occur. Increasing the Events interval to align with the Aggregated
Events interval makes sense because of this API lag.
@Alex-Bair Alex-Bair added the change:unplanned Unplanned change, useful for things like doc updates label Oct 7, 2024
Copy link
Member

@williamhbaker williamhbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Alex-Bair Alex-Bair merged commit fbf281f into main Oct 7, 2024
68 of 75 checks passed
@Alex-Bair Alex-Bair deleted the bair/pendo-backfill-process branch October 7, 2024 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
change:unplanned Unplanned change, useful for things like doc updates
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants