Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
source-twilio: improve
MessageMedia
incremental sync speed
`MessageMedia` was previously checking every message between the config's start date and the present for new media, then filtering out any media created before the cursor value. That made incremental syncs take an extremely long time without any apparent progress; the stream could be searching through the past few years of messages when it usually only needs to search through the past few minutes. This change makes the `MessageMedia` stream only check messages created since the most recent cursor value, falling back to the config's start date if no cursor value is present. This significantly speeds up the connector during incremental syncs. This change also increases the date window size used when fetching a message's media from 1 year to 100 years. This reduces the number of API requests needed when backfilling media records over a year old; instead of requesting a single year of media at a time, the connector essentially requests all of a message's media in one request. It would make more sense to not use a sliding date window strategy for fetching a single message's media, but rewriting the `MessageMedia` stream in a backwards compatible way is a large effort I'd like to avoid, especially when small, targeted changes address the current issue.
- Loading branch information