Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

source-sqlserver: Consider populating /_meta/source/ts_ms based on cdc.lsn_time_mapping #2176

Open
willdonnelly opened this issue Dec 3, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@willdonnelly
Copy link
Member

The /_meta/source/ts_ms property is supposed to hold the "Unix timestamp (in millis) at which this event was recorded by the database" if we have that information.

Currently for SQL Server this field is never populated because that information isn't part of the actual CDC change event table, however it would be possible in principle for us to fill it out by looking up change event LSNs against the cdc.lsn_time_mapping metadata.

We should investigate doing this, and probably profile the impact on typical connector throughput in order to decide whether it should be always-on or an optional feature. It feels like it ought to be possible to handle this fairly efficiently in the common case of small polling intervals, but I'm a bit concerned with how it might scale to larger catchup intervals that we'd see immediately after significant task downtime.

@willdonnelly willdonnelly added the enhancement New feature or request label Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant