source-sqlserver: Consider populating /_meta/source/ts_ms
based on cdc.lsn_time_mapping
#2176
Labels
enhancement
New feature or request
The
/_meta/source/ts_ms
property is supposed to hold the "Unix timestamp (in millis) at which this event was recorded by the database" if we have that information.Currently for SQL Server this field is never populated because that information isn't part of the actual CDC change event table, however it would be possible in principle for us to fill it out by looking up change event LSNs against the
cdc.lsn_time_mapping
metadata.We should investigate doing this, and probably profile the impact on typical connector throughput in order to decide whether it should be always-on or an optional feature. It feels like it ought to be possible to handle this fairly efficiently in the common case of small polling intervals, but I'm a bit concerned with how it might scale to larger catchup intervals that we'd see immediately after significant task downtime.
The text was updated successfully, but these errors were encountered: