v0.13.0

What's Changed

This release of PeerDB stars the introduction of a brand new ElasticSearch connector for query replication. Secondly, it ships an in-house min.io bucket integration for the Clickhouse peer, boosting its usability. Also included is a major effort towards perfection of Postgres to Postgres CDC and improved performance of replication to queues. Improvements to UI, tests and telemetry are featured as well.

• ⚙️ Elasticsearch connector for QRep by @heavycrystal in #1618
• 📦 In-house S3 stage for Clickhouse by @Amogh-Bharadwaj in #1634
• 🔄 PgValue variant bypassing QValue for more efficient & correct PG<>PG mirroring by @serprex in #1565
• 🌐 Lua: peerdb.LuaRowTable by @serprex in #1612
• ⚡ Parallelized Lua script execution with serialized output by @serprex in #1613
• 🖥️ UI Create Peer: replace dropdown with columns of buttons by @serprex in #1611
• ❄️ Snowflake: Addition of dynamic numeric columns by @Amogh-Bharadwaj in #1590
• 📊 Queues: communicate flushes back to source in order to keep slot size down by @serprex in #1626
• 💻 UI for opting into PG type system by @Amogh-Bharadwaj in #1630
• 🎯 Eventhubs E2E foundation by @Amogh-Bharadwaj in #1631
• 🛠️ AWS: add credentials loading mechanism using SDK config by @iamKunalGupta in #1507
• 🧪 Framework for ES tests + basic QRep test by @heavycrystal in #1644
• 🖼️ UI: Better Create QRep by @Amogh-Bharadwaj in #1643
• 🗑️ Soft delete false for Snowflake initial load by @Amogh-Bharadwaj in #1647
• 🔔 Alerting: enhance telemetry logging by @iamKunalGupta in #1642

Full Changelog: v0.12.2...v0.13.0

v0.12.2

What's Changed

• grpc: use NewClient by @serprex in #1587
• Touch up Scripts UI by @serprex in #1592
• Warehouse numeric interface by @Amogh-Bharadwaj in #1591
• BigQuery: add flag-gated time partitioning by @Amogh-Bharadwaj in #1588
• UI Peer dropdowns: add tembo and crunchy by @Amogh-Bharadwaj in #1599
• Lua: add unchanged_columns field to Record by @serprex in #1608
• UI: split initial load table into 2 by @Amogh-Bharadwaj in #1607
• UI Mirror page: better overview by @Amogh-Bharadwaj in #1605

Full Changelog: v0.12.1...v0.12.2

v0.12.1

v0.12.1 release

v0.12.0

What's Changed

Version 12 of PeerDB features a new wing to the product - queues. This release introduces support for mirrors to Kafka, PubSub and Azure Eventhubs with transformations using Lua scripts. Documentation for this feature is ongoing at: https://docs.peerdb.io/lua/reference
This version also provides new UI indicators for the mirror page, improved data type mapping and performance/robustness boosts for initial load and CDC.

• Kafka and Lua scripting by @serprex in #1461
• Add scripting to eventhubs by @serprex in #1571
• UI: Create Eventhub Peer and Eventhub Mirror by @Amogh-Bharadwaj in #1566
• PubSub support @serprex in #1524
• UI: Initial load status for initial load tab by @Amogh-Bharadwaj in #1522
• UI CDC: Show inserts, updates and deletes by @Amogh-Bharadwaj in #1525
• UI: Scripts console by @Amogh-Bharadwaj in #1547
• UI: Also show partitioned tables in table picker by @Amogh-Bharadwaj in #1574
• Initial Load: Improved parallelism performance @serprex in #683
• Change QValue into an interface by @serprex in #1528
• Peerdb stats: add source and dest table info for qrep run by @Amogh-Bharadwaj in #1540
• Faster replication connection reset by @serprex in #1549
• qrep: list views, materialized views, & foreign tables in drop down by @serprex in #1550
• Snowflake: null out numerics >38 digits by @serprex in #1554
• UI: Add create mirror buttons by @Amogh-Bharadwaj in #1556
• queues: default script json encodes record by @serprex in #1557
• Eventhub peer refactor by @Amogh-Bharadwaj in #1563
• QRep: Adjust start value for TID watermark column by @Amogh-Bharadwaj in #1569
• kafka/pubsub: PEERDB_QUEUE_FLUSH_TIMEOUT_SECONDS by @serprex in #1559

New Contributors

• @yasinzaehringer-paradime made their first contribution in #1537

Full Changelog: v0.11.4...v0.12.0

v0.11.4

Full Changelog: v0.11.3...v0.11.4

v0.11.3

What's Changed

This release includes some data type improvements for Snowflake, along with some more improvements and test additions to boost robustness

• column exclusion - properly handle schema changes by @heavycrystal in #1512
• UI: remove pub load check by @Amogh-Bharadwaj in #1510
• SF storage integration fixes by @heavycrystal in #1517
• Geospatial data types: set SRID for geometry by @Amogh-Bharadwaj in #1514
• Mirror page: actions dropdown by @Amogh-Bharadwaj in #1513
• Snowflake: Support interval data type by @Amogh-Bharadwaj in #1515
• Add heartbeats to account for QRepPartitionIsSynced by @Amogh-Bharadwaj in #1518
• test for column exclusion with schema changes by @heavycrystal in #1519
• Refactor peer validate check functions by @Amogh-Bharadwaj in #1521

Full Changelog: v0.11.2...v0.11.3

v0.11.2

Full Changelog: v0.11.1...v0.11.2

v0.11.1

What's Changed

This release includes some important bug fixes in our CDC code, some related to how Postgres records LSNs and TID scanning in PostgreSQL 12.

• PullRecords: update replState.Offset in defer by @serprex in #1481
• Move PG/SF simple schema changes to generic test by @serprex in #1477
• Add environment variable to control parallelism of snowflake merge by @serprex in #1487
• Add environment variable to disable one-sync by @serprex in #1488
• Simplify pgtype.Time to time.Time conversion by @serprex in #1492
• UI: New default page and peer buttons by @Amogh-Bharadwaj in #1501
• [snapshot] fallback to full partitions in <PG12, cleanup by @heavycrystal in #1499
• Snowflake: remove ON_ERROR=CONTINUE by @serprex in #1494

Full Changelog: v0.11.0...v0.11.1

v0.11.0

What's Changed

In this version of PeerDB, we restructured our flow architecture resulting in improved reliability (ex: drop mirror) and performance. We added support for a brand new connector in Clickhouse! On the UI side, we now have upgraded validation checks across the board, along with cutting-edge features such as pausing and editing mirrors.
This release sees a major investment in the alerting and monitoring systems, with support for email and slack alerts. We also added support for SSH connections to PostgreSQL peers. When it comes to the core ETL logic, we've solidified our data type handling, memory and storage management, logging and testing.

• 🔄 Split cdc_flow into cdc_flow / sync_flow by @serprex in #1365
• 🛠️ Clickhouse cdc by @pankaj-peerdb in #1096
• 📧 feat(alerting): add email sender by @iamKunalGupta in #1433
• ⏸️ Pause and resume mirror buttons, along with state reflection by @heavycrystal in #1133
• ⚙️ Add Tables Feature: Added dynamic table addition to existing props signal by @heavycrystal in #1106
• 🚀 feat: add telemetry/alerts via sns by @iamKunalGupta in #1411
• 📦 Setup partitioning and clustering for raw table by @iskakaushik in #915
• 🗝️ SSH Support: Use SSHWrappedPool when querying the Postgres Peer by @iskakaushik in #1148
• ⚙️ Custom threshold support for slack alerts by @heavycrystal in #1277
• 🔄 Normalize concurrently with sync flows by @serprex in #893
• 💾 Spill to disk based on flow-worker memory usage by @heavycrystal in #1231
• 🔄 Update max batch size on signal by @iskakaushik in #910
• 🔄 drop_flow: retry until both source/destination succeed by @serprex in #1201
• ⚙️ Adds idletimeout to flow config, ui and temporal signal by @Amogh-Bharadwaj in #952
• ⚙️ Added capability for BQ CDC across datasets by @heavycrystal in #904
• 💾 Support dynamic numeric with defaults by @Amogh-Bharadwaj in #1194
• 🖥️ UI: Edit Page and Refactor Edit Mirror Route by @Amogh-Bharadwaj in #1156
• 🔄 Retry when WAL segment has not been found by @iskakaushik in #930
• 🗺️ HStore and Geospatial for Postgres by @Amogh-Bharadwaj in #1091
• ⚙️ Dynamically add new tables to CDC mirrors by @heavycrystal in #1084
• 🔄 cdc_flow: listen for shutdown request while sync flow in progress by @serprex in #1103
• 🔄 Go 1.22 by @serprex in #1219
• 🖥️ Clickhouse UI by @pankaj-peerdb in #1022
• 🛠️ Validate Mirror: PostgreSQL Checks by @Amogh-Bharadwaj in #1110
• ⚙️ Support specifying host key for ssh config by @serprex in #1125
• 🗑️ drop_flow: drop destination/source concurrently by @serprex in #1101
• 📦 BQ Peer: Support project dot dataset by @Amogh-Bharadwaj in #1073
• 🛠️ Validate peer: permission check for snowflake by @Amogh-Bharadwaj in #1126
• 🗑️ Removing deprecated params from QRep by @heavycrystal in #1154
• 💓 HeartbeatRoutine: use explicit ticker by @serprex in #1157
• 🛠️ Validate peer: check bigquery permissions by @Amogh-Bharadwaj in #1119
• 🖥️ UI: Show peer configuration on clicking peer by @Amogh-Bharadwaj in #1168
• 🔄 Always set application_name when connecting to postgres by @serprex in #1169
• 🖥️ UI: Resync button by @Amogh-Bharadwaj in #1178
• 🖥️ UI: Line chart for slot growth by @Amogh-Bharadwaj in #1184
• 🗑️ Remove configurable postgres metadata database by @serprex in #1189
• 🛠️ Snowflake: stop storing metadata on warehouse; store in catalog by @serprex in #1179
• 🛠️ BigQuery: stop storing metadata on warehouse; store in catalog by @serprex in #1191
• 🔄 Clickhouse cdc data types by @pankaj-peerdb in #1210
• 📝 Better CDC error logging by @iskakaushik in #1275
• 🛠️ Connectors: build on a single GetConnector using generic GetConnectorAs function by @serprex in #1281

Full Changelog: v0.10.2...v0.11.0

v0.10.2

v0.10.2 release