22 Dec 03:28

What's Changed

This release includes some features and bug fixes. The notable change is that the connector can be integrated with Flink CDC 3.0 to easily build a streaming ELT pipeline from CDC sources (such as MySQL, Kafka) to StarRocks. You can see Flink CDC Synchronization for details.

Features

Implement catalog to support Flink CDC 3.0. #295
Implement new sink api in FLP-191 to support Flink CDC 3.0. #301
Support Flink 1.18. #305

Bug Fixes

Fix misleading thread name and log. #290
Fix wrong configurations when writing to multiple tables . #298

Contributors

Jin-H, zaorangyang, banmoy

Assets 6

15 Sep 11:43

banmoy

v1.2.8

e91f47c

Release 1.2.8

What's Changed

This release includes some improvements and bug fixes. The notable changes are as follows:

Support Flink 1.16 and 1.17.
Recommend to set sink.label-prefix when the sink is configured to guarantee the exactly-once semantics. For the specific instructions, see Exactly Once.

Improvements

Support to configure whether to use Stream Load transaction interface to guarantee at-least-once. #228
Add retry metrics for sink V1. #229
No need to getLabelState when EXISTING_JOB_STATUS is FINISHED. #231
Remove useless stack trace log for sink V1. #232
[Refactor] Move StarRocksSinkManagerV2 to stream-load-sdk. #233
Automatically detect partial updates according to a Flink table's schema instead of the sink.properties.columns parameter explicitly specified by users. #235
[Refactor] Move probeTransactionStreamLoad to stream-load-sdk. #240
Add git-commit-id-plugin for stream-load-sdk. #242
Use info log for DefaultStreamLoader#close. #243
Support to generate stream-load-sdk JAR file without dependencies. #245
Replace fastjson with jackson in stream-load-sdk. #247
Support to process update_before record. #250
Add the Apache license into files. #251
Support to get the exception in stream-load-sdk. #252
Enable strip_outer_array and ignore_json_size by default. #259
Try to cleanup lingering transactions when a Flink job restores and the sink semantics is exactly-once. #271
Return the first exception after the retrying fails. #279

Bug Fixes

Fix typos in StarRocksStreamLoadVisitor. #230
Fix the fastjson classloader leak. #260

Tests

Add the test framework for loading from Kafka to StarRocks. #249

Doc

Refactor the docs. #262
Improve the doc for the sink. #268 #275
Add examples of DataStream API for the sink. #253

Contributors

EMsnap, szza, zaorangyang, hellolilyliuyi, banmoy

Assets 9

10 May 08:35

banmoy

v1.2.7

11c9c91

Release 1.2.7

What's Changed

Improvements

Support to configure 100-continue wait timeout #208
Add pull request template #204
Exit getLabelState() if retry for more than a certain time #212
Use git-commit-id-plugin to get git information at runtime #213

Bug Fixes

Check the http response status code for (transaction) stream load #203
The returned result for transaction rollback should be "OK" #205
The frequency of loading in a transaction should only be controlled by maxCacheBytes #206
Fix that transaction stream load can't configure timeout #210
Don't commit if the data is never flushed #216

Contributors

PengFei Li

Assets 11

06 Mar 09:34

banmoy

v1.2.6

94bf783

Release 1.2.6

What's Changed

Improvements

Optimized the performance of sink V2 by using batch processing, making write and stream load parallel, and using HeapByteBuffer.get() to make data copy more efficient. #181

Features

Added the ability to use transaction stream load for at-least-once mode, which requires StarRocks version 2.4 or higher. #185

Bug Fixes

Fixed an issue with column names containing SQL keywords in the StarRocks Connector for Apache Flink by adding backticks to column names in the HTTP header. #173
Correctly call the complete() method on a Region object after checking the state of the label. #176
Fixed the issue of not using HTTP chunk to send JSON format data for StarRocks versions <=2.2. #174
Resolved the issue of adding backticks when sink.properties.columns are already configured. #183
Fixed the issue of adding the first and end new line delimiters for CSV format. #186
Aborting transactions if the flush operation fails, helping save resources. #188
Removed the HTTP prefix when probing transaction load, allowing the transaction load probing to pass. #189
Appended the CSV row delimiter to the end in the transaction load. #190
Fixed the name mismatch when deserializing JSON string responses. #192
Resolved the dead loop issue in the getAvailableHost() method. #193
Improved the commit process by checking the label state if the commit fails, regardless of the reason for the failure. #195