LakeSoul Release v2.4.0 and Python 1.0 Beta

[xuchen-plus]

What's New In This Release

1. RBAC support for all query engines.
2. Auto cleaning of old compaction data and partition TTL.
3. Upgrade Flink version to 1.17 and support row level update/delete in batch sql.
4. Optimize whole database Flink cdc sync throughput by 80%: [Flink] Optimize CDC sink serde with Fury #307
5. Python reader and integration with PyTorch and HuggingFace.

本次更新内容

1. 支持 RBAC 角色权限控制，对所有引擎、所有语言API均有效；
2. 自动清理旧的 compaction 数据，支持分区级生命周期（TTL）；
3. 升级 Flink 版本到 1.17，并支持批模式下行级别更新和删除；
4. 优化整库同步 Flink 作业，吞吐提升 80%： [Flink] Optimize CDC sink serde with Fury #307 ；
5. 支持原生 Python 读取，提供 PyTorch、HuggingFace 的集成。

What's Changed

• [NativeIO] Upgrade datafusion to 27 by @xuchen-plus in [NativeIO] Upgrade datafusion to 27 #282
• [Flink] implement filter pushdown and fix partition pushdown in flink by @xuchen-plus in [Flink] implement filter pushdown and fix partition pushdown in flink #287
• Upgrade Flink to 1.17 by @xuchen-plus in Upgrade Flink to 1.17 #288
• [Python][NativeIO] Add C interface definition by @xuchen-plus in [Python][NativeIO] Add C interface definition #291
• [NativeIO] update arrow version by @xuchen-plus in [NativeIO] update arrow version #290
• Add Built-in RBAC support by @clouddea in Add Built-in RBAC support #292
• fix apache license by @clouddea in fix apache license #293
• [Native-Metadata] Rust implementation of DAO layer by @Ceng23333 in [Native-Metadata] Rust implementation of DAO layer #294
• [Flink] fix jackson-core package in flink by @xuchen-plus in [Flink] fix jackson-core package in flink #297
• [Docs] update docs by @xuchen-plus in [Docs] update docs #298
• [Flink] upgrade flink cdc connector to 2.4 by @xuchen-plus in [Flink] upgrade flink cdc connector to 2.4 #303
• clean old compaction data and redundant data by @ChenYunHey in clean old compaction data and redundant data #304
• [Python][Native-Metadata] Python interface of lakesoul metadata by @Ceng23333 in [Python][Native-Metadata] Python interface of lakesoul metadata #305
• [Python] C callback with data by @xuchen-plus in [Python] C callback with data #306
• [Python][Dataset] PyArrow and PyTorch dataset api for LakeSoul by @codingfun2022 in [Python][Dataset] PyArrow and PyTorch dataset api for LakeSoul #308
• [Flink] rollback flink cdc to 2.3.0 and supplement tables check in benchmark by @F-PHantam in [Flink] rollback flink cdc to 2.3.0 and supplement tables check in benchmark #309
• [Flink] Optimize CDC sink serde with Fury by @xuchen-plus in [Flink] Optimize CDC sink serde with Fury #307
• [NativeIO] add hdfs feature in lakesoul-io-c by @xuchen-plus in [NativeIO] add hdfs feature in lakesoul-io-c #311
• [Python] exclude partition column at get_arrow_schema_by_table_name by @Ceng23333 in [Python] exclude partition column at get_arrow_schema_by_table_name #312
• [Native-Metadata] Retry when native metadata client fail by @Ceng23333 in [Native-Metadata] Retry when native metadata client fail #313
• [Flink] cdc supplement data delay check mechanism and fix logicallyDropColumn bug by @F-PHantam in [Flink] cdc supplement data delay check mechanism and fix logicallyDropColumn bug #315
• Presto Connector Support by @clouddea in Presto Connector Support #314
• add scala in common to address build in idea intellij by @xuchen-plus in add scala in common to address build in idea intellij #316
• [Flink] Ignore exception when hadoop env missing by @xuchen-plus in [Flink] Ignore exception when hadoop env missing #317
• [NativeIO] Merge native modules by @Ceng23333 in [NativeIO] Merge native modules #318
• bump version to 2.4.0 by @xuchen-plus in bump version to 2.4.0 #319
• [RBAC] Set hdfs dir owner by @xuchen-plus in [RBAC] Set hdfs dir owner #321
• [BugFix]support query metadata with null string by @Ceng23333 in [BugFix]support query metadata with null string #324
• [Spark] list namespace should return empty array by @xuchen-plus in [Spark] list namespace should return empty array #323
• [Python][Dataset] Update Python dataset api for LakeSoul by @codingfun2022 in [Python][Dataset] Update Python dataset api for LakeSoul #325
• [Python] Examples using Python API for AI model training by @Ceng23333 in [Python] Examples using Python API for AI model training #327
• update docs and readme for release 2.4 by @xuchen-plus in update docs and readme for release 2.4 #328
• [Docs] Usage on auto table clean by @ChenYunHey in [Docs] Usage on auto table clean #326
• [Docs] Add presto connector deployment docs by @xuchen-plus in [Docs] Add presto connector deployment docs #329
• [Docs] Add docs for Python and PyTorch by @Ceng23333 in [Docs] Add docs for Python and PyTorch #330
• [Docs] add workspace and rbac docs by @xuchen-plus in [Docs] add workspace and rbac docs #331
• [Bug] turn off native meta query and temporarily disable io prefetch by @F-PHantam in [Bug] turn off native meta query and temporarily disable io prefetch #333
• [Bug]filter should not pushdown before merge on read by @Ceng23333 in [Bug]filter should not pushdown before merge on read #310
• Support view、batch update、batch delete in flink by @moresun in Support view、batch update、batch delete in flink #332
• [Docs ] Refine flink sql and python docs by @xuchen-plus in [Docs ] Refine flink sql and python docs #337

Full Changelog: https://github.com/lakesoul-io/LakeSoul/commits/v2.4.0

---

This discussion was created from the release LakeSoul Release v2.4.0 and Python 1.0 Beta.