[Feature] Paimon Spark 2025 Roadmap

[Zouxxyy]

Motivation

2025 has arrived, and we would like to thank everyone for the contributions in the past! Here we present the 2025 Paimon Spark roadmap, and welcome to take ownership of them or expand upon them!

Name	Introduction	Link
Variant Type	[feat] Support Variant Type, unlocking support for semi-structured data.	#4471
Optimized Write	[perf] Optimize table writing, including automatic repartitioning and rebalancing data and so on.
Distributed Planning	[perf] Support distributed planning in the scan phase.
Dataframe V2	[feat] Integrate Spark's DataFrame V2 API.
Liquid Clustering	[perf] Support liquid clustering.	#4815
Isolation Level	[feat] Transaction isolation that supports more levels, like serializable isolation level.	#4616
Support For Spark Connect	[feat] Support Spark Connect, calling "Paimon Connect".
Default Value	[feat] Support default values for specified fields.
Constraints	[feat] Support adding constraints to fields, such as not null or other custom constraints.
Partition Stats	[feat] Support partition stats.
Row Lineage	[feat] Support tracking row lineage.
Identity Column	[feat] When no explicit values are provided during writing, generate unique values for identity column.
Generated Columns	[feat] Support generated columns whose values are automatically generated based on a user-specified function over other columns.
CDC For Non-PK Table	[feat] Support CDC for non-pk table.

[YannByron]

Any other features or requirements that you would like to have, please comment here. Then we can discuss and modify this roadmap together. Thanks.

[YannByron]

And if someone want to take one or some, take it and let us know.