[Feature][Core] Support cdc task ddl restore for zeta #7463

hailin0 · 2024-08-22T11:03:14Z

Purpose of this pull request

[Improve][CDC] Improve cdc task state restore for zeta

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added

https://github.com/apache/seatunnel/pull/7463/files#diff-56c06c7dde242a3733bec78c1d71f5a32baa3530be2c8f336e28de7e390312a8

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

Hisoka-X

It seem lack test case. Shall we add some?
For example, read updated/deleted column data to unsupported transform/sink, what's will happen?
And read table -> savepoint -> update column -> restore? I think the change will solve the problem of this scenario. But we need test case.

seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SinkWriter.java

Hisoka-X · 2024-08-24T11:54:46Z

...api/src/main/java/org/apache/seatunnel/api/table/factory/ChangeStreamTableSourceFactory.java

+import java.util.Collections;
+import java.util.List;
+
+public interface ChangeStreamTableSourceFactory extends TableSourceFactory {


What's meaning of ChangeStreamTable? CDC?
Any introduce of why we should add this?

e.g. CDC/MQ Source The factory can be used to restore the source state from the checkpoint

dailai · 2024-08-26T08:54:16Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SupportSchemaEvolutionSinkWriter.java

+
+import java.io.IOException;
+
+public interface SupportSchemaEvolutionSinkWriter {


Should this interface extend SinkWriter to reduce the amount of interface for the implementing class.

DDL should have an independent interface, and the connector can choose to support it, and use test cases to detect whether the user has implemented it correctly.

https://github.com/apache/seatunnel/pull/7463/files/4d062ebcfa9fabf37aff535280874dc1443cb30c#diff-4e38101d35f3d5901ed61ddd52530609bc8facb2d1e41476a981b18b3b2ffb9cR198

Other than that, I don't think of a better way. Using extends classes is more difficult to maintain.

dailai · 2024-08-26T08:59:37Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSink.java

+
+    @Override
+    public List<SchemaChangeType> supports() {
+        SeaTunnelSink firstSink = sinks.entrySet().iterator().next().getValue();


The sink action may include different kinds of sink, if the first sink that do not support SupportSchemaEvolutionSink and other support?

For MultiTableSink it should only contain sinks of the same kind
cc @Hisoka-X

For MultiTableSink it should only contain sinks of the same kind

Right for now.

hailin0 · 2024-08-27T15:42:27Z

It seem lack test case. Shall we add some? For example, read updated/deleted column data to unsupported transform/sink, what's will happen? And read table -> savepoint -> update column -> restore? I think the change will solve the problem of this scenario. But we need test case.

Added

https://github.com/apache/seatunnel/pull/7463/files#diff-56c06c7dde242a3733bec78c1d71f5a32baa3530be2c8f336e28de7e390312a8

Carl-Zhou-CN · 2024-09-03T06:53:37Z

@hailin0 hi, I think this part needs a design to help me understand the whole process

hailin0 · 2024-10-28T07:10:20Z

@hailin0 hi, I think this part needs a design to help me understand the whole process

cc @Carl-Zhou-CN

#7930

Hisoka-X · 2024-10-29T01:45:39Z

seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSink.java

+
+    @Override
+    public List<SchemaChangeType> supports() {
+        SeaTunnelSink firstSink = sinks.entrySet().iterator().next().getValue();


For MultiTableSink it should only contain sinks of the same kind

Right for now.

...ient/src/main/java/org/apache/seatunnel/engine/client/job/ClientJobExecutionEnvironment.java

Hisoka-X

LGTM. cc @dailai for a look

dailai · 2024-11-04T03:09:37Z

LGTM. cc @dailai for a look

Waiting to merge #7908

github-actions bot added Zeta connectors-v2 Zeta Rest API api labels Aug 22, 2024

hailin0 force-pushed the dev-merge-ddl branch 2 times, most recently from 23d99f9 to 9d7d79f Compare August 22, 2024 12:59

hailin0 marked this pull request as ready for review August 22, 2024 13:33

dailai mentioned this pull request Aug 23, 2024

Recovery job failed #7472

Closed

3 tasks

hailin0 force-pushed the dev-merge-ddl branch from 9d7d79f to 4d062eb Compare August 24, 2024 02:21

Hisoka-X reviewed Aug 24, 2024

View reviewed changes

dailai reviewed Aug 26, 2024

View reviewed changes

hailin0 mentioned this pull request Aug 26, 2024

SeaTunnel community meeting Topic collect #7499

Closed

3 tasks

hailin0 force-pushed the dev-merge-ddl branch 2 times, most recently from 2a26cea to 1bf5812 Compare September 4, 2024 03:10

dailai mentioned this pull request Sep 5, 2024

[Feature][CDC] Support DDL parse for oracle-cdc #7204

Closed

3 tasks

hailin0 mentioned this pull request Sep 9, 2024

SeaTunnel community meeting Topic collect #7614

Closed

3 tasks

hailin0 force-pushed the dev-merge-ddl branch 2 times, most recently from 550ff5f to fbb5e10 Compare September 14, 2024 13:42

github-actions bot added the e2e label Sep 14, 2024

hailin0 force-pushed the dev-merge-ddl branch from fbb5e10 to 262b77b Compare September 14, 2024 14:49

hailin0 force-pushed the dev-merge-ddl branch from 262b77b to 32e3bae Compare September 27, 2024 04:01

hailin0 force-pushed the dev-merge-ddl branch 5 times, most recently from af42692 to 5ae5961 Compare October 18, 2024 15:44

hailin0 force-pushed the dev-merge-ddl branch from 5ae5961 to fc72bf6 Compare October 25, 2024 15:48

hailin0 force-pushed the dev-merge-ddl branch 2 times, most recently from 4ba4c76 to 30c44a9 Compare October 26, 2024 11:58

[Improve][CDC] Improve cdc task state restore for zeta

e5e63b8

hailin0 force-pushed the dev-merge-ddl branch from 30c44a9 to e5e63b8 Compare October 26, 2024 15:38

hailin0 mentioned this pull request Oct 28, 2024

SeaTunnel community meeting Topic collect #7929

Closed

3 tasks

hailin0 changed the title ~~[Improve][CDC] Improve cdc task state restore for zeta~~ [Improve][CDC] Improve cdc task ddl restore for zeta Oct 28, 2024

hailin0 mentioned this pull request Oct 28, 2024

[Umbrella] CDC DDL Sync Design(Zeta) #7930

Open

7 tasks

hailin0 changed the title ~~[Improve][CDC] Improve cdc task ddl restore for zeta~~ [Feature][CDC] Support cdc task ddl restore for zeta Oct 28, 2024

Hisoka-X reviewed Oct 29, 2024

View reviewed changes

Hisoka-X added this to the 2.3.9 milestone Oct 29, 2024

Hisoka-X approved these changes Oct 29, 2024

View reviewed changes

github-actions bot added approved reviewed labels Oct 29, 2024

hailin0 changed the title ~~[Feature][CDC] Support cdc task ddl restore for zeta~~ [Feature][Core] Support cdc task ddl restore for zeta Oct 29, 2024

Hisoka-X requested a review from dailai November 4, 2024 03:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Core] Support cdc task ddl restore for zeta #7463

[Feature][Core] Support cdc task ddl restore for zeta #7463

hailin0 commented Aug 22, 2024 •

edited

Loading

Hisoka-X left a comment

Hisoka-X Aug 24, 2024

hailin0 Oct 28, 2024 •

edited

Loading

dailai Aug 26, 2024

hailin0 Aug 27, 2024

dailai Aug 26, 2024 •

edited

Loading

hailin0 Aug 27, 2024

Hisoka-X Oct 29, 2024

hailin0 commented Aug 27, 2024 •

edited

Loading

Carl-Zhou-CN commented Sep 3, 2024

hailin0 commented Oct 28, 2024

Hisoka-X Oct 29, 2024

Hisoka-X left a comment

dailai commented Nov 4, 2024


		import java.io.IOException;

		public interface SupportSchemaEvolutionSinkWriter {

[Feature][Core] Support cdc task ddl restore for zeta #7463

Are you sure you want to change the base?

[Feature][Core] Support cdc task ddl restore for zeta #7463

Conversation

hailin0 commented Aug 22, 2024 • edited Loading

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Hisoka-X left a comment

Choose a reason for hiding this comment

Hisoka-X Aug 24, 2024

Choose a reason for hiding this comment

hailin0 Oct 28, 2024 • edited Loading

Choose a reason for hiding this comment

dailai Aug 26, 2024

Choose a reason for hiding this comment

hailin0 Aug 27, 2024

Choose a reason for hiding this comment

dailai Aug 26, 2024 • edited Loading

Choose a reason for hiding this comment

hailin0 Aug 27, 2024

Choose a reason for hiding this comment

Hisoka-X Oct 29, 2024

Choose a reason for hiding this comment

hailin0 commented Aug 27, 2024 • edited Loading

Carl-Zhou-CN commented Sep 3, 2024

hailin0 commented Oct 28, 2024

Hisoka-X Oct 29, 2024

Choose a reason for hiding this comment

Hisoka-X left a comment

Choose a reason for hiding this comment

dailai commented Nov 4, 2024

hailin0 commented Aug 22, 2024 •

edited

Loading

hailin0 Oct 28, 2024 •

edited

Loading

dailai Aug 26, 2024 •

edited

Loading

hailin0 commented Aug 27, 2024 •

edited

Loading