[Kernel] Add coordinated commits interfaces and table properties #3370

EstherBear · 2024-07-12T22:07:16Z

Which Delta project/connector is this regarding?

Description

Add coordinated commits interfaces and related table properties in Kernel to prepare for Kernel read and write supported by coordinated commits.

How was this patch tested?

Unit tests

Does this PR introduce any user-facing changes?

Yes. User can define and register their own commit coordinator builder and get the corresponding commit coordinator client handler from engine.

kernel/kernel-api/src/main/java/io/delta/kernel/commit/Commit.java

kernel/kernel-api/src/main/java/io/delta/kernel/internal/TransactionImpl.java

build.sbt

...lts/src/main/java/io/delta/kernel/defaults/engine/DefaultCommitCoordinatorClientHandler.java

kernel/kernel-api/src/main/java/io/delta/kernel/commit/actions/AbstractCommitInfo.java

kernel/kernel-api/src/main/java/io/delta/kernel/internal/TableConfig.java

dhruvarya-db · 2024-07-18T23:06:56Z

kernel/kernel-api/src/main/java/io/delta/kernel/internal/TableConfig.java

@@ -201,4 +260,35 @@ private void validate(String value) {
 private static void addConfig(HashMap<String, TableConfig<?>> configs, TableConfig<?> config) {
 configs.put(config.getKey().toLowerCase(Locale.ROOT), config);
 }
+
+ private static Map<String, String> parseStringToJsonMap(String jsonString) {


Could add some tests?

A few happy path cases

Handling of escaped characters

Test the two areas where you throw exception
I would recommend creating some complex static json with unicode characters using a standard serializer and then deserializing the static json in a test.

yep, and this can be its own separate PR with the design decision doc attached to it.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/TableConfig.java

dhruvarya-db · 2024-07-18T23:09:57Z

kernel/kernel-api/src/main/java/io/delta/kernel/internal/TableConfig.java

+ /*
+ * This table property is used to track the table managed by commit-coordinator internally.
+ */
+ public static final TableConfig<Map<String, String>> COORDINATED_COMMITS_TABLE_CONF =


@vkorukanti should a new type be created here that encapsulates Map<String, String> or is this okay?

This is ok.

kernel/kernel-api/src/main/java/io/delta/kernel/commit/Commit.java

kernel/kernel-api/src/main/java/io/delta/kernel/engine/CommitCoordinatorClientHandler.java

kernel/kernel-api/src/main/java/io/delta/kernel/engine/Engine.java

vkorukanti · 2024-07-18T23:29:04Z

kernel/kernel-api/src/main/java/io/delta/kernel/internal/TableConfig.java

+ /*
+ * This table property is used to track the table managed by commit-coordinator internally.
+ */
+ public static final TableConfig<Map<String, String>> COORDINATED_COMMITS_TABLE_CONF =


This is ok.

vkorukanti · 2024-07-18T23:29:41Z

kernel/kernel-api/src/main/java/io/delta/kernel/internal/TableConfig.java

@@ -201,4 +260,35 @@ private void validate(String value) {
 private static void addConfig(HashMap<String, TableConfig<?>> configs, TableConfig<?> config) {
 configs.put(config.getKey().toLowerCase(Locale.ROOT), config);
 }
+
+ private static Map<String, String> parseStringToJsonMap(String jsonString) {


yep, and this can be its own separate PR with the design decision doc attached to it.

kernel/kernel-defaults/src/main/java/io/delta/kernel/defaults/engine/DefaultEngine.java

...ernel-api/src/main/java/io/delta/kernel/engine/coordinatedcommits/CommitFailedException.java

...l/kernel-api/src/main/java/io/delta/kernel/engine/coordinatedcommits/GetCommitsResponse.java

kernel/kernel-api/src/main/java/io/delta/kernel/engine/coordinatedcommits/UpdatedActions.java

kernel/kernel-api/src/main/java/io/delta/kernel/engine/CommitCoordinatorClientHandler.java

kernel/kernel-api/src/main/java/io/delta/kernel/internal/TableConfig.java

...ain/java/io/delta/kernel/defaults/internal/coordinatedcommits/CommitCoordinatorProvider.java

vkorukanti

Looks great! Thanks for just posting the interfaces. Easy to read and understand.

kernel/kernel-api/src/main/java/io/delta/kernel/engine/CommitCoordinatorClientHandler.java

kernel/kernel-api/src/main/java/io/delta/kernel/engine/Engine.java

...ernel-api/src/main/java/io/delta/kernel/engine/coordinatedcommits/CommitFailedException.java

vkorukanti · 2024-07-19T06:11:18Z

kernel/kernel-api/src/main/java/io/delta/kernel/engine/CommitCoordinatorClientHandler.java

+import io.delta.kernel.engine.coordinatedcommits.actions.AbstractProtocol;
+import io.delta.kernel.utils.CloseableIterator;
+
+/**


general comment to all APIs in this interface: what exception is thrown if the table is not managed by a coordinated commit?

I think the commit coordinator assumes that the logPath passed as an argument represents a table managed by the commit coordinator. For example, the InMemoryCommitCoordinator in Delta Spark doesn't throw a specific exception for this case (will just throw a NullPointerException). So the user should ensure that they pass a correct logPath.
@dhruvarya-db Do I understand correctly?

I don't think that there has been any standardization around that. @prakharjain09 any idea?

No standardization for now.

vkorukanti · 2024-07-19T06:11:45Z

kernel/kernel-api/src/main/java/io/delta/kernel/engine/CommitCoordinatorClientHandler.java

+ /**
+ * Determines whether this CommitCoordinatorClientHandler is semantically equal to another
+ * CommitCoordinatorClientHandler.
+ * <p>
+ * Semantic equality is determined by each CommitCoordinatorClientHandler implementation based
+ * on whether the two instances can be used interchangeably when invoking any of the
+ * CommitCoordinatorClientHandler APIs, such as {@link #commit}, {@link #getCommits}, etc. For
+ * example, both instances might be pointing to the same underlying endpoint.
+ */
+ Boolean semanticEquals(CommitCoordinatorClientHandler other);


why Boolean and not boolean?

I think boolean should work.

raveeram-db

thanks for refactoring these out, very clear!

vkorukanti

LGTM!

EstherBear force-pushed the cc-interface branch 2 times, most recently from 37b3148 to 210a48f Compare July 12, 2024 23:04

raveeram-db reviewed Jul 16, 2024

View reviewed changes

kernel/kernel-api/src/main/java/io/delta/kernel/commit/Commit.java Outdated Show resolved Hide resolved

kernel/kernel-api/src/main/java/io/delta/kernel/internal/TransactionImpl.java Outdated Show resolved Hide resolved

EstherBear force-pushed the cc-interface branch from 210a48f to c74a3bf Compare July 16, 2024 22:47

vkorukanti reviewed Jul 16, 2024

View reviewed changes

build.sbt Outdated Show resolved Hide resolved

dhruvarya-db reviewed Jul 17, 2024

View reviewed changes

...lts/src/main/java/io/delta/kernel/defaults/engine/DefaultCommitCoordinatorClientHandler.java Outdated Show resolved Hide resolved

EstherBear force-pushed the cc-interface branch from 178c9f0 to e10ffd6 Compare July 18, 2024 22:33

dhruvarya-db reviewed Jul 18, 2024

View reviewed changes

vkorukanti requested changes Jul 18, 2024

View reviewed changes

EstherBear force-pushed the cc-interface branch from e10ffd6 to 9ea06a1 Compare July 19, 2024 01:38

dhruvarya-db reviewed Jul 19, 2024

View reviewed changes

...ain/java/io/delta/kernel/defaults/internal/coordinatedcommits/CommitCoordinatorProvider.java Outdated Show resolved Hide resolved

EstherBear force-pushed the cc-interface branch 2 times, most recently from d84b28c to 8c06239 Compare July 19, 2024 02:34

vkorukanti requested changes Jul 19, 2024

View reviewed changes

raveeram-db approved these changes Jul 19, 2024

View reviewed changes

prakharjain09 approved these changes Jul 19, 2024

View reviewed changes

EstherBear force-pushed the cc-interface branch 2 times, most recently from b7fa65a to 9ed9ced Compare July 19, 2024 10:45

vkorukanti added the kernel label Jul 19, 2024

[Kernel] Add coordinated commits interfaces and table properties

6bf228d

EstherBear force-pushed the cc-interface branch from 9ed9ced to 6bf228d Compare July 19, 2024 20:43

vkorukanti approved these changes Jul 19, 2024

View reviewed changes

vkorukanti merged commit 589caba into delta-io:master Jul 19, 2024
10 checks passed

EstherBear mentioned this pull request Jul 22, 2024

[Feature Request] Support Coordinated Commits in Delta Kernel #3406

Open

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Kernel] Add coordinated commits interfaces and table properties #3370

[Kernel] Add coordinated commits interfaces and table properties #3370

EstherBear commented Jul 12, 2024

dhruvarya-db Jul 18, 2024

vkorukanti Jul 18, 2024

dhruvarya-db Jul 18, 2024

vkorukanti Jul 18, 2024

vkorukanti Jul 18, 2024

vkorukanti Jul 18, 2024

vkorukanti left a comment

vkorukanti Jul 19, 2024

EstherBear Jul 19, 2024 •

edited

Loading

dhruvarya-db Jul 19, 2024

EstherBear Jul 19, 2024

vkorukanti Jul 19, 2024

EstherBear Jul 19, 2024 •

edited

Loading

raveeram-db left a comment

vkorukanti left a comment

[Kernel] Add coordinated commits interfaces and table properties #3370

[Kernel] Add coordinated commits interfaces and table properties #3370

Conversation

EstherBear commented Jul 12, 2024

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vkorukanti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EstherBear Jul 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

EstherBear Jul 19, 2024 • edited Loading

Choose a reason for hiding this comment

raveeram-db left a comment

Choose a reason for hiding this comment

vkorukanti left a comment

Choose a reason for hiding this comment

EstherBear Jul 19, 2024 •

edited

Loading

EstherBear Jul 19, 2024 •

edited

Loading