Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-7012][SHIMS] Make spark shim provider configurable #7013

Closed
wants to merge 3 commits into from

Conversation

wForget
Copy link
Member

@wForget wForget commented Aug 26, 2024

What changes were proposed in this pull request?

Make spark shim provider configurable

closes #7012

How was this patch tested?

TODO

@github-actions github-actions bot added the CORE works for Gluten Core label Aug 26, 2024
Copy link

#7012

Copy link

Run Gluten Clickhouse CI

@zhztheplayer
Copy link
Member

What's the use case of this feature? I think shim layers may require refactors in near future so we may hardly maintain a stable interface.

@wForget
Copy link
Member Author

wForget commented Aug 26, 2024

What's the use case of this feature? I think shim layers may require refactors in near future so we may hardly maintain a stable interface.

Our internal spark version is based on 3.5.0 and actually backports most of the features of the 3.5 branch. I want to make gluten spark 3.5 shim work by configuring.

@wForget
Copy link
Member Author

wForget commented Aug 26, 2024

I think shim layers may require refactors in near future so we may hardly maintain a stable interface.

I'm not trying to implement custom SparkShims, it's just that the current SparkShims loading is too strict.

@zhztheplayer
Copy link
Member

I think shim layers may require refactors in near future so we may hardly maintain a stable interface.

I'm not trying to implement custom SparkShims, it's just that the current SparkShims loading is too strict.

So in your use case, the code of 3.5.0 shim will still be replaced in Gluten? Perhaps in your Gluten fork repo?

@wForget
Copy link
Member Author

wForget commented Aug 26, 2024

So in your use case, the code of 3.5.0 shim will still be replaced in Gluten?

Yeah, I want to make gluten spark35 shims work internal spark 3.5.0 version.

Perhaps in your Gluten fork repo?

Thank you, I will do that if this PR doesn't get accepted.

@zhztheplayer
Copy link
Member

Thank you, I will do that if this PR doesn't get accepted.

My thought was, if you already have a Gluten fork, the we can put off merging this kind of PR, since the same job can be done in your forked repo.

Otherwise if you directly use upstream Gluten with 3.5.0 Spark, then perhaps we could proceed.

@wForget
Copy link
Member Author

wForget commented Aug 26, 2024

My thought was, if you already have a Gluten fork, the we can put off merging this kind of PR, since the same job can be done in your forked repo.

Thanks, this is feasible for us, we have a gluten fork repo.

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

@wForget
Copy link
Member Author

wForget commented Aug 26, 2024

SparkShimLoader: Using Shim provider: List(org.apache.gluten.sql.shims.spark35.SparkShimProvider@3a2a959e)

@zhztheplayer Sorry, I missed this log, actually org.apache.gluten.sql.shims.spark35.SparkShimProvider is effective for spark 3.5.0, so the current PR is unnecessary and I will close it.

@wForget wForget closed this Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CORE works for Gluten Core
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[SHIMS] Make spark shim provider configurable
2 participants