-
Notifications
You must be signed in to change notification settings - Fork 443
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[GLUTEN-7012][SHIMS] Make spark shim provider configurable #7013
Conversation
Run Gluten Clickhouse CI |
What's the use case of this feature? I think shim layers may require refactors in near future so we may hardly maintain a stable interface. |
Our internal spark version is based on 3.5.0 and actually backports most of the features of the 3.5 branch. I want to make gluten spark 3.5 shim work by configuring. |
I'm not trying to implement custom SparkShims, it's just that the current SparkShims loading is too strict. |
So in your use case, the code of 3.5.0 shim will still be replaced in Gluten? Perhaps in your Gluten fork repo? |
Yeah, I want to make gluten spark35 shims work internal spark 3.5.0 version.
Thank you, I will do that if this PR doesn't get accepted. |
My thought was, if you already have a Gluten fork, the we can put off merging this kind of PR, since the same job can be done in your forked repo. Otherwise if you directly use upstream Gluten with 3.5.0 Spark, then perhaps we could proceed. |
Thanks, this is feasible for us, we have a gluten fork repo. |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
@zhztheplayer Sorry, I missed this log, actually |
What changes were proposed in this pull request?
Make spark shim provider configurable
closes #7012
How was this patch tested?
TODO