Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add Spark json_array_length function #11946

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

feat: Add Spark json_array_length function #11946

wants to merge 10 commits into from

Conversation

WangGuangxin
Copy link
Contributor

@WangGuangxin WangGuangxin commented Dec 24, 2024 •
edited by rui-mo
Loading

The 'json_array_length' functions of Presto and Spark are similar, however
Presto returns int64_t, while Spark returns int32_t. In this PR, the
'JsonArrayLengthFunction' implementation is extracted to 'velox/functions/lib'
and registered with differenct return types for Presto and Spark.

Spark doc: https://spark.apache.org/docs/latest/api/sql/#json_array_length

@WangGuangxin WangGuangxin force-pushed the feat_json_array_length branch from a971121 to a5c0d3b Compare December 24, 2024 01:52
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 24, 2024
@netlify Netlify
Copy link

netlify bot commented Dec 24, 2024 •
edited
Loading

✅ Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit a971121
🔍 Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/676a13f34f6f9b00089907c6

@netlify Netlify
Copy link

netlify bot commented Dec 24, 2024 •
edited
Loading

✅ Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit be91204
🔍 Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/6794df504d26020008003ce2

@WangGuangxin WangGuangxin mentioned this pull request Dec 24, 2024
Draft
@WangGuangxin
Copy link
Contributor Author

@rui-mo @JkSelf @PHILO-HE Can you please help review this?

PHILO-HE
PHILO-HE reviewed Dec 24, 2024
View reviewed changes
velox/functions/sparksql/JsonArrayLength.h Outdated
struct JsonArrayLengthFunction {
VELOX_DEFINE_FUNCTION_TYPES(T);

FOLLY_ALWAYS_INLINE bool call(int32_t& len, const arg_type<Json>& json) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion:
Change const arg_type<Json>& to const arg_type<Varchar>& and remove the above corresponding header.

I think JsonType is just for presto.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, fixed

PHILO-HE
PHILO-HE approved these changes Dec 24, 2024
View reviewed changes
Copy link
Contributor

@PHILO-HE PHILO-HE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

rui-mo
rui-mo reviewed Dec 24, 2024
View reviewed changes
Copy link
Collaborator

@rui-mo rui-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution!

velox/functions/sparksql/registration/RegisterJson.cpp Show resolved Hide resolved
rui-mo
rui-mo reviewed Dec 27, 2024
View reviewed changes
Copy link
Collaborator

@rui-mo rui-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Added some comments.

velox/functions/lib/JsonArrayLength.h Show resolved Hide resolved
velox/functions/lib/JsonArrayLength.h Outdated Show resolved Hide resolved
velox/functions/lib/JsonArrayLength.h Show resolved Hide resolved
@rui-mo rui-mo changed the title feat: Add json_array_length Spark function feat: Add Spark json_array_length function Jan 6, 2025
rui-mo
rui-mo reviewed Jan 8, 2025
View reviewed changes
Copy link
Collaborator

@rui-mo rui-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating. Added some comments.

velox/functions/lib/JsonArrayLength.h Outdated Show resolved Hide resolved
velox/functions/lib/JsonArrayLength.h Outdated Show resolved Hide resolved
velox/functions/sparksql/tests/JsonArrayLengthTest.cpp Outdated Show resolved Hide resolved
velox/functions/sparksql/tests/JsonArrayLengthTest.cpp Outdated Show resolved Hide resolved
velox/functions/sparksql/registration/RegisterJson.cpp Show resolved Hide resolved
@WangGuangxin WangGuangxin requested a review from rui-mo January 9, 2025 13:46
rui-mo
rui-mo reviewed Jan 10, 2025
View reviewed changes
Copy link
Collaborator

@rui-mo rui-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Added several nits on the doc.

velox/docs/functions/spark/json.rst Outdated Show resolved Hide resolved
velox/docs/functions/spark/json.rst Outdated Show resolved Hide resolved
velox/docs/functions/spark/json.rst Outdated Show resolved Hide resolved
velox/docs/functions/spark/json.rst Outdated Show resolved Hide resolved
velox/functions/lib/JsonArrayLength.h Outdated Show resolved Hide resolved
@WangGuangxin WangGuangxin requested a review from rui-mo January 13, 2025 01:50
@WangGuangxin WangGuangxin force-pushed the feat_json_array_length branch 2 times, most recently from dbd4e2f to 19936a1 Compare January 20, 2025 14:42
@WangGuangxin
Copy link
Contributor Author

@rui-mo Can you please take a look again?

rui-mo
rui-mo reviewed Jan 24, 2025
View reviewed changes
Copy link
Collaborator

@rui-mo rui-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The compilation fails with below exception. Would you take a look? Thanks.

/__w/velox/velox/./velox/functions/Registerer.h:45:6: note: template argument deduction/substitution failed:
/__w/velox/velox/velox/functions/prestosql/benchmarks/JsonExprBenchmark.cpp:186:61: error: template argument 1 is invalid
186 | registerFunction<JsonArrayLengthFunction, int64_t, Json>(
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
187 | {"json_array_length"});
| ~~~~~~~~~~~~~~~~~~~~~~

@WangGuangxin WangGuangxin force-pushed the feat_json_array_length branch from a256a33 to 28dbf21 Compare January 25, 2025 11:07
@WangGuangxin
Copy link
Contributor Author

The compilation fails with below exception. Would you take a look? Thanks.

/__w/velox/velox/./velox/functions/Registerer.h:45:6: note: template argument deduction/substitution failed:
/__w/velox/velox/velox/functions/prestosql/benchmarks/JsonExprBenchmark.cpp:186:61: error: template argument 1 is invalid
186 | registerFunction<JsonArrayLengthFunction, int64_t, Json>(
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
187 | {"json_array_length"});
| ~~~~~~~~~~~~~~~~~~~~~~

@rui-mo Fixed. the fuzzer job failed seems irrelevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rui-mo rui-mo rui-mo left review comments

@PHILO-HE PHILO-HE PHILO-HE approved these changes

@assignUser assignUser Awaiting requested review from assignUser assignUser is a code owner

@majetideepak majetideepak Awaiting requested review from majetideepak majetideepak is a code owner

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@WangGuangxin @PHILO-HE @rui-mo @facebook-github-bot