-
Notifications
You must be signed in to change notification settings - Fork 553
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add #[recursive]
#1522
Add #[recursive]
#1522
Conversation
458f748
to
a4a5794
Compare
a4a5794
to
73891ee
Compare
I was wondering if we should remove RecursionCounter with this PR. In my opinion, we shouldn't, because the ability to limit max recursion might be useful for some users still |
It seems this PR doesn't have any significant performance impact.
|
FYI, I marked this ready for review. @peter-toth @Eason0729, if you guys want to take a look 🌻 |
@blaginin thanks for looking to fix this! Currently the preference is to avoid a third-party dependency for this issue, ideally fixing up the parser behavior instead to properly handle deeply nested input. See comment here for a bit more context on rationale |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @blaginin. LGTM except for some number in unit test.
I left some comments, and any number above 40000 seems to overflow stack without #[recursive]
.
@@ -42,6 +42,46 @@ fn basic_queries(c: &mut Criterion) { | |||
group.bench_function("sqlparser::with_select", |b| { | |||
b.iter(|| Parser::parse_sql(&dialect, with_query)); | |||
}); | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For large_statement
, making separated test would make differentiating potential(future) regression easier.
It's a suggestion(fine to leave it as it is).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure I understood your comment, sorry. I added tests for parsing large statements in tests/sqlparser_common.rs. Do you think we should test something else?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thinks it would be better to add as separated test.
@iffyio, we had done some research on stacker / recursive in apache/datafusion#13310 to verify concerns before we added it to datafusion: apache/datafusion#13310 (review) / apache/datafusion#13177 (comment) |
Sorry for missing that context when reviewing. So we would like to avoid using recursive for stablility, maybe try...
|
Thanks for the review!! 🙂
We ended up using |
Ah I see, thanks for the context @peter-toth! cc @alamb for overall thoughts on adding this dependency to sqlparser? |
While adding new dependencies in general ls 🤮 I don't think there is any viable alternative in this case We have tried to avoid doing something like I am hopeful that if we adopt this particular crate we won't have to worry about it again 🤞 |
I think this PR needs a bit more documentation and we shoudl figure out how to rationalize with the existing https://docs.rs/sqlparser/latest/sqlparser/parser/struct.Parser.html#method.with_recursion_limit |
# Conflicts: # tests/sqlparser_common.rs
@alamb, I feel like
I added notes in the methods I touched; should be better now ✋ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
It seems like we reached the decision to add |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it looks good -- any other thoughts @iffyio
Thank you @blaginin and @Eason0729 for pushing this along
Just for transparency, there's apache/datafusion#13513 raised in Datafusion but I believe it shouldn't be the reason not to merge this one (happy to be challenged) |
Maybe we could make it an optional dependency 🤔 |
nice idea actually! will do |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Co-authored-by: Ifeanyi Ubah <ify1992@yahoo.com>
# Conflicts: # tests/sqlparser_common.rs
resolved conflicts, should be good to merge now 🤗 |
Given the potential for unintended side effects with this change, I think we should merge it in asap after we have released |
Should we merge this? or something is missing in this PR. |
# Conflicts: # tests/sqlparser_common.rs
hey, based on the previous comment I think we want to make a release first 🙂 |
Thanks. |
Sorry -- the reason I haven't previously merged this is exactly to meger it after release to give it enough "bake time" . I am glad we did wait, actually, as using this macro has caused trouble downstream in datafusion
I think we are good to go Thank you again @blaginin and @Eason0729 for your contributions and patience |
Closes #984, related to apache/datafusion#9375 (comment)
Todo: