Making the recursive dependency an optional feature

[andygrove]

Is your feature request related to a problem or challenge?

The addition of the recursive crate as a dependency in #13310 may cause issues for some downstream projects.

For example, in the DataFusion Comet subproject, we started to see CI failures when running with miri.

error: unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux`
    --> /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/psm-0.1.24/src/lib.rs:319:14
     |
319  |     unsafe { rust_psm_stack_pointer() }
     |              ^^^^^^^^^^^^^^^^^^^^^^^^ can't call foreign function `rust_psm_stack_pointer` on OS `linux`
  

For now, we disabled the miri checks, but we would prefer to keep them and opt out of the recursive feature when testing in CI.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[berkaysynnada]

I have experienced a similar problem for wasm, #13513. After disabling it, got no problem.

[alamb]

@blaginin has a PR to add recursive to sqlparser-rs as well:

• Add #[recursive] datafusion-sqlparser-rs#1522

Perhaps we can follow his model for making this feature optional in DataFusion

[alamb]

I have added this ticket to the the list of things we should fix before 44 release: #13334

[alamb]

FYI @peter-toth

[findepi]

• Add #[recursive] datafusion-sqlparser-rs#1522

Perhaps we can follow his model for making this feature optional in DataFusion

Sounds simple.

@berkaysynnada would you like to make a PR?

[berkaysynnada]

cc @buraksenn

[peter-toth]

Seems like rust_psm_stack_pointer is already on this wishlist for miri: rust-lang/miri#2057

[buraksenn]

Thanks for the heads up @berkaysynnada . Opened a PR to make it optional

[buraksenn]

take

[blaginin]

i’m curious if there are any cases when downstream can’t use recursive yet needs stack overflow protection 🤔 if so, we may need switch to iterative after all

[alamb]

i’m curious if there are any cases when downstream can’t use recursive yet needs stack overflow protection 🤔 if so, we may need switch to iterative after all

I think the challenge will be that converting algorithms to an iterative approach is non trivial effort and will likely complicate the code. So if people can't use recursive but need more efficient stack usage, perhaps they can contribute changes to help improve DataFusion in this area