Release DataFusion 44.0.0

[alamb]

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Last release was https://crates.io/crates/datafusion/43.0.0 November 8th, 2024 so next major release would be around December 8, 2024

Prior release tickets:

• 43.0.0 Release DataFusion 43.0.0 #12470

Items to possibly fix before release

• Regression: Invalid comparison operation: Utf8 == Utf8View error during LEFT ANTI JOIN #13510
• [Epic] A Collection of Additional UTF8View support tickets #13504

Improved upgrade experience:

• Make migration to Signature::nullary in 44.0.0 easier / less confusing #13763
• Make DocBuilder migration in 44.0.0 easier #13764
• Making the recursive dependency an optional feature #13766
• Deprecate ScalarUDFImpl::return_type #13717

Downstream project testing:

• datafusion python: chore: Upgrade to DataFusion 44 datafusion-python#972
• comet: Test DataFusion 44.0.0 with Comet #13835
• delta.rs: Test DataFusion 44.0.0 with delta.rs #13834
• Test DataFusion 44.0.0 with Sail #13855

[alamb]

I would love to get this one in:

• Perf: Allow User defined functions to potentially reuse their argument arrays (to avoid new allocations) #13516

Some things to highlight (that are already merged)

• Support vectorized append and compare for multi group by #12996 from @Rachelint as well

[alamb]

Also this hopefully:

• Add SessionConfig reference to ScalarFunctionArgs #13519

[alamb]

@Omega359 and @andygrove suggested #13525 (comment) that for this release we

we file tickets as part of every release to test DF with latest versions of common external dependencies (datafusion-python, iceberg, lance, delta, etc) as part of the gating criteria for a release. Perhaps this would be as simple as changing the DF version in those projects and running their test suite. Any blockers that are found in DF have tickets filed and are fixed, other blockers in the respective projects have tickets filed in those projects

Perhaps for 44.0.0 we can try with own subprojects (Ballista, Comet, DF Python, DF Ray)

[alamb]

I started gathering a list of items we think we should fix prior to the release in the description

[alamb]

Does anyone have any opinion about holding the DataFusion 44 release for the next major arrow release?

• Release arrow-rs / parquet major version 54.0.0 (December 2024) arrow-rs#6342

That would fix a bunch of the StringView issues, but would likely delay DataFusion 44 for a few more weeks

[Omega359]

It's soon to be the holiday season so I'm all for cooking the release a bit more.

[alamb]

I would personally love to see DataFusion 44.0.0 be lauded as "super stable" and have few upgrade issues (we would largely achieve this by testing upgrades with other projects prior to release)

The downside is that the changeset in DataFusion 44.0.0 might be larger

[andygrove]

I'd like to see an upgrade guide for the 44 release (and am willing to take the lead on this). I am trying to upgrade Comet now and am running into some issues.

I am +1 for waiting for the next Arrow release.

[andygrove]

I filed #13702

[andygrove]

How would everyone feel about increasing the length of release votes from 3 days to 7 days to give downstream projects more time to test the release?

Sometimes the vote starts on a Friday and passes on a Monday and we can't expect everyone to be working weekends.

[alamb]

How would everyone feel about increasing the length of release votes from 3 days to 7 days to give downstream projects more time to test the release?

Sometimes the vote starts on a Friday and passes on a Monday and we can't expect everyone to be working weekends.

In my opinion, we would ideally do the testing before we make the release candidate (as the overhead of making RCs is non trivial). However, I am not opposed to extending the voting timeline if that would get us more testing time

[andygrove]

Testing before we create the RC makes sense

[andygrove]

I'm still working on updating Comet to work with latest DF from main (upgrading from 43.0.0). This is quite a disruptive upgrade for us so far.

[alamb]

I have been thinking about this upgrade and the next arrow release

• Release arrow-rs / parquet major version 54.0.0 (December 2024) arrow-rs#6342

Specifically I think we need this upgrade in order to resolve some of the issues in 44 related to string view

• [Epic] A Collection of Additional UTF8View support tickets #13504

However, what I would like to propose is:

1. Plan to release DataFusion 44.0.0 without also upgrading arrow-rs

The downside of this plan is

1. it will take longer to get the arrow upgrade (and this the string view fixes) throughout the ecosystem

The upsides are

1. It will reduce the complexity of the DataFusion 44 upgrade
2. it will give us a longer period to test the upgrade integration DataFusion

If someone is critically waiting on the string view fixes, we can also contemplate making another arrow incremental release

[alamb]

I am also going to coordinate with the delta-rs folks to try and test upgrading prior to release:

• chore: upgrade to datafusion 43 delta-io/delta-rs#2886 (comment)

[timsaucer]

Starting an issue and branch in datafusion-python : apache/datafusion-python#972

[rluvaton]

I'll give my 2 cents from my experience regarding how other popular project handle this issue

In Node.js (I'm a core collaborator) before each release we run the tests of the most popular packages with the new node version and making sure we are not breaking them, if we are we let them know. this is really helpful to avoid breaking the entire world

this process is a CITGM, you can watch this talk for more info

[andygrove]

I am +1 for releasing DF 44 without the next major version of Arrow. However, there are some bug fixes and performance improvements in Arrow that I would like to use so would like to see a new minor release (and am happy to help with this if needed).

[mbwhite]

Hi - would like to get #13647 out as soon as is practical - a small fix but does unblock my use of
DataFusion for Substrait.

Thanks!

[alamb]

I am +1 for releasing DF 44 without the next major version of Arrow. However, there are some bug fixes and performance improvements in Arrow that I would like to use so would like to see a new minor release (and am happy to help with this if needed).

• I have filed Release arrow-rs / parquet minor version 53.4.0 (Jan 2025) arrow-rs#6887 to track this potential release

[alamb]

I think we should start pushing to make this release before it accumulates more API changes such as

• Replace execution_mode with emission_type and boundedness #13823

@andygrove are you willing to try and make this release later this week? Maybe we can rally people to agree on what we need for the next release and try and get the upgrades lines up

[Blizzara]

(Just for what it's worth, I appreciate smaller releases more often compared to bigger ones less often, even if it means more more times I need to fix downstream stuff due to breaks, presumably it usually also means less big breaks. And also faster time to get my and other's improvements and fixes into test/use, and thus faster finding of unintended regressions when they happen. That said, I appreciate there's a bunch of factors including the work needed etc, just wanted to offer my 2 cents 😄)

[andygrove]

@andygrove are you willing to try and make this release later this week? Maybe we can rally people to agree on what we need for the next release and try and get the upgrades lines up

@alamb Sure, but I didn't have time yet to help with creating a migration guide, other than the notes I made on #13702

[andygrove]

The main thing we are waiting on for Comet is #13778

[alamb]

Just for what it's worth, I appreciate smaller releases more often compared to bigger ones less often, even if it means more more times I need to fix downstream stuff due to breaks, presumably it usually also means less big breaks. And also faster time to get my and other's improvements and fixes into test/use, and thus faster finding of unintended regressions when they happen. That said, I appreciate there's a bunch of factors including the work needed etc, just wanted to offer my 2 cents 😄)

I agree -- thank you @Blizzara

In this case I think we are running into bandwidth / maintainer limitations -- I would really like to make sure our releases are less jarring

Does anyone have time to help test with delta.rs?

• Test DataFusion 44.0.0 with delta.rs #13834

[shehabgamin]

Just for what it's worth, I appreciate smaller releases more often compared to bigger ones less often, even if it means more more times I need to fix downstream stuff due to breaks, presumably it usually also means less big breaks. And also faster time to get my and other's improvements and fixes into test/use, and thus faster finding of unintended regressions when they happen. That said, I appreciate there's a bunch of factors including the work needed etc, just wanted to offer my 2 cents 😄)

I agree -- thank you @Blizzara

In this case I think we are running into bandwidth / maintainer limitations -- I would really like to make sure our releases are less jarring

Does anyone have time to help test with delta.rs?

* [Test DataFusion 44.0.0 with delta.rs #13834](https://github.com/apache/datafusion/issues/13834)

https://github.com/lakehq/sail has imported over 3,800 tests from the Spark code base. With each DataFusion upgrade, we have found that any bugs/breaks likely cause one of our tests to fail. If helpful, we're happy to test DataFusion with Sail and include Sail as part of the testing process for each release!

[findepi]

@shehabgamin that would be awesome, thanks!
It could also help if we could land some of these tests in datafusion itself. This would help avoid breakages before a PR is merged.

[shehabgamin]

@findepi Great will do! Also, over time, we're happy to work on porting any tests into DataFusion that would make sense to have in DataFusion.

We'll integrate Sail with the latest commit in the DataFusion main branch and report back our findings. When should Sail complete testing by? We do have bandwidth this week and next week. We’d be happy to prioritize testing this on our side, just lemme know expectations around timing!

[ozankabak]

It could also help if we could land some of these tests in datafusion itself. This would help avoid breakages before a PR is merged.

Agreed, let's do this

[alamb]

We'll integrate Sail with the latest commit in the DataFusion main branch and report back our findings. When should Sail complete testing by? We do have bandwidth this week and next week. We’d be happy to prioritize testing this on our side, just lemme know expectations around timing!

I am personally hoping we are ready to make an RC from main by early next week

It could also help if we could land some of these tests in datafusion itself. This would help avoid breakages before a PR is merged.

Agreed, let's do this

100% agree -- a good place to start might be any tests that you find fail in your project but pass all DataFusion tests

[andygrove]

I am personally hoping we are ready to make an RC from main by early next week

Just fyi, I may not be online much next week, not that I need to be involved in cutting the RC, but will likely have time the following week to help, if needed.

[alamb]

What I plan to do is try and clean up any remaining outstanding PRs and get them merged over the next few days and I'll try and prepare the RC asap

Any help people can give to help figure out how to get the approved PRs merged would be super helpful:

• https://github.com/apache/datafusion/pulls?q=is%3Apr+is%3Aopen+review%3Aapproved+-is%3Adraft

[shehabgamin]

What I plan to do is try and clean up any remaining outstanding PRs and get them merged over the next few days and I'll try and prepare the RC asap

Any help people can give to help figure out how to get the approved PRs merged would be super helpful:

* https://github.com/apache/datafusion/pulls?q=is%3Apr+is%3Aopen+review%3Aapproved+-is%3Adraft

Given that many changes will be merged in after pre-release testing, I think it makes sense to have people run their tests one more time once there is a release candidate. This should be effortless since further breaking changes should be minimal or non-existent. Otherwise, this defeats the purpose of pre-release testing because we cannot guarantee the stability of the final release without validating the accumulated changes.

I'm getting started now with testing on Sail. Using the current latest commit from main branch:5d563d9. Work is being done here lakehq/sail#335. Once the code is able to build, a test report will be generated automatically on the PR.

EDIT: Adding #13855

[shehabgamin]

Test complete: #13855 (comment)
Will re-test once there is a release candidate.