Skip to content

Conversation

@timsaucer
Copy link
Contributor

This PR is to update the following dependencies:

  • DataFusion to 49.0.2
  • Arrow to 55.2
  • Object Store to 0.12.3

The rest of the changes are just due to updates upstream, but none are breaking.

@github-actions
Copy link
Contributor

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@timsaucer timsaucer changed the title Update DataFusion, Arrow, Object Store chore: update DataFusion, Arrow, Object Store Aug 29, 2025
@github-actions github-actions bot added the chore label Aug 29, 2025
@jackye1995
Copy link
Contributor

I am trying to upgrade and received the same error in #4593. I will wait for this then 😆

@timsaucer
Copy link
Contributor Author

Digging into this for this and also #4593 the problem I'm seeing in lance_encoding::encodings::logical::struct::tests::test_empty_struct is that the update of arrow from 55.1 to 55.2 now has a check for struct arrays having at least one field during the call to concat_structs. In arrow 56, this is corrected because it calls StructArray::try_new_with_length instead of StructArray::try_new.

I don't think we want to bump arrow to 56 with datafusion at 49 because datafusion is at arrow 55 and that would have duplicate dependencies.

I added a TODO in the code I pushed to work around this error.

@codecov-commenter
Copy link

codecov-commenter commented Aug 29, 2025

Codecov Report

❌ Patch coverage is 84.44444% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.56%. Comparing base (da99228) to head (242fd2d).

Files with missing lines Patch % Lines
rust/lance/src/dataset/utils.rs 0.00% 4 Missing ⚠️
rust/lance-index/src/scalar/ngram.rs 0.00% 1 Missing ⚠️
rust/lance/src/dataset/scanner.rs 90.90% 1 Missing ⚠️
rust/lance/src/io/exec/rowids.rs 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4598      +/-   ##
==========================================
- Coverage   80.58%   80.56%   -0.02%     
==========================================
  Files         317      317              
  Lines      119272   119281       +9     
  Branches   119272   119281       +9     
==========================================
- Hits        96111    96103       -8     
- Misses      19713    19727      +14     
- Partials     3448     3451       +3     
Flag Coverage Δ
unittests 80.56% <84.44%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@timsaucer
Copy link
Contributor Author

@westonpace I'm not familiar with the CI failure I'm seeing here - it looks like something related to file system errors on windows. Is that something you've seen in this repo?

I'm also not wild about the fix I put in for the test I describe in the comments above, but I believe it can be removed in the next version, DF50 which should bring Arrow 56

@timsaucer timsaucer force-pushed the tsaucer/upgrade-datafusion-49 branch from 9c7d469 to 8f8afdb Compare September 9, 2025 12:27
@timsaucer
Copy link
Contributor Author

It looks like the CI failure I was getting earlier was resolved after rebase. I think this is ready to go.

Copy link
Contributor

@jackye1995 jackye1995 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me!

@jackye1995 jackye1995 merged commit 621602c into lance-format:main Sep 9, 2025
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants