Fix Invalid offset in sparse column chunk data error for multiple predicates#9509
Merged
alamb merged 1 commit intoapache:mainfrom Mar 11, 2026
Merged
Conversation
39d2986 to
72ca82b
Compare
Closed
adriangb
approved these changes
Mar 5, 2026
Contributor
adriangb
left a comment
There was a problem hiding this comment.
Nice find! The fix makes sense and the test is well designed and comprehensive. I'm not a committer but 👍🏻 from my end.
adriangb
reviewed
Mar 5, 2026
| /// - Without the fix, the override for predicate 2 returns early (policy=Mask, not Auto), | ||
| /// so Mask is used and tries to read missing pages → "Invalid offset" error | ||
| #[tokio::test] | ||
| async fn test_multi_predicate_mask_policy_carryover() { |
Contributor
There was a problem hiding this comment.
I ran this on main and can confirm that it fails on main!
Contributor
|
@cetra3 can you please file a bug ticket to track this issue (it makes the release notes easier) Also do you think we should contemplate backporting this to the previous stable branch (57 I think) and making a patch set? |
Contributor
Author
|
It does make sense to possibly backport this change, given it's a one liner. We have done it for ourselves internally and seen the issue go away |
Contributor
Contributor
|
🚀 BTW I plan to pull together a 58.1.0 release next week |
friendlymatthew
pushed a commit
to pydantic/arrow-rs
that referenced
this pull request
Mar 21, 2026
…dicates (apache#9509) # Which issue does this PR close? Raised an issue at apache#9516 for this one Same issue as apache#9239 but extended to another scenario # Rationale for this change When there are multiple predicates being evaluated, we need to reset the row selection policy before overriding the strategy. Scenario: - Dense initial RowSelection (alternating select/skip) covers all pages → Auto resolves to Mask - Predicate 1 evaluates on column A, narrows selection to skip middle pages - Predicate 2's column B is fetched sparsely with the narrowed selection (missing middle pages) - Without the fix, the override for predicate 2 returns early (policy=Mask, not Auto), so Mask is used and tries to read missing pages → "Invalid offset" error # What changes are included in this PR? This is a one line change to reset the selection policy in the `RowGroupDecoderState::WaitingOnFilterData` arm # Are these changes tested? Yes a new test added that fails currently on `main`, but as you can see it's a doozy to set up. # Are there any user-facing changes? Nope
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Raised an issue at #9516 for this one
Same issue as #9239 but extended to another scenario
Rationale for this change
When there are multiple predicates being evaluated, we need to reset the row selection policy before overriding the strategy.
Scenario:
What changes are included in this PR?
This is a one line change to reset the selection policy in the
RowGroupDecoderState::WaitingOnFilterDataarmAre these changes tested?
Yes a new test added that fails currently on
main, but as you can see it's a doozy to set up.Are there any user-facing changes?
Nope