fix: canonicalize sliced VarBinArray failure #5962
Conversation
| fn test_massive() { | ||
| // Attempt to convert a really large dataset to Arrow. | ||
| let strings = VarBinArray::from_iter_nonnull( | ||
| ["1234567890123"].iter().cycle().take(500_000_000), | ||
| DType::Utf8(Nullability::NonNullable), | ||
| ); | ||
|
|
||
| let sliced = strings.slice(0..5); | ||
|
|
||
| let vbv = sliced.to_varbinview(); | ||
| assert_eq!(vbv.len(), 5); | ||
| } |
There was a problem hiding this comment.
i can't check this in because it's too slow, but this was failing before and now it's not (you actually need to update VarBinArray::from_iter_nonnull to use u64 offsets to make it pass)
There was a problem hiding this comment.
can we not use the slow run on post commit?
9ab1f44 to
c5d63c5
Compare
CodSpeed Performance ReportMerging this PR will not alter performanceComparing Summary
Footnotes
|
c5d63c5 to
ca83938
Compare
fd32064 to
f42a231
Compare
Codecov Report❌ Patch coverage is
☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
f42a231 to
85b569c
Compare
A user report came in with the following stacktrace:
The failure is inside of arrow-rs, in its conversion from LargeStringArray -> StringViewArray.
Arrow checks that the final offset is < i32::MAX, and then pushes the buffer. This will not work if we're trying to convert a large VarBinArray that's been sliced, so that the final offset is not the maximum size of the buffer.
Upstream arrow-rs should get a fix as well, but in the meantime, this should fix the behavior of compressing large chunks, by rezeroing offsets before we canonicalize VarBin.