-
Notifications
You must be signed in to change notification settings - Fork 186
rust: ADBC driver panics when consuming Arrow batches without properly aligned buffers #2526
Copy link
Copy link
Open
Labels
Type: bugSomething isn't workingSomething isn't working
Milestone
Description
What happened?
Running SQL queries through the Snowflake ADBC driver, from Rust, can cause a crash due to buffer alignment issues.
thread 'main' panicked at ~/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-buffer-53.4.0/src/buffer/scalar.rs:143:17:
Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type. Before importing buffer through FFI, please make sure the allocation is aligned.
This is known problem for Rust [1] when it's consuming buffers from systems that don't care as much about alignment as rustc does. It's been fixed in the FFI integration with Python [2].
The ADBC Rust driver wrapper should do the same. How can it trigger the ArrayData::align_buffers() call that makes the Python FFI integration work without alignment issues?
[1] apache/arrow#43552
[2] apache/arrow-rs#6472
Stack Trace
thread 'tokio-runtime-worker' panicked at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-buffer/src/buffer/scalar.rs:138:17:
Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type. Before importing buffer through FFI, please make sure the allocation is aligned.
stack backtrace:
0: rust_begin_unwind
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:662:5
1: core::panicking::panic_fmt
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:74:14
2: <arrow_buffer::buffer::scalar::ScalarBuffer<T> as core::convert::From<arrow_buffer::buffer::immutable::Buffer>>::from
3: <T as core::convert::Into<U>>::into
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/convert/mod.rs:759:9
4: arrow_buffer::buffer::scalar::ScalarBuffer<T>::new
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-buffer/src/buffer/scalar.rs:72:9
5: <arrow_array::array::primitive_array::PrimitiveArray<T> as core::convert::From<arrow_data::data::ArrayData>>::from
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/array/primitive_array.rs:1538:22
6: arrow_array::array::make_array
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/array/mod.rs:793:48
7: <arrow_array::array::struct_array::StructArray as core::convert::From<arrow_data::data::ArrayData>>::from::{{closure}}
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/array/struct_array.rs:302:23
8: core::iter::adapters::map::map_fold::{{closure}}
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/adapters/map.rs:88:28
9: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::fold
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/slice/iter/macros.rs:232:27
10: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/adapters/map.rs:128:9
11: core::iter::traits::iterator::Iterator::for_each
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/traits/iterator.rs:813:9
12: alloc::vec::Vec<T,A>::extend_trusted
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/mod.rs:3121:17
13: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_extend.rs:26:9
14: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_from_iter_nested.rs:60:9
15: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_from_iter.rs:33:9
16: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/mod.rs:2985:9
17: core::iter::traits::iterator::Iterator::collect
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/traits/iterator.rs:2000:9
18: <arrow_array::array::struct_array::StructArray as core::convert::From<arrow_data::data::ArrayData>>::from
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/array/struct_array.rs:299:22
19: <arrow_array::ffi_stream::ArrowArrayStreamReader as core::iter::traits::iterator::Iterator>::next::{{closure}}
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/ffi_stream.rs:367:54
20: core::result::Result<T,E>::map
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/result.rs:771:25
21: <arrow_array::ffi_stream::ArrowArrayStreamReader as core::iter::traits::iterator::Iterator>::next
at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/ffi_stream.rs:367:18
22: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/adapters/map.rs:107:9
23: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_from_iter_nested.rs:24:32
24: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_from_iter.rs:33:9
25: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/mod.rs:2985:9
26: core::iter::traits::iterator::Iterator::collect
at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/traits/iterator.rs:2000:9
How can we reproduce the bug?
No response
Environment/Setup
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
Type: bugSomething isn't workingSomething isn't working