Skip to content

rust: ADBC driver panics when consuming Arrow batches without properly aligned buffers #2526

@felipecrv

Description

@felipecrv

What happened?

Running SQL queries through the Snowflake ADBC driver, from Rust, can cause a crash due to buffer alignment issues.

thread 'main' panicked at ~/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-buffer-53.4.0/src/buffer/scalar.rs:143:17:                                                                        
Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type. Before importing buffer through FFI, please make sure the allocation is aligned.

This is known problem for Rust [1] when it's consuming buffers from systems that don't care as much about alignment as rustc does. It's been fixed in the FFI integration with Python [2].

The ADBC Rust driver wrapper should do the same. How can it trigger the ArrayData::align_buffers() call that makes the Python FFI integration work without alignment issues?

[1] apache/arrow#43552
[2] apache/arrow-rs#6472

Stack Trace

thread 'tokio-runtime-worker' panicked at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-buffer/src/buffer/scalar.rs:138:17:
Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type. Before importing buffer through FFI, please make sure the allocation is aligned.
stack backtrace:
   0: rust_begin_unwind
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:662:5
   1: core::panicking::panic_fmt
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:74:14
   2: <arrow_buffer::buffer::scalar::ScalarBuffer<T> as core::convert::From<arrow_buffer::buffer::immutable::Buffer>>::from
   3: <T as core::convert::Into<U>>::into
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/convert/mod.rs:759:9
   4: arrow_buffer::buffer::scalar::ScalarBuffer<T>::new
             at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-buffer/src/buffer/scalar.rs:72:9
   5: <arrow_array::array::primitive_array::PrimitiveArray<T> as core::convert::From<arrow_data::data::ArrayData>>::from
             at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/array/primitive_array.rs:1538:22
   6: arrow_array::array::make_array
             at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/array/mod.rs:793:48
   7: <arrow_array::array::struct_array::StructArray as core::convert::From<arrow_data::data::ArrayData>>::from::{{closure}}
             at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/array/struct_array.rs:302:23
   8: core::iter::adapters::map::map_fold::{{closure}}
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/adapters/map.rs:88:28
   9: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::fold
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/slice/iter/macros.rs:232:27
  10: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::fold
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/adapters/map.rs:128:9
  11: core::iter::traits::iterator::Iterator::for_each
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/traits/iterator.rs:813:9
  12: alloc::vec::Vec<T,A>::extend_trusted
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/mod.rs:3121:17
  13: <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_extend.rs:26:9
  14: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_from_iter_nested.rs:60:9
  15: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_from_iter.rs:33:9
  16: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/mod.rs:2985:9
  17: core::iter::traits::iterator::Iterator::collect
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/traits/iterator.rs:2000:9
  18: <arrow_array::array::struct_array::StructArray as core::convert::From<arrow_data::data::ArrayData>>::from
             at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/array/struct_array.rs:299:22
  19: <arrow_array::ffi_stream::ArrowArrayStreamReader as core::iter::traits::iterator::Iterator>::next::{{closure}}
             at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/ffi_stream.rs:367:54
  20: core::result::Result<T,E>::map
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/result.rs:771:25
  21: <arrow_array::ffi_stream::ArrowArrayStreamReader as core::iter::traits::iterator::Iterator>::next
             at /home/felipe/.cargo/git/checkouts/arrow-rs-25fc54bc5e9ed300/f5b51ff/arrow-array/src/ffi_stream.rs:367:18
  22: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/adapters/map.rs:107:9
  23: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_from_iter_nested.rs:24:32
  24: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/spec_from_iter.rs:33:9
  25: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/alloc/src/vec/mod.rs:2985:9
  26: core::iter::traits::iterator::Iterator::collect
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/iter/traits/iterator.rs:2000:9

How can we reproduce the bug?

No response

Environment/Setup

No response

Metadata

Metadata

Assignees

Labels

Type: bugSomething isn't working

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions