Skip to content

Buffer from external memory doesn't match ScalarBuffer alignment requirement #4431

@viirya

Description

@viirya

Describe the bug

Recently some memory alignment error was found while passing arrays from Java arrow to Rust arrow. The error comes from From<Buffer> for ScalarBuffer<T>. I inspected the error further and am wondering if the alignment requirement doesn't make sense (at least for working with external allocated memory).

The idea of alignment check in From<Buffer> is making sure that the buffer pointer is aligned with scalar type T. The existing test shows that:

fn test_unaligned() {
  let expected = [0_i32, 1, 2];
  let buffer = Buffer::from_iter(expected.iter().cloned());
  let buffer = buffer.slice(1);
  ScalarBuffer::<i32>::new(buffer, 0, 2);
}

So the buffer slice is not aligned anymore with i32, so converting it to i32 ScalarBuffer panics with alignment error. But the assumption here is the memory region of the buffer is allocated with type. In Java Arrow, it allocates buffer simply with required number of bytes without type information, it's basically a contiguous memory region with n bytes. So obviously it doesn't necessarily have alignment with some type T there. When we try to pass array backed by the buffer to Rust arrow, the alignment check in From<Buffer> will fail occasionally (sometimes it passes the check if the allocated buffer happens to align with T).

To Reproduce

Expected behavior

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions