Skip to content

arrow2 does _not_ refcount schema metadata #1805

@teh-cmc

Description

@teh-cmc

All arrow2 arrays are defined roughly as the following:

pub struct Array {
    data_type: DataType,
    values: Buffer<T>,
    validity: Option<Bitmap>,
}

When you clone/slice/index an Array, you get another Array in roughly O(1) thanks to both the values and validity bitmaps being refcounted behind the scenes:

pub struct Buffer<T> {
    data: Arc<Bytes<T>>,
    offset: usize,
    length: usize,
}

pub struct Bitmap {
    bytes: Arc<Bytes<u8>>,
    offset: usize,
    length: usize,
    unset_bits: usize,
}

Well... not really, turns out the DataType is not refcounted, and it can get huge: it's a massive heap-recursive enum potentially filled with strings and such.

Say you have a ListArray that contains a bunch of StructArrays (i.e. a column of component data) and you want to extract references to the individual StructArrays in that list (i.e. the individual DataCells): each of these arrays is now going to carry a full copy of the StructArray's schema.

For tiny DataCells (which are very common in Rerun), the overhead is enormous.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions