Skip to content

Write down a plan for scalar datatype generics #7049

@teh-cmc

Description

@teh-cmc

We are now not too far from being able to support generic bit-sizes (e.g. a 64bit Point3D or a 16bit Scalar) for all things backed by primitive types or arrays of primitive types, for two reasons:

  • The first reason is that Chunks allow it, and therefore the ChunkStore also (mostly) allows it.
  • The second reason is that space views now mostly work with primitive-casted data, e.g.:
let data = re_query2::range_zip_1x5(
    all_positions_indexed,
    all_colors.primitive::<u32>(),
    all_radii.primitive::<f32>(),
    all_labels.string(),
    all_class_ids.primitive::<u16>(),
    all_keypoint_ids.primitive::<u16>(),
)

I.e. one can easily imagine casting to different native types based on the current arrow datatype (likely tedious, but also probably straightforward in most cases).

Note that I'm only talking about bit-size transformations here, nothing beyond that, and definitely nothing that involves changes in semantics (so e.g. Color is likely out).

Still, even that kind of limited change will likely have ripple effects across the entire stack, and those need to be planned for appropriately:

  • What's the impact on our data model (interactions with all things tensor-like in particular, probably also all things transforms).
  • What's the impact on logging APIs: C++, Python & Rust.
  • What's the impact on our query APIs.
  • What's the impact on the different space views implementation (e.g. I imagine a 64bit vertex would not make the renderer happy, so some transformation is required, maybe?).
  • What other impact? E.g. wrt blueprint defaults/overrides?

Looking further into the future, what about generic bit-sizes for timeline values?

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions