We want to be able to address individual pixels in an image, and values in a tensor. For instance: click on a image pixels to select it.
One way to do so is to use our InstanceIds as a pixel index.
This means log_image would be similar to log_points, and we would never have something like log_images.
We would then be able to have multiple values per pixel. For instance: the same tensor could both have a Density component and a Confidence component. This could replace our special-casing of segmentation images.
We would need to store the tensor shape as some sort of meta-data of the Entity, perhaps as a "mono-component" (new concept) of a multi-instance entity.
We would also get sparse tensors for free. A sparse bit-map (e.g. a connectivity graph) could be stored using a zero-sized component.
Before we can work on this we might need to fix the component-converters (https://github.com/rerun-io/rerun/blob/main/design/component_datatypes.md) first, in order to handle JPEG-encoded tensors.
We want to be able to address individual pixels in an image, and values in a tensor. For instance: click on a image pixels to select it.
One way to do so is to use our
InstanceIds as a pixel index.This means
log_imagewould be similar tolog_points, and we would never have something likelog_images.We would then be able to have multiple values per pixel. For instance: the same tensor could both have a
Densitycomponent and aConfidencecomponent. This could replace our special-casing of segmentation images.We would need to store the tensor shape as some sort of meta-data of the Entity, perhaps as a "mono-component" (new concept) of a multi-instance entity.
We would also get sparse tensors for free. A sparse bit-map (e.g. a connectivity graph) could be stored using a zero-sized component.
Before we can work on this we might need to fix the component-converters (https://github.com/rerun-io/rerun/blob/main/design/component_datatypes.md) first, in order to handle JPEG-encoded tensors.