⚠️ This is a fairly large issue that's going to require some thought and refactoring.
Status: Draft Being Written
Context
Our Images currently exist as a thin wrapper over TensorData.
As we started to add support for compressed images, we introduced a special-case for JPEG data inside the TensorData type. This should be cleaned up in:
However, support for new image encodings, added in:
Because multi-plane / multi-resolution images are not well represented by dense tensors, we followed the pattern established for jpegs. We can continue to do this even through the cleanup of #3803, but ultimately this is not really a semantically correct representation.
Semantically:
- A multi-plane image (e.g. NV12) should still have a deterministic mapping of pixel-coordinates -> data values. While multi-plane images might involve looking up and combining multiple values, the offsets into the separate channels are still deterministic.
- A compressed image requires a specific decoding step (likely with caching) to convert it into an image that can be used by the viewer.
Proposal
The data representation of images and tensors ultimately needs to be split into 3 separate things:
- Tensors - dense multi-dimensional arrays
- Images - N planes of dense two-dimensional arrays
- ImageAssets - BinaryBlobs with a specified format.
Ideally we should be able to view an image as a tensor, but even then that view should feature a choice of decoder. For example, converting to 3 planes of RGB vs leaving as up-sampled YCrCb.
Status: Draft Being Written
Context
Our Images currently exist as a thin wrapper over TensorData.
As we started to add support for compressed images, we introduced a special-case for JPEG data inside the TensorData type. This should be cleaned up in:
TensorBuffer::JPEGwithImageEncodedarchetype #3803However, support for new image encodings, added in:
Because multi-plane / multi-resolution images are not well represented by dense tensors, we followed the pattern established for jpegs. We can continue to do this even through the cleanup of #3803, but ultimately this is not really a semantically correct representation.
Semantically:
Proposal
The data representation of images and tensors ultimately needs to be split into 3 separate things:
Ideally we should be able to view an image as a tensor, but even then that view should feature a choice of decoder. For example, converting to 3 planes of RGB vs leaving as up-sampled YCrCb.