Skip to content

re_datastore: store component rows larger than a threshold behind an indirection #487

@teh-cmc

Description

@teh-cmc

Let's say you have an arbitrarily large component (e.g. an Image, whose contents we have no control over).
As the contents of the Image gets larger and larger, the amount of rows we can fit in a single bucket gets smaller and smaller.
In the worst case, you get all of the overhead of bucketing, for none of the benefits.

Not only that, but it might also be the case that the contents of the Image are actually identical e.g. every other frame.
So not only we're wasting bucketing overhead, we're wasting space too.

In both situations, hashing the contents of the image if it gets beyond a certain threshold, and storing that hash instead of the actual contents, fixes the issue.
Of course, we need to store an extra map to resolve the hashes into actual data at query time.

TODO:

  • implement
  • test

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions