Skip to content

Very slow data ingestion of light data #3086

@emilk

Description

@emilk

A user reported extremely slow data ingestion of rather simple data (a few lines and points, 13 MB for ~30s worth of data).

When turning on RUST_LOG=debug the terminal was spammed with:

2023-08-23T08:33:14Z DEBUG re_arrow_store::store_write] couldn't split indexed bucket, proceeding to ignore limits kind="insert" timeline=frame time="#0" entity=pitch/gt_ball/cam06/projections len_limit=512 len=2298 len_overflow=true

I suspect this is related.

Metadata

Metadata

Assignees

No one assigned

    Labels

    📉 performanceOptimization, memory use, etc🪳 bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions