Rerun looks great! I have many hours of 1+ kHz IMU data to analyze, so I am hoping rerun can help with that. I know it is early days of support for the time series visualizations support for kHz data, but I saw https://www.rerun.io/blog/fast-plots and figured it would be good to provide some more data.

To Reproduce
Steps to reproduce the behavior:

1. Install rerun cargo install rerun ✅
2. Create a new Rust project (Cargo.toml and main.rs below) ✅
3. Start the viewer rerun ✅
4. Stream 3.3khz IMU accelerometer data cargo run --release ✅
5. See slow plotting and very high memory usage (>100x increase in RAM data size)
6. ~1 hour to plot and 22GiB used for 16MIB parquet file

Expected behavior
I would expect memory usage 1-10x greater than the accelerometer in the data frame. Here I had 5M rows of ts, x, y, z (32 bytes total), so I expected between 150MiB and 1.5GiB.

Screenshots

Desktop (please complete the following information):

• OS: macOS Sonoma 14.3.1

Rerun version
rerun-cli 0.15.0 [rustc 1.77.1 (7cf61ebde 2024-03-27), LLVM 17.0.6] aarch64-apple-darwin

Additional context
Here is the code for the streaming. I can provide larger or smaller IMU accelerometer data at 416, 833, 1.6k, 3.3k, or 6.6kHz if desired. To bog down the UI, I used about ~25min of data or about 15MiB of parquet.

The dependencies from Cargo.toml

[package]
name = "imu-xyz"
version = "0.1.0"
edition = "2021"

[dependencies]
itertools = "0.12.1"
polars = { version = "0.38.3", features = ["parquet", "lazy", "timezones"] }
rerun = "0.15.0"

The streaming code from main.rs

use itertools::izip;
use polars::prelude::*;
use std::error::Error;

fn main() -> Result<(), Box<dyn Error>> {
    // Load data from a parquet file
    let df = LazyFrame::scan_parquet("stream-0x30.pq", Default::default())?.collect()?;
    println!("{:?}", df);

    // Use Rerun to load accelerometer data from parquet
    let recorder = rerun::RecordingStreamBuilder::new("imu_xyz").connect()?;

    let ts = df.column("ts")?;
    let x = df.column("c1")?;
    let y = df.column("c2")?;
    let z = df.column("c3")?;
    for row in izip!(
        ts.datetime()?.into_no_null_iter(),
        x.i64()?.into_no_null_iter(),
        y.i64()?.into_no_null_iter(),
        z.i64()?.into_no_null_iter()
    ) {
        let (ts, x, y, z) = row;
        recorder.set_time_nanos("ts", ts * 1000);
        recorder.log("imu/x", &rerun::Scalar::new(x as f64))?;
        recorder.log("imu/y", &rerun::Scalar::new(y as f64))?;
        recorder.log("imu/z", &rerun::Scalar::new(z as f64))?;
    }
    Ok(())
}

The stream-0x30.pq (in a .zip)
stream-0x30.pq.zip

imu-xyz % cargo run --release
    Finished release [optimized] target(s) in 0.72s
     Running `target/release/imu-xyz`
shape: (4_809_375, 4)
┌────────────────────────────────┬─────┬─────┬──────┐
│ ts                             ┆ c1  ┆ c2  ┆ c3   │
│ ---                            ┆ --- ┆ --- ┆ ---  │
│ datetime[μs, UTC]              ┆ i64 ┆ i64 ┆ i64  │
╞════════════════════════════════╪═════╪═════╪══════╡
│ 2024-04-10 21:11:27.020704 UTC ┆ 0   ┆ 0   ┆ 460  │
│ 2024-04-10 21:11:27.021003 UTC ┆ 0   ┆ 0   ┆ 2323 │
│ 2024-04-10 21:11:27.021302 UTC ┆ 0   ┆ 0   ┆ 4384 │
│ 2024-04-10 21:11:27.021601 UTC ┆ 0   ┆ 0   ┆ 4125 │
│ 2024-04-10 21:11:27.021900 UTC ┆ 0   ┆ -1  ┆ 3409 │
│ …                              ┆ …   ┆ …   ┆ …    │
│ 2024-04-10 21:35:24.780093 UTC ┆ 30  ┆ -61 ┆ 4028 │
│ 2024-04-10 21:35:24.780392 UTC ┆ 30  ┆ -61 ┆ 4020 │
│ 2024-04-10 21:35:24.780691 UTC ┆ 30  ┆ -62 ┆ 4031 │
│ 2024-04-10 21:35:24.780990 UTC ┆ 29  ┆ -62 ┆ 4025 │
│ 2024-04-10 21:35:24.781289 UTC ┆ 29  ┆ -62 ┆ 4015 │
└────────────────────────────────┴─────┴─────┴──────┘