Implement ChunkStore and integrate it everywhere by teh-cmc · Pull Request #6570 · rerun-io/rerun

teh-cmc · 2024-06-13T16:50:20Z

Implements the new chunk store and integrates it all over the place.

The highest-level query APIs and caches are unchanged -- for now.

Requires 0.17 to be branched off of main before merging.

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I have tested the web demo (if applicable):
- Using examples from latest main build: rerun.io/viewer
- Using full set of examples from nightly build: rerun.io/viewer
The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
If applicable, add a new check to the release checklist!

To run all checks from main, comment on the PR with @rerun-bot full-check.

github-actions · 2024-06-13T16:50:50Z

Deployed docs

Commit	Link
`6b42567`	https://landing-1mhz4bqec-rerun.vercel.app/docs

teh-cmc · 2024-06-17T16:01:19Z

@rerun-bot full-check

github-actions · 2024-06-17T16:01:52Z

Started a full build: https://github.com/rerun-io/rerun/actions/runs/9550919972

crates/re_chunk_store/src/writes.rs

crates/re_space_view_text_log/src/space_view_class.rs

crates/re_chunk_store/src/query.rs

jleibs

Really like how this turned out. Very nice work.

crates/re_chunk_store/src/query.rs

crates/re_entity_db/src/entity_db.rs

crates/re_chunk_store/src/writes.rs

teh-cmc · 2024-07-01T14:16:59Z

@rerun-bot full-check

github-actions · 2024-07-01T14:17:30Z

Started a full build: https://github.com/rerun-io/rerun/actions/runs/9745909133

Wumpf · 2024-10-18T09:06:48Z

examples/rust/dna/src/main.rs

@@ -12,7 +12,8 @@ use rerun::{
 const NUM_POINTS: usize = 100;

 fn main() -> Result<(), Box<dyn std::error::Error>> {
-    let rec = rerun::RecordingStreamBuilder::new("rerun_example_dna_abacus").spawn()?;
+    let rec = rerun::RecordingStreamBuilder::new("rerun_example_dna_abacus")


This looks like whoopsi

teh-cmc added the do-not-merge Do not merge this PR label Jun 13, 2024

teh-cmc force-pushed the cmc/store_chunks_scratch branch from d647ab2 to d27cbd5 Compare June 13, 2024 16:59

rerun-io deleted a comment from github-actions bot Jun 13, 2024

teh-cmc force-pushed the cmc/store_chunks_scratch branch from d27cbd5 to 83d9019 Compare June 13, 2024 17:13

rerun-io deleted a comment from github-actions bot Jun 13, 2024

teh-cmc mentioned this pull request Jun 14, 2024

Chunk metadata should be passed as an outer container #6572

Open

teh-cmc force-pushed the cmc/store_chunks_scratch branch 8 times, most recently from b38a0e9 to 801ef7e Compare June 17, 2024 15:55

rerun-io deleted a comment from github-actions bot Jun 17, 2024

teh-cmc force-pushed the cmc/store_chunks_scratch branch from 801ef7e to 06f9efb Compare June 17, 2024 16:19

teh-cmc changed the base branch from main to cmc/read_time_clears June 17, 2024 16:27

teh-cmc force-pushed the cmc/store_chunks_scratch branch 3 times, most recently from a8e7f95 to 239c1b4 Compare June 17, 2024 16:49

teh-cmc added ⛃ re_datastore affects the datastore itself 📉 performance Optimization, memory use, etc include in changelog labels Jun 17, 2024

teh-cmc force-pushed the cmc/store_chunks_scratch branch from 239c1b4 to 28a8415 Compare June 18, 2024 08:03

teh-cmc marked this pull request as ready for review June 18, 2024 08:13

teh-cmc force-pushed the cmc/read_time_clears branch from bcdb2a4 to 787a1eb Compare June 18, 2024 08:39

teh-cmc commented Jun 21, 2024

View reviewed changes

crates/re_chunk_store/src/writes.rs Outdated Show resolved Hide resolved

teh-cmc commented Jun 21, 2024

View reviewed changes

crates/re_space_view_text_log/src/space_view_class.rs Show resolved Hide resolved

teh-cmc commented Jun 21, 2024

View reviewed changes

crates/re_chunk_store/src/query.rs Outdated Show resolved Hide resolved

teh-cmc commented Jun 21, 2024

View reviewed changes

crates/re_chunk_store/src/query.rs Show resolved Hide resolved

jleibs approved these changes Jun 21, 2024

View reviewed changes

crates/re_chunk_store/src/query.rs Outdated Show resolved Hide resolved

crates/re_chunk_store/src/query.rs Outdated Show resolved Hide resolved

crates/re_chunk_store/src/query.rs Outdated Show resolved Hide resolved

crates/re_entity_db/src/entity_db.rs Show resolved Hide resolved

teh-cmc mentioned this pull request Jun 24, 2024

Bring back multi-timelines in text-log view #6611

Closed

teh-cmc commented Jun 24, 2024

View reviewed changes

crates/re_chunk_store/src/writes.rs Show resolved Hide resolved

teh-cmc force-pushed the cmc/read_time_clears branch from 5993e19 to ab759fe Compare July 1, 2024 08:58

teh-cmc force-pushed the cmc/store_chunks_scratch branch from 7e789be to 1e24a6b Compare July 1, 2024 09:47

teh-cmc force-pushed the cmc/read_time_clears branch from ab759fe to f509c85 Compare July 4, 2024 13:14

teh-cmc force-pushed the cmc/store_chunks_scratch branch from 56207f3 to 6b42567 Compare July 4, 2024 13:39

teh-cmc changed the title ~~Chunk store waiting room~~ Implement ChunkStore and integrate it everywhere Jul 4, 2024

Base automatically changed from cmc/read_time_clears to main July 4, 2024 15:14

teh-cmc added 10 commits July 4, 2024 17:15

implement and integrate chunk store

fb2d198

support for overlapping chunks in latest-at queries

0f4658b

link to text-log view issue

b8a41a9

sorted by time > sorted by row id

413fd92

missing emptiness checks

d7e90a7

deal with false is_sorted negatives

09daea0

address review comments

e4977d1

lints

f4d295f

even more lints

a861ff4

support per-component row-id ranges and fix static writes accordingly

d304e97

teh-cmc force-pushed the cmc/store_chunks_scratch branch from 6b42567 to d304e97 Compare July 4, 2024 15:15

teh-cmc merged commit ed20cba into main Jul 4, 2024

teh-cmc deleted the cmc/store_chunks_scratch branch July 4, 2024 15:15

teh-cmc removed the do-not-merge Do not merge this PR label Jul 4, 2024

Wumpf reviewed Oct 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement ChunkStore and integrate it everywhere#6570

Implement ChunkStore and integrate it everywhere#6570
teh-cmc merged 10 commits intomainfrom
cmc/store_chunks_scratch

teh-cmc commented Jun 13, 2024 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Jun 13, 2024 •

edited

Loading

Uh oh!

teh-cmc commented Jun 17, 2024

Uh oh!

github-actions bot commented Jun 17, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jleibs left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

teh-cmc commented Jul 1, 2024

Uh oh!

github-actions bot commented Jul 1, 2024

Uh oh!

Wumpf Oct 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

teh-cmc commented Jun 13, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

github-actions bot commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deployed docs

Uh oh!

teh-cmc commented Jun 17, 2024

Uh oh!

github-actions bot commented Jun 17, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jleibs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

teh-cmc commented Jul 1, 2024

Uh oh!

github-actions bot commented Jul 1, 2024

Uh oh!

Wumpf Oct 18, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

teh-cmc commented Jun 13, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Jun 13, 2024 •

edited

Loading