Skip to content

Datastore revamp 7: garbage collection#1801

Merged
teh-cmc merged 1 commit intomainfrom
cmc/datastore/new_gc
Apr 12, 2023
Merged

Datastore revamp 7: garbage collection#1801
teh-cmc merged 1 commit intomainfrom
cmc/datastore/new_gc

Conversation

@teh-cmc
Copy link
Copy Markdown
Contributor

@teh-cmc teh-cmc commented Apr 9, 2023 •
edited
Loading

This PR implements garbage collection in the new store and therefore puts an end to the revamp series.

  • Implement new RowId-based GC
    • Focused on correctness, millions of opportunities to improve performance in the future
    • Full support for out of order data
    • Row-level granularity: not tied to the bucket size anymore
  • Lots of improvements to size measurements
    • (Very) Important to reduce the mismatch between what the GC sees vs. the ground truth reported by the global allocator
  • Kill a bunch of unrelated bugs that were found when stress-testing the GC

Highlighted a bunch of issues:

  1. Datastore revamp 1: new indexing model & core datastructures #1727
  2. Datastore revamp 2: serialization & formatting #1735
  3. Datastore revamp 3: efficient incremental stats #1739
  4. Datastore revamp 4: sunset MsgId #1785
  5. Datastore revamp 5: DataStore::to_data_tables() #1791
  6. Datastore revamp 6: sunset LogMsg storage + save store to disk #1795
  7. Datastore revamp 7: garbage collection #1801

On top of #1795
Part of #1619

Fixes #894

Future work:

(Fixed some bugs last minute that improved the graphs in the following demos, but you get the general idea)

23-04-10_12.37.44.patched.mp4
23-04-10_12.32.04.patched.mp4
23-04-10_12.49.06.patched.mp4
Benchmarks
Benchmark suite Current: a889abf Previous: e3f572f Ratio
datastore/num_rows=1000/num_instances=1000/packed=false/insert/default 2493172 ns/iter (± 32525) 11575476 ns/iter (± 688622) 0.22
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at/default 370 ns/iter (± 3) 1809 ns/iter (± 15) 0.20
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/primary/default 262 ns/iter (± 0) 278 ns/iter (± 0) 0.94
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/secondaries/default 420 ns/iter (± 0) 440 ns/iter (± 2) 0.95
datastore/num_rows=1000/num_instances=1000/packed=false/range/default 2625131 ns/iter (± 45606) 12352247 ns/iter (± 772207) 0.21
mono_points_arrow/generate_message_bundles 27981679 ns/iter (± 427787) 48276625 ns/iter (± 1397029) 0.58
mono_points_arrow/generate_messages 114218442 ns/iter (± 914602) 172700989 ns/iter (± 1490257) 0.66
mono_points_arrow/encode_log_msg 155130549 ns/iter (± 2102982) 200215428 ns/iter (± 1409723) 0.77
mono_points_arrow/encode_total 290410867 ns/iter (± 1482761) 423220801 ns/iter (± 2426710) 0.69
mono_points_arrow/decode_log_msg 178057447 ns/iter (± 673276) 253529959 ns/iter (± 1198838) 0.70
mono_points_arrow/decode_message_bundles 60925200 ns/iter (± 507883) 84116550 ns/iter (± 1373716) 0.72
mono_points_arrow/decode_total 236696708 ns/iter (± 1062047) 344891240 ns/iter (± 2570130) 0.69
mono_points_arrow_batched/generate_message_bundles 21948129 ns/iter (± 990466) 41535573 ns/iter (± 1799262) 0.53
mono_points_arrow_batched/generate_messages 4580723 ns/iter (± 159754) 8656223 ns/iter (± 993434) 0.53
mono_points_arrow_batched/encode_log_msg 1379279 ns/iter (± 4409) 1658024 ns/iter (± 5637) 0.83
mono_points_arrow_batched/encode_total 29621541 ns/iter (± 1163999) 53520656 ns/iter (± 2841832) 0.55
mono_points_arrow_batched/decode_log_msg 784267 ns/iter (± 2603) 945416 ns/iter (± 3479) 0.83
mono_points_arrow_batched/decode_message_bundles 7717233 ns/iter (± 186748) 12660668 ns/iter (± 673928) 0.61
mono_points_arrow_batched/decode_total 9002045 ns/iter (± 313841) 14547484 ns/iter (± 923681) 0.62
batch_points_arrow/generate_message_bundles 240791 ns/iter (± 458) 336960 ns/iter (± 1141) 0.71
batch_points_arrow/generate_messages 5122 ns/iter (± 15) 6376 ns/iter (± 28) 0.80
batch_points_arrow/encode_log_msg 265927 ns/iter (± 1247) 453146 ns/iter (± 1968) 0.59
batch_points_arrow/encode_total 539116 ns/iter (± 2351) 798510 ns/iter (± 3665) 0.68
batch_points_arrow/decode_log_msg 213761 ns/iter (± 517) 392739 ns/iter (± 1564) 0.54
batch_points_arrow/decode_message_bundles 1900 ns/iter (± 10) 2314 ns/iter (± 6) 0.82
batch_points_arrow/decode_total 220580 ns/iter (± 861) 393182 ns/iter (± 1465) 0.56
arrow_mono_points/insert 1792578633 ns/iter (± 2165172) 7518086366 ns/iter (± 21814282) 0.24
arrow_mono_points/query 1605489 ns/iter (± 12244) 1856154 ns/iter (± 17882) 0.86
arrow_batch_points/insert 780085 ns/iter (± 1119) 3334078 ns/iter (± 13647) 0.23
arrow_batch_points/query 15975 ns/iter (± 15) 17120 ns/iter (± 49) 0.93
arrow_batch_vecs/insert 26293 ns/iter (± 67) 47904 ns/iter (± 197) 0.55
arrow_batch_vecs/query 535506 ns/iter (± 495) 522963 ns/iter (± 1398) 1.02
tuid/Tuid::random 34 ns/iter (± 0) 34 ns/iter (± 0) 1

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@emilk emilk emilk approved these changes

Assignees

No one assigned

Labels

⛃ re_datastore affects the datastore itself

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Unstable test, re_arrow_store::store_polars::to_dataframe fails

2 participants

@teh-cmc @emilk