Skip to content

Datastore: revamp bench suite#1733

Merged
emilk merged 5 commits intomainfrom
cmc/datastore/revamp_benchmarks
Mar 30, 2023
Merged

Datastore: revamp bench suite#1733
emilk merged 5 commits intomainfrom
cmc/datastore/revamp_benchmarks

Conversation

@teh-cmc
Copy link
Copy Markdown
Contributor

@teh-cmc teh-cmc commented Mar 29, 2023 •
edited
Loading

This reworks the datastore benchmark suite to actually cover all of the important stuff, and make it scalable to more parameters in the future (without breaking existing graphs).
This is a prerequisite for #1619 and in particular #1727 and its upcoming follow ups.

Needs to be merged in separately as it breaks existing benchmarks.

Benchmark suite Current: 29626cd
datastore/num_rows=1000/num_instances=1000/packed=false/insert/default 12625677 ns/iter (± 607274)
datastore/num_rows=1000/num_instances=1000/packed=false/insert/bucketsz=0 14646870 ns/iter (± 1133610)
datastore/num_rows=1000/num_instances=1000/packed=false/insert/bucketsz=2 13995839 ns/iter (± 782248)
datastore/num_rows=1000/num_instances=1000/packed=false/insert/bucketsz=32 13002161 ns/iter (± 984171)
datastore/num_rows=1000/num_instances=1000/packed=false/insert/bucketsz=2048 12813783 ns/iter (± 906175)
datastore/num_rows=1000/num_instances=1000/packed=true/insert/default 12206704 ns/iter (± 599724)
datastore/num_rows=1000/num_instances=1000/packed=true/insert/bucketsz=0 14043694 ns/iter (± 609431)
datastore/num_rows=1000/num_instances=1000/packed=true/insert/bucketsz=2 13270571 ns/iter (± 772281)
datastore/num_rows=1000/num_instances=1000/packed=true/insert/bucketsz=32 11689675 ns/iter (± 452193)
datastore/num_rows=1000/num_instances=1000/packed=true/insert/bucketsz=2048 11165205 ns/iter (± 474775)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at/default 1856 ns/iter (± 20)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at/bucketsz=0 1873 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at/bucketsz=2 1865 ns/iter (± 1)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at/bucketsz=32 1867 ns/iter (± 2)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at/bucketsz=2048 1854 ns/iter (± 33)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at/default 1869 ns/iter (± 18)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at/bucketsz=0 1877 ns/iter (± 2)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at/bucketsz=2 1872 ns/iter (± 6)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at/bucketsz=32 1885 ns/iter (± 3)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at/bucketsz=2048 1883 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/primary/default 286 ns/iter (± 1)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/secondaries/default 438 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/primary/bucketsz=0 281 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/secondaries/bucketsz=0 444 ns/iter (± 6)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/primary/bucketsz=2 280 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/secondaries/bucketsz=2 443 ns/iter (± 1)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/primary/bucketsz=32 282 ns/iter (± 1)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/secondaries/bucketsz=32 443 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/primary/bucketsz=2048 282 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/latest_at_missing/secondaries/bucketsz=2048 439 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/primary/default 281 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/secondaries/default 437 ns/iter (± 1)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/primary/bucketsz=0 280 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/secondaries/bucketsz=0 444 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/primary/bucketsz=2 281 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/secondaries/bucketsz=2 447 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/primary/bucketsz=32 281 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/secondaries/bucketsz=32 444 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/primary/bucketsz=2048 282 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=true/latest_at_missing/secondaries/bucketsz=2048 438 ns/iter (± 0)
datastore/num_rows=1000/num_instances=1000/packed=false/range/default 12517856 ns/iter (± 924973)
datastore/num_rows=1000/num_instances=1000/packed=false/range/bucketsz=0 2216396 ns/iter (± 9037)
datastore/num_rows=1000/num_instances=1000/packed=false/range/bucketsz=2 2188885 ns/iter (± 10207)
datastore/num_rows=1000/num_instances=1000/packed=false/range/bucketsz=32 1957717 ns/iter (± 8345)
datastore/num_rows=1000/num_instances=1000/packed=false/range/bucketsz=2048 1864529 ns/iter (± 6538)
datastore/num_rows=1000/num_instances=1000/packed=true/range/default 12318758 ns/iter (± 392622)
datastore/num_rows=1000/num_instances=1000/packed=true/range/bucketsz=0 2212547 ns/iter (± 6209)
datastore/num_rows=1000/num_instances=1000/packed=true/range/bucketsz=2 2178050 ns/iter (± 7677)
datastore/num_rows=1000/num_instances=1000/packed=true/range/bucketsz=32 1920285 ns/iter (± 12660)
datastore/num_rows=1000/num_instances=1000/packed=true/range/bucketsz=2048 1857672 ns/iter (± 4359)

@teh-cmc teh-cmc added ⛃ re_datastore affects the datastore itself 📉 performance Optimization, memory use, etc labels Mar 29, 2023
@teh-cmc teh-cmc marked this pull request as draft March 29, 2023 18:52
teh-cmc
teh-cmc commented Mar 29, 2023
View reviewed changes
@teh-cmc teh-cmc mentioned this pull request Mar 29, 2023
Merged
9 tasks
@teh-cmc teh-cmc marked this pull request as ready for review March 29, 2023 19:56
@emilk emilk self-requested a review March 30, 2023 08:27
@emilk emilk merged commit 680f791 into main Mar 30, 2023
@emilk emilk deleted the cmc/datastore/revamp_benchmarks branch March 30, 2023 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@emilk emilk emilk approved these changes

Assignees

No one assigned

Labels

📉 performance Optimization, memory use, etc ⛃ re_datastore affects the datastore itself

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@teh-cmc @emilk