bench: add MS-MARCO v2 full benchmark and profiling scripts#271
Merged
bench: add MS-MARCO v2 full benchmark and profiling scripts#271
Conversation
Add end-to-end benchmark orchestrator (run_full_benchmark.sh) that runs build, single-client latency, and concurrent throughput (power test) for both pg_textsearch and ParadeDB on the 138M passage corpus. Scripts added: - run_full_benchmark.sh: orchestrator with step-based execution - setup_power_test.sql: creates pgbench query table with dense IDs - power_tapir.sql: pgbench script for pg_textsearch throughput - power_systemx.sql: pgbench script for ParadeDB throughput - profile_build.sh: perf/flamegraph profiling for index builds - profile_queries.sql: per-query latency profiling by token bucket
tjgreen42
added a commit
that referenced
this pull request
Mar 27, 2026
Cherry-picked from release-1.0.0 branch.
2 tasks
tjgreen42
added a commit
that referenced
this pull request
Mar 27, 2026
…DB (#297) ## Summary - Add weighted query pool setup and pgbench scripts for concurrent throughput benchmarking with MS-MARCO v1 lexeme distribution - Refresh comparison.html with latest benchmark numbers (commit f31d1af) - Cherry-pick MS-MARCO v2 full benchmark and profiling scripts from release branch (#271) - Rename all `systemx` references to `paradedb` across benchmarks, CI workflow, and docs (from release branch) ## Test plan - [ ] Run MS-MARCO v2 benchmark locally to verify numbers are comparable to published results - [ ] Verify pgbench concurrent throughput scripts work end-to-end
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
run_full_benchmark.shend-to-end orchestrator for reproducible pg_textsearch vs ParadeDB benchmarks on 138M passagessetup_power_test.sql,power_tapir.sql,power_systemx.sql) for concurrent throughput measurement via pgbenchprofile_build.sh,profile_queries.sql) for perf/flamegraph and per-query latency analysisDetails
run_full_benchmark.shsupports step-based execution:Test plan
run_full_benchmark.sh envcaptures system infopower-tapirstep end-to-end on a loaded corpusprofile_queries.sqland verify latency breakdown output