Summary

• Add benchmark infrastructure for the MS MARCO v2 passage ranking dataset (138M passages) with download, load, query, and System X comparison scripts
• Introduce weighted-average query latency as a new tracked metric across all benchmark datasets (MS MARCO v1, v2, Wikipedia)
• Per-bucket p50 latencies are weighted by the observed MS-MARCO v1 lexeme distribution (1M queries: 3-token mode at 30%, mean 3.7 tokens) to produce a single summary number that reflects realistic workload performance

Files changed

New (MS MARCO v2 benchmark infrastructure):

• benchmarks/datasets/msmarco-v2/ — download, load, query scripts + benchmark queries TSV
• benchmarks/datasets/msmarco-v2/systemx/ — System X (ParadeDB) comparison scripts

Modified (weighted-average metric):

• benchmarks/datasets/msmarco/queries.sql — add weighted-average computation
• benchmarks/datasets/wikipedia/queries.sql — add weighted-average computation
• benchmarks/runner/extract_metrics.sh — extract WEIGHTED_LATENCY_RESULT from logs
• benchmarks/runner/format_for_action.sh — report weighted latency in github-action-benchmark format
• benchmarks/runner/run_benchmark.sh — add msmarco-v2 as a dataset option
• benchmarks/gh-pages/methodology.html — document weighted methodology + MS MARCO v2 dataset

Distribution weights (MS-MARCO v1, 1,010,905 queries)

Test plan

• Verify weighted-average computation is correct by manual calculation
• Run MS MARCO v1 benchmark and confirm WEIGHTED_LATENCY_RESULT appears in output
• Run extract_metrics.sh on output and verify JSON includes weighted_latency section
• Run format_for_action.sh and verify "Weighted Latency" appears in action output
• CI passes (benchmark-only changes, no C code modified)

🤖 Generated with Claude Code