Skip to content

docs: update MS-MARCO v2 comparison with PR #274 numbers#275

Merged
tjgreen42 merged 1 commit intomainfrom
docs/update-comparison-v2-numbers
Mar 10, 2026
Merged

docs: update MS-MARCO v2 comparison with PR #274 numbers#275
tjgreen42 merged 1 commit intomainfrom
docs/update-comparison-v2-numbers

Conversation

@tjgreen42
Copy link
Copy Markdown
Collaborator

Summary

Key number changes

Metric Before After
Weighted p50 47.62ms (2.0x) 40.61ms (2.3x)
7-token p50 163ms (1.0x) 159ms (1.1x)
8+ token p50 212ms (0.9x) 178ms (1.1x)
Throughput 70ms/q (1.5x) 63ms/q (1.7x)

Test plan

  • Benchmark run twice for consistency on same hardware/config as original
  • System X numbers unchanged (same hardware, not re-run)

BMW cache optimizations (PR #274) improved multi-token query latency
by 20-25%, making pg_textsearch faster than System X across all 8
token buckets at p50.

Key changes:
- Weighted p50: 47.62ms -> 40.61ms (2.0x -> 2.3x vs System X)
- 8+ tokens p50: 212ms -> 178ms (was 0.9x, now 1.1x vs System X)
- Single-client throughput: 70ms/q -> 63ms/q (1.5x -> 1.7x)
- p95 improved on 1-5 token buckets; 6-8+ still mixed
@tjgreen42 tjgreen42 merged commit f080240 into main Mar 10, 2026
1 check passed
@tjgreen42 tjgreen42 deleted the docs/update-comparison-v2-numbers branch March 10, 2026 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant