Skip to content

Fix BM25 field weights to include all 3 FTS columns#462

Merged
tobi merged 1 commit into
tobi:mainfrom
goldsr09:fix/bm25-field-weights
Mar 28, 2026
Merged

Fix BM25 field weights to include all 3 FTS columns#462
tobi merged 1 commit into
tobi:mainfrom
goldsr09:fix/bm25-field-weights

Conversation

@goldsr09

Copy link
Copy Markdown

Summary

  • The bm25() call only specified 2 weights for 3 FTS5 columns (filepath, title, body), which meant the body column implicitly received weight 0
  • This made body-only matches invisible to BM25 scoring
  • Changed from bm25(documents_fts, 10.0, 1.0) to bm25(documents_fts, 1.5, 4.0, 1.0) — title gets 4x weight, filepath gets 1.5x, body gets 1.0

Test plan

  • Added test verifying title match outranks higher body frequency
  • Existing BM25 eval tests pass (test/eval-bm25.test.ts)
  • Full test suite passes

The bm25() call only had 2 weights for 3 columns (filepath, title, body),
giving body an implicit weight of 0. Add proper weights: filepath=1.5,
title=4.0, body=1.0 so title matches are boosted and body content is scored.
zeattacker pushed a commit to zeattacker/qmd that referenced this pull request Mar 26, 2026
body column was getting weight 0, making body-only matches invisible.

Cherry-picked from tobi#462

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zeattacker pushed a commit to zeattacker/qmd that referenced this pull request Mar 26, 2026
Wraps FTS5 query in CTE to prevent query planner from abandoning
FTS5 index when combined with collection filter. Adapted with
corrected BM25 weights from tobi#462.

Cherry-picked from tobi#455

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zeattacker pushed a commit to zeattacker/qmd that referenced this pull request Mar 26, 2026
Merges dev-upstream-fixes (cherry-picked PRs tobi#462, tobi#463, tobi#455, tobi#418,
tobi#456, tobi#442, tobi#453) into dev. Resolved mcp/server.ts bind conflict —
keep 0.0.0.0 for Docker container accessibility.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@tobi tobi merged commit 08566ec into tobi:main Mar 28, 2026
tobi added a commit that referenced this pull request Mar 28, 2026
Resolve conflict: use CTE approach from #455 with updated BM25
weights (1.5, 4.0, 1.0) from #462.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jaylfc added a commit to jaylfc/qmd that referenced this pull request Apr 5, 2026
Fix BM25 field weights to include all 3 FTS columns
jaylfc added a commit to jaylfc/qmd that referenced this pull request Apr 5, 2026
Resolve conflict: use CTE approach from tobi#455 with updated BM25
weights (1.5, 4.0, 1.0) from tobi#462.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jaylfc added a commit to jaylfc/qmd that referenced this pull request Apr 5, 2026
Fix BM25 field weights to include all 3 FTS columns
jaylfc added a commit to jaylfc/qmd that referenced this pull request Apr 5, 2026
Resolve conflict: use CTE approach from tobi#455 with updated BM25
weights (1.5, 4.0, 1.0) from tobi#462.
tanarchytan referenced this pull request in tanarchytan/lotl Apr 8, 2026
Fix BM25 field weights to include all 3 FTS columns
tanarchytan referenced this pull request in tanarchytan/lotl Apr 8, 2026
Resolve conflict: use CTE approach from #455 with updated BM25
weights (1.5, 4.0, 1.0) from #462.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
lucndm pushed a commit to lucndm/qmd that referenced this pull request Jun 7, 2026
Fix BM25 field weights to include all 3 FTS columns
lucndm pushed a commit to lucndm/qmd that referenced this pull request Jun 7, 2026
Resolve conflict: use CTE approach from tobi#455 with updated BM25
weights (1.5, 4.0, 1.0) from tobi#462.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants