What was the issue?

The existing bm25RankToScore implementation incorrectly normalized SQLite FTS5 BM25 scores.

SQLite FTS5’s bm25() function returns negative values, where more negative values indicate better matches. However, the previous implementation clamped all negative ranks to 0 using Math.max(0, rank). As a result:

• All valid BM25 results produced the same normalized score (1.0)
• Relative BM25 relevance was completely lost
• Hybrid scoring effectively treated keyword matches as a boolean signal rather than a ranked one

This caused the text component of hybrid search to ignore actual BM25 ranking differences.

Fix

The incorrect normalization logic was replaced with a proper BM25 normalization function that:

• Correctly handles FTS5’s “smaller is better” (negative) BM25 values
• Preserves relative relevance ordering
• Produces a bounded score in the range [0, 1)

New implementation:

export function normalizeBm25Score(bm25Score: number, k = 0.5): number {
  if (!Number.isFinite(bm25Score)) return 0;
  const positive = -bm25Score;
  if (positive <= 0) return 0;
  return 1 - 1 / (1 + k * positive);
}

This function converts BM25 scores into a monotonic, saturating relevance score suitable for weighted hybrid fusion.

Further recommendations

• Add Reranker: Already implemented and sent PR feat(AI assisted): add RRF reranker for hybrid search #3184

• Use AI to generate hybrid-test.ts file
• Use AI to check value from FTS5

Greptile Overview

Greptile Summary

Fixed critical bug in BM25 scoring normalization that was causing hybrid search to lose all keyword ranking information.

The previous implementation incorrectly clamped SQLite FTS5's negative BM25 scores to zero using Math.max(0, rank), which caused all valid results to normalize to the same score (1.0). This completely eliminated the ranking signal from keyword search in hybrid results.

The new implementation properly handles FTS5's "more negative = better match" scoring by:

• Converting negative BM25 values to positive (-bm25Score)
• Applying sigmoid normalization: 1 - 1/(1 + k * positive)
• Producing bounded scores in [0, 1) that preserve relative ranking

The fix is mathematically sound and properly tested. The sigmoid function with k=0.5 provides monotonic transformation where better matches (more negative) produce higher normalized scores suitable for weighted fusion with cosine similarity scores.

Confidence Score: 5/5

• This PR is safe to merge with no risk
• The fix correctly addresses a critical bug in BM25 normalization with proper mathematical implementation, comprehensive test coverage, and clear documentation. The sigmoid transformation is appropriate for the use case and edge cases are handled correctly.
• No files require special attention