Skip to content

[Bug] recallScoreThreshold bypassed in pickMemoriesForInjection function #1106

@Suidge

Description

@Suidge

[Bug] recallScoreThreshold bypassed in pickMemoriesForInjection function

Description

The recallScoreThreshold configuration is being bypassed during memory recall, causing low-scoring memories (below threshold) to be injected into context.

Evidence

  • Config: recallScoreThreshold: 0.25
  • Observed: Retrieved memories include scores as low as 0.11 (below threshold)
  • Expected: All returned memories should have score ≥ 0.25

Root Cause Analysis

The issue is in memory-ranking.ts, specifically in the pickMemoriesForInjection function.

Flow:

  1. postProcessMemories correctly filters memories by threshold (line 49):

    if (clampScore(item.score) < options.scoreThreshold) {
      continue; // ✓ Correct filtering
    }
  2. However, pickMemoriesForInjection (line 221) bypasses this filtering when trying to fill up to the limit:

    const leaves = deduped.filter((item) => isLeafLikeMemory(item));
    if (leaves.length >= limit) {
      return leaves.slice(0, limit);  // ✗ No threshold check
    }
    
    const picked = [...leaves];
    const used = new Set(leaves.map((item) => item.uri));
    for (const item of deduped) {
      if (picked.length >= limit) {
        break;
      }
      if (used.has(item.uri)) {
        continue;
      }
      picked.push(item);  // ✗ Adds items without threshold check!
    }
    return picked;

Problem:

When there aren't enough "leaf-like" memories (level === 2), the function supplements from the general deduped array without re-checking the score threshold. This allows low-scoring memories to pass through.

Proposed Fix

Add threshold validation in pickMemoriesForInjection:

for (const item of deduped) {
  if (picked.length >= limit) {
    break;
  }
  if (used.has(item.uri)) {
    continue;
  }
  // ADD: Re-check threshold before adding
  if (clampScore(item.score) < options.scoreThreshold) {
    continue;
  }
  picked.push(item);
}

Note: The function signature would need to accept scoreThreshold as a parameter, or use a default value.

Impact

  • Low-quality/irrelevant memories are being injected into agent context
  • Wastes token budget on low-scoring results
  • Reduces overall recall quality

Environment

  • OpenViking: v0.2.15
  • Plugin installation: path-based (/Users/neoshi/.openviking/plugins/openviking)
  • Config mode: remote

Additional Context

This was discovered during investigation of recall quality after disabling rerank. Statistics from 14 queries:

  • Average score: 0.2827
  • Score range: 0.1100 - 0.7222 (minimum below 0.25 threshold)
  • Zero-result rate: 0%

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status
    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions