Skip to content

search: structurally-relevant files rank below lexical hits (engram benchmark MRR 0.40 vs 0.57) #537

@justrach

Description

@justrach

Auto-generated by engram codedb-insights. engram learns to re-rank codedb's results; this is where its learned policy disagrees most with codedb's lexical order, using git history as ground truth (commit subject → keyword = query, changed file = gold).

Problem

Over 30 git-derived queries (28 comparable), codedb's lexical search ranks the actually-changed file poorly: codedb MRR 0.400, vs 0.569 when re-ranked by engram's learned policy (centrality + eponymy). engram ranked the gold strictly higher on 17/28 comparable queries. 10/30 changed files weren't lexical hits at all — codedb's lexical ranker is structurally blind to them.

Evidence (codedb rank → engram rank · query → changed file)

codedb rank engram rank query changed file
17 1 correctness src/explore.zig
15 1 validation src/explore.zig
13 2 prefilter src/explore.zig
12 1 document CHANGELOG.md
10 1 daemon src/index.zig
8 1 navigation src/main.zig
6 1 symbol src/snapshot.zig
6 1 project src/cio.zig

Failing test case (red)

Per this repo's CONTRIBUTING.md ("Red-To-Green"), a failing case may be "an exact benchmark command and failing output". Reproduce from a codedb-indexed git checkout of this repo:

engram codedb-report .

Red — worst gap. Query correctness (gold = src/explore.zig, the file changed in that commit):

Manual check: codedb search "correctness" should surface src/explore.zig near the top — today it is buried at #17.

Every row in the Evidence table above is an additional failing assertion (codedb rank > engram rank). Labels are git-derived (commit subject → query, changed file → gold): noisy, but the ranks are reproducible against a fixed index.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority:p2Medium priority

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions