Commit ddf00b4
docs: soften engram-2 '17% E2E QA' framing per research doc fact-check (#321)
Closes #319.
docs/research/2026-05-24-memory-system-benchmarks.md:215 documented
that the "17% E2E QA for MemPalace" attribution to engram-2 was not
substantiated in their published materials — what engram-2 actually
published is a ~17-point gap between their own LoCoMo score (74.5%)
and SOTA (91.7%), attributed to the answerer model.
README.md:163 and docs/ECOSYSTEM.md:45 predated that fact-check and
still stated the attribution as fact. This change adopts the 2026-05-24
doc as source-of-truth:
- README "Active investigations" reframes the entry as "End-to-end QA
measurement on the post-structural-fix palace". The corpus-shape
pathology that prompted #168 is real (pre-migration kind=content
returned 3 tokens/Q vs post-migration 1,267) and is closed; the
deliverable becomes a positive measurement rather than a rebuttal.
- docs/ECOSYSTEM.md describes what engram-2 actually published and
cross-links the research doc.
#168 itself stays open — the deliverable at notebook/data/
cat9-postmigrate-e2e/REPORT.md is unchanged. Only the framing of what
the numbers answer changes.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 3fb9428 commit ddf00b4
5 files changed
Lines changed: 247 additions & 172 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
251 | 251 | | |
252 | 252 | | |
253 | 253 | | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
254 | 289 | | |
255 | 290 | | |
256 | 291 | | |
257 | | - | |
| 292 | + | |
258 | 293 | | |
259 | 294 | | |
260 | 295 | | |
| |||
0 commit comments