Skip to content

docs: soften engram-2 '17% E2E QA' framing per research doc (closes #319)#321

Merged
jphein merged 1 commit into
mainfrom
fix/319-engram2-framing
May 28, 2026
Merged

docs: soften engram-2 '17% E2E QA' framing per research doc (closes #319)#321
jphein merged 1 commit into
mainfrom
fix/319-engram2-framing

Conversation

@jphein

@jphein jphein commented May 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • README.md "Active investigations" and docs/ECOSYSTEM.md previously stated the "engram-2 17% E2E QA for MemPalace" attribution as fact.
  • docs/research/2026-05-24-memory-system-benchmarks.md:215 documented that this attribution was unsubstantiated in engram-2's actual published materials.
  • This adopts the research doc as source-of-truth and reframes both surfaces.

Test plan

🤖 Generated with Claude Code

Closes #319.

docs/research/2026-05-24-memory-system-benchmarks.md:215 documented
that the "17% E2E QA for MemPalace" attribution to engram-2 was not
substantiated in their published materials — what engram-2 actually
published is a ~17-point gap between their own LoCoMo score (74.5%)
and SOTA (91.7%), attributed to the answerer model.

README.md:163 and docs/ECOSYSTEM.md:45 predated that fact-check and
still stated the attribution as fact. This change adopts the 2026-05-24
doc as source-of-truth:

- README "Active investigations" reframes the entry as "End-to-end QA
  measurement on the post-structural-fix palace". The corpus-shape
  pathology that prompted #168 is real (pre-migration kind=content
  returned 3 tokens/Q vs post-migration 1,267) and is closed; the
  deliverable becomes a positive measurement rather than a rebuttal.
- docs/ECOSYSTEM.md describes what engram-2 actually published and
  cross-links the research doc.

#168 itself stays open — the deliverable at notebook/data/
cat9-postmigrate-e2e/REPORT.md is unchanged. Only the framing of what
the numbers answer changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 28, 2026 19:34
@gemini-code-assist

Copy link
Copy Markdown

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@jphein jphein merged commit ddf00b4 into main May 28, 2026
10 of 12 checks passed
@jphein jphein review requested due to automatic review settings May 28, 2026 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant