85.8% on LoCoMo (non-adversarial) | +23 points over raw ingestion | Single GPU, no cloud dependencies

Research benchmark testing conversational memory learning on a corrected LoCoMo dataset. Instead of ingesting conversation transcripts (the standard approach), the system learns through simulated natural dialogue with 20 character roles sharing 3,015 life facts. All evaluation is end-to-end answer accuracy — dual-graded by two independent models — not retrieval recall.

Paper: Beyond Ingestion: What Conversational Memory Learning Reveals on a Corrected LoCoMo Benchmark

• Conversation learning beats ingestion by 23 points (76.6% vs 53.0%, MiniMax-regraded, p<0.0001) — memories formed through dialogue are more precise retrieval targets than raw conversation chunks
• Architecture dominates model choice — swapping a local 20B for GPT-4o-mini changes accuracy by 1.5-2.5 points (TagCascade p=0.004, CE-Only p=0.085, MiniMax-regraded; memories created by the 20B); the retrieval architecture contributes 23+ points (p<0.0001)
• Wilson confidence scoring hurts retrieval ranking at every stage tested (p<0.001) — removed from retrieval; still visible in memory metadata (contribution not isolated)
• Tag routing helps retrieval ranking (p<0.0001) but not exam accuracy (p=0.618) — both architectures converge with 8 retrieval slots
• Poison resilience: 1,135 adversarial memories with spoofed trust signals cause only 2.6-4.2pt degradation
• 98.4% non-adversarial ceiling — the 12.6pt gap to best system (85.8%) is primarily retrieval variance, with model capability contributing a smaller effect

444 of 446 adversarial questions in the original LoCoMo dataset have no answer field — the entire adversarial category is untestable under standard grading. I added premise-rejection ground truths to enable uniform evaluation across all 5 categories. 3 non-adversarial answers requiring external domain knowledge were reverted to empty. All 10 source conversations and all 1,986 questions are unchanged.

See paper.md Section 2.4 for full details. Verified: 0/200 sampled errors (95% CI: 0-1.8%).

Retrieval: Two-lane (4 summaries + 4 facts), CE reranks top 40 candidates per lane. Lifecycle: Working -> History -> Patterns tiers with promotion, demotion, and decay. Models: gpt-oss:20b via Ollama (conversation + grading), ms-marco-MiniLM-L-6-v2 (cross-encoder, CUDA).

LoCoMo (1,986 questions, MiniMax M2.7 regraded):

Non-adversarial (1,540 questions, MiniMax M2.7):

• Best: TagCascade 20B at 85.8%
• Ceiling: 98.4% (at least one configuration correct)

# Requirements: Python 3.10+, NVIDIA GPU with 16GB+ VRAM (cross-encoder requires CUDA), Ollama
git clone https://github.com/roampal-ai/roampal-labs
cd roampal-labs
pip install -e .
ollama pull gpt-oss:20b

# Clean pipeline (conversation learning + exams, ~16 hours per strategy)
python run_pipeline.py

# Poison pipeline (inject + conversation healing + exams)
python run_poison_pipeline.py

# Model swap: GPT-4o-mini answering on existing DBs (requires OPENAI_API_KEY)
python run_model_swap.py

# Statistical tests (no GPU needed, uses saved transcripts)
python results/statistical_tests.py

# MiniMax regrading (requires MINIMAX_API_KEY)
python results/minimax_regrader.py results/exam_*.json

# Watch progress
python -m benchmark.dashboard

data/                    # LoCoMo conversations, exam questions, hard exam
  locomo_full.json       # 10 conversations + 1,986 corrected exam questions
  hard_exam.json         # 76 held-out multi-retrieval reasoning questions
  character_sheets/      # 3,015 facts across 20 character roles
  poison_memories_v2.json # 1,135 adversarial memories for poison testing
strategies/              # Memory retrieval strategies
  ce_lifecycle.py        # CE-Only and TagCascade (final strategies)
  semantic_reranker.py   # Raw ingestion baseline
benchmark/               # Runner, grader, dashboard, retrieval analysis
results/                 # Exam transcripts, MiniMax regrades, statistical tests
runs/                    # ChromaDB databases per strategy (clean + poison)
archive/                 # Phase 1 archived DBs (used in Discussion analysis)
paper.md                 # Full research paper

• Primary metric: Strict correct (ternary: correct/partial/wrong)
• Dual grading: gpt-oss:20b (live) + MiniMax M2.7 (post-hoc regrade)
• Statistical tests: McNemar's on paired per-question outcomes; 95% Wilson binomial CIs
• 5/5 categories including adversarial (published systems report 4/5)
• Per-category breakdowns for direct comparison with systems reporting different category subsets

Primary pipeline runs on a single GPU with 16GB+ VRAM (tested on NVIDIA RTX 5090, AMD Ryzen 9 5950X). Cloud APIs used for independent regrading (MiniMax M2.7) and model swap validation (GPT-4o-mini, approximately $5 total).

Apache 2.0. See LICENSE.

@article{teague2026beyond,
  title={Beyond Ingestion: What Conversational Memory Learning Reveals on a Corrected LoCoMo Benchmark},
  author={Teague, Logan},
  year={2026}
}