Summary

Add content-density-aware archival to the consolidation strategy: LLM abstractive summaries for sparse content vs verbatim text anchor extraction for dense/factual content.

Motivation

Two independent sources identify that naive summarization is catastrophically lossy on dense content:

1. Memex (arXiv:2603.04257): Dual-mode archival — LLM summaries OR verbatim text anchors (start/mid/end) preserving exact evidence (object IDs, code snippets, data) that summaries destroy
2. KV Cache Attention Matching (arXiv:2602.16284): Proved summarization is catastrophically lossy on information-dense content

Design

Density Classification

Before choosing archival mode, classify content density:

• Sparse/conversational: Use abstractive LLM summary (safe, good compression)
• Dense/factual (code, structured data, IDs): Use extractive preservation — verbatim key facts, start/mid/end anchors

Deterministic Index-Based Restore

For self-archived content, add direct ID-based restore(agent_id, entry_id) lookup instead of semantic search. Faster and lossless for own archived memories.

Integration with ConsolidationStrategy

Extend ConsolidationStrategy protocol to support mode selection. SimpleConsolidationStrategy currently does retention cleanup — add density-aware mode as a new strategy or parameter.

Affected Files

• src/ai_company/memory/consolidation/strategy.py (protocol extension)
• src/ai_company/memory/consolidation/simple_strategy.py (implement dual-mode)
• src/ai_company/memory/consolidation/service.py (pass mode through)

Research

• Memex Indexed Experience (arXiv:2603.04257)
• KV Cache Attention Matching (arXiv:2602.16284)