-
Notifications
You must be signed in to change notification settings - Fork 0
Evaluate memory layer candidates: Mem0, Zep, Letta, Cognee, custom #39
Description
Context
The memory layer library is TBD per DESIGN_SPEC §15.2. Multiple candidates are under evaluation. Before committing to any library, we need a thorough comparative evaluation to determine the best fit for our requirements.
Candidates under evaluation:
- Mem0 — Universal memory layer for AI agents (open-source, production-proven)
- Graphiti — Temporal knowledge graph for AI agents (Zep's open-source successor)
- Letta — Stateful LLM agents with memory management (formerly MemGPT)
- Cognee — Knowledge engine for AI agents (graph-based)
- Custom solution — Neo4j + Qdrant + FastEmbed/LiteLLM composed stack
- Plus 12+ additional candidates discovered during open search
Evaluation Criteria
For each candidate, assess:
- Does it support all our memory types (working, episodic, semantic, procedural, social)?
- Retrieval quality — published benchmarks and community reputation
- Per-agent memory isolation support
- Fully local operation (no cloud dependencies required)
- Memory consolidation/summarization capabilities
- API compatibility with our abstract memory interface (§7.1-7.3)
- Compatibility with
OrgMemoryBackendprotocol (§7.4) — does the library provide graph memory capabilities that could serve as a backend for organizational memory? - Active maintenance, community size, license compatibility (must be compatible with BUSL-1.1)
- Python 3.14+ compatibility
- Async support quality
- Container architecture and operational complexity
- Embedding provider configurability (local + cloud)
Acceptance Criteria
- Open discovery: broad search beyond spec-listed candidates (16+ found)
- Gate checks: pass/fail on local-first, license, Docker, Python 3.14, isolation, embeddings
- Comparison table: all gate-passing candidates scored against weighted criteria (S1-S11, 100 points)
- Decision documented with rationale: ADR-001 in
docs/decisions/ADR-001-memory-layer.md - Evaluate whether selected candidate can serve §7.4 Shared Organizational Memory backends
- Downstream impact documented for Design individual agent memory interface: working, episodic, semantic, procedural (DESIGN_SPEC §7.1-7.3) #32 (memory interface), Implement pluggable PersistenceBackend protocol with SQLite backend (DESIGN_SPEC §7.5) #36 (persistence), Implement shared organizational memory with OrgMemoryBackend protocol (DESIGN_SPEC §7.4) #125 (org memory)
- DESIGN_SPEC.md §15.2 updated to reflect decision
- Backend swappability strategy: protocol-based design allows future backend swaps via config
Decision Summary
Mem0 as initial backend (in-process, Qdrant embedded + SQLite, persistent to Docker volume) behind pluggable MemoryBackend protocol. Custom stack (Neo4j + Qdrant external) as planned future upgrade. Cognee/Letta on watch list pending Python 3.14 support.
Key findings:
- 16+ candidates evaluated, 3 passed all gates (Mem0, Graphiti, Custom Stack)
- Letta and Cognee eliminated by Python
<3.14constraint (conservative bounds, not technical — on watch list) - Supermemory eliminated: proprietary engine, SDK-only open source
- Mem0 chosen as initial for production-readiness (v1.0+, 49k stars) and low setup cost
- Protocol architecture ensures any backend can be swapped in later via config
Dependencies
- Depends on memory system interface (Design individual agent memory interface: working, episodic, semantic, procedural (DESIGN_SPEC §7.1-7.3) #32)
Design Spec Reference
- §7 — Memory System (§7.1-7.3 individual agent memory, §7.4 shared org memory)
- §15.2 — Technology Stack (updated with decision)
Updated 2026-03-08: Evaluation complete. ADR-001 accepted. Acceptance criteria revised to match research-based evaluation approach (prototypes and latency benchmarks deferred — unnecessary given pragmatic decision to use Mem0 as initial backend with protocol-based swappability).