Skip to content

Evaluate memory layer candidates: Mem0, Zep, Letta, Cognee, custom #39

@Aureliolo

Description

@Aureliolo

Context

The memory layer library is TBD per DESIGN_SPEC §15.2. Multiple candidates are under evaluation. Before committing to any library, we need a thorough comparative evaluation to determine the best fit for our requirements.

Candidates under evaluation:

  • Mem0 — Universal memory layer for AI agents (open-source, production-proven)
  • Graphiti — Temporal knowledge graph for AI agents (Zep's open-source successor)
  • Letta — Stateful LLM agents with memory management (formerly MemGPT)
  • Cognee — Knowledge engine for AI agents (graph-based)
  • Custom solution — Neo4j + Qdrant + FastEmbed/LiteLLM composed stack
  • Plus 12+ additional candidates discovered during open search

Evaluation Criteria

For each candidate, assess:

  • Does it support all our memory types (working, episodic, semantic, procedural, social)?
  • Retrieval quality — published benchmarks and community reputation
  • Per-agent memory isolation support
  • Fully local operation (no cloud dependencies required)
  • Memory consolidation/summarization capabilities
  • API compatibility with our abstract memory interface (§7.1-7.3)
  • Compatibility with OrgMemoryBackend protocol (§7.4) — does the library provide graph memory capabilities that could serve as a backend for organizational memory?
  • Active maintenance, community size, license compatibility (must be compatible with BUSL-1.1)
  • Python 3.14+ compatibility
  • Async support quality
  • Container architecture and operational complexity
  • Embedding provider configurability (local + cloud)

Acceptance Criteria

Decision Summary

Mem0 as initial backend (in-process, Qdrant embedded + SQLite, persistent to Docker volume) behind pluggable MemoryBackend protocol. Custom stack (Neo4j + Qdrant external) as planned future upgrade. Cognee/Letta on watch list pending Python 3.14 support.

Key findings:

  • 16+ candidates evaluated, 3 passed all gates (Mem0, Graphiti, Custom Stack)
  • Letta and Cognee eliminated by Python <3.14 constraint (conservative bounds, not technical — on watch list)
  • Supermemory eliminated: proprietary engine, SDK-only open source
  • Mem0 chosen as initial for production-readiness (v1.0+, 49k stars) and low setup cost
  • Protocol architecture ensures any backend can be swapped in later via config

Dependencies

Design Spec Reference

  • §7 — Memory System (§7.1-7.3 individual agent memory, §7.4 shared org memory)
  • §15.2 — Technology Stack (updated with decision)

Updated 2026-03-08: Evaluation complete. ADR-001 accepted. Acceptance criteria revised to match research-based evaluation approach (prototypes and latency benchmarks deferred — unnecessary given pragmatic decision to use Mem0 as initial backend with protocol-based swappability).

Metadata

Metadata

Assignees

No one assigned

    Labels

    prio:highImportant, should be prioritizedscope:medium1-3 days of workspec:memoryDESIGN_SPEC Section 7 - Memory & Persistencetype:researchEvaluate options, make tech decisions

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions