feat: add Hindsight memory integration (local memory option) by benfrank241 · Pull Request #1811 · NousResearch/hermes-agent

benfrank241 · 2026-03-17T20:07:00Z

Adds persistent long-term memory for Hermes via Hindsight, an open-source memory system for AI agents. Unlike the existing memory tool (flat markdown files), Hindsight extracts structured facts, resolves entities, builds a knowledge graph, and uses multi-strategy retrieval with cross-encoder reranking.

Three new tools:

hindsight_retain — store information with automatic fact extraction and entity resolution
hindsight_recall — multi-strategy retrieval (semantic + BM25 + graph + temporal) with reranking
hindsight_reflect — LLM-powered synthesis across stored memories

Two modes:

Cloud — Hindsight Cloud API (hindsight-client, API key required)
Local — fully on-machine via hindsight-all, which auto-starts an embedded API server backed by embedded PostgreSQL. No Hindsight account needed — just an LLM API key for fact extraction.

Tools are hidden unless configured — zero impact for users who do not set up Hindsight.

When Hindsight is active, background writes to MEMORY.md/USER.md are disabled (mirrors Honcho behaviour) so facts are not split across two stores.

Setup:

hermes hindsight setup   # interactive wizard — prompts for cloud or local mode
hermes hindsight status  # verify connection

How to test (local mode, no Docker or account needed):

pip install -e ".[hindsight-local]"   # installs hindsight-all
hermes hindsight setup                 # choose "2. Local", enter an LLM API key
hermes chat
# > remember my favourite colour is green
# > [new session]
# > whats my favourite colour?        <- answered from recall context, no tool call

How to test (cloud mode):

pip install -e ".[hindsight]"          # installs hindsight-client
hermes hindsight setup                 # choose "1. Cloud", enter Hindsight API key
hermes chat

Changes:

tools/hindsight_tools.py — three self-registering tools
hindsight_integration/client.py — HindsightClientConfig (cloud + local modes, env var fallbacks), client factory
hindsight_integration/cli.py — hermes hindsight setup|status|bank|budget commands
hindsight_integration/session.py — HindsightSessionManager (per-turn recall prefetch + context injection)
run_agent.py — Hindsight activation, tool surface rebuild, memory write disabling, _hindsight_should_activate() for both modes
model_tools.py — added hindsight_tools to tool discovery
toolsets.py — added tools to core tool list and hindsight toolset definition
pyproject.toml — hindsight and hindsight-local optional extras; hindsight_integration in packages.find
CONTRIBUTING.md — updated project structure

Platforms tested: macOS (Apple Silicon)

Registers hindsight_retain, hindsight_recall, and hindsight_reflect as built-in tools. Configured via HINDSIGHT_API_URL, HINDSIGHT_API_KEY, and HINDSIGHT_BANK_ID environment variables. Tools are only available when hindsight-client is installed and configured. Hindsight provides structured fact extraction, entity resolution, knowledge graph traversal, and cross-encoder reranking for memory retrieval. See https://github.com/vectorize-io/hindsight

nicoloboschi

LGTM

nicoloboschi · 2026-03-17T20:25:12Z

@teknium1 could you review this PR?

Adds Hindsight as a long-term memory layer alongside Honcho, mirroring the Honcho integration pattern end-to-end. Core integration: - hindsight_integration/client.py: HindsightClientConfig + client factory - hindsight_integration/session.py: HindsightSessionManager with async retain queue and background recall prefetch - hindsight_integration/cli.py: setup/status/bank/budget subcommands Agent loop (run_agent.py): - Auto-retains each turn to build the knowledge graph - Prefetches recall in background before each turn - Injects recalled context into system prompt (first turn) or user message (subsequent turns) - sync_honcho renamed to sync_memory to cover both integrations Tools (tools/hindsight_tools.py): - hindsight_retain (with optional context label) - hindsight_recall - hindsight_reflect - All tool calls run in isolated threads with fresh aiohttp sessions to avoid event loop conflicts with Hermes's asyncio runtime CLI (hermes_cli/main.py): - hermes hindsight setup/status/bank/budget toolsets.py: adds hindsight toolset entry

Adds a local Hindsight mode backed by hindsight-all/HindsightEmbedded, which auto-starts a local API server with embedded PostgreSQL. No Hindsight account or API key needed — only an LLM API key is required. - HindsightClientConfig: new fields (mode, llm_provider, llm_api_key, llm_model, llm_base_url, local_profile) with env var fallbacks (HINDSIGHT_MODE, HINDSIGHT_LLM_PROVIDER, HINDSIGHT_LLM_API_KEY, HINDSIGHT_LLM_MODEL) - get_hindsight_client(): branches on mode to return HindsightEmbedded (local) or Hindsight (cloud) - Setup wizard: mode selection as first step; local flow prompts LLM provider (numbered list with defaults), API key, model, bank ID, budget - cmd_status(): shows Mode, LLM, Profile, and data path for local mode - _hindsight_should_activate(): accepts local mode (checks llm_api_key instead of api_key) - _fresh_client() and _check_hindsight_available(): mode-aware branching - When Hindsight is active, disable background MEMORY.md/USER.md writes (mirrors Honcho behaviour) - pyproject.toml: add hindsight-local = ["hindsight-all>=0.4.18"] extra

Keep both _inject_hindsight_turn_context (our addition) and the new _should_parallelize_tool_batch / _extract_parallel_scope_path / _paths_overlap helpers from main.

Introduces MemoryProvider ABC, MemoryManager orchestrator, and BuiltinMemoryProvider for the existing MEMORY.md/USER.md system. Key design decisions: - Built-in memory is ALWAYS active, never disabled by external providers - Multiple providers can be active simultaneously - Prefetch results from all providers are merged per-turn - Sync fans out to all providers after each turn - Each provider can expose its own tools Three registration paths: 1. Built-in (BuiltinMemoryProvider) — always first, not removable 2. First-party (Honcho stays as-is for now, migration in follow-up) 3. Plugin — ctx.register_memory_provider() in plugin system Files: - agent/memory_provider.py — ABC with core + optional lifecycle hooks - agent/memory_manager.py — orchestrator, single integration point - agent/builtin_memory_provider.py — wraps existing MemoryStore - hermes_cli/plugins.py — register_memory_provider() + accessor - run_agent.py — MemoryManager wired alongside existing Honcho code - tests/agent/test_memory_provider.py — 37 tests This establishes the interface for all pending memory backend PRs (#1811 Hindsight, #2732 RetainDB, #2933 Mem0, #3499 Byterover, #3369 OpenViking, #2351 Holographic, #727 Cognitive) to implement as plugins rather than one-off integrations.

Adapts PR #1811 (Hindsight by benfrank241) and PR #2933 (Mem0 by kartik-mem0) to the MemoryProvider interface as drop-in plugins. Hindsight plugin (plugins/hindsight-memory/): - 3 tools: hindsight_retain, hindsight_recall, hindsight_reflect - Cloud (API key) or local (embedded PostgreSQL) modes - Background prefetch with thread isolation for aiohttp - Auto-sync turns to knowledge graph Mem0 plugin (plugins/mem0-memory/): - 4 tools: mem0_profile, mem0_search, mem0_context, mem0_conclude - Server-side LLM fact extraction and deduplication - Semantic search with optional reranking - Verbatim fact storage via conclude (infer=False) Both require API keys (HINDSIGHT_API_KEY / MEM0_API_KEY) and the respective SDK packages (hindsight-client / mem0ai). is_available() gates on credentials so installing the plugin without a key is safe.

Introduces MemoryProvider ABC, MemoryManager orchestrator, and BuiltinMemoryProvider for the existing MEMORY.md/USER.md system. Key design decisions: - Built-in memory is ALWAYS active, never disabled by external providers - Multiple providers can be active simultaneously - Prefetch results from all providers are merged per-turn - Sync fans out to all providers after each turn - Each provider can expose its own tools Three registration paths: 1. Built-in (BuiltinMemoryProvider) — always first, not removable 2. First-party (Honcho stays as-is for now, migration in follow-up) 3. Plugin — ctx.register_memory_provider() in plugin system Files: - agent/memory_provider.py — ABC with core + optional lifecycle hooks - agent/memory_manager.py — orchestrator, single integration point - agent/builtin_memory_provider.py — wraps existing MemoryStore - hermes_cli/plugins.py — register_memory_provider() + accessor - run_agent.py — MemoryManager wired alongside existing Honcho code - tests/agent/test_memory_provider.py — 37 tests This establishes the interface for all pending memory backend PRs (NousResearch#1811 Hindsight, NousResearch#2732 RetainDB, NousResearch#2933 Mem0, NousResearch#3499 Byterover, NousResearch#3369 OpenViking, NousResearch#2351 Holographic, NousResearch#727 Cognitive) to implement as plugins rather than one-off integrations.

Adapts PR NousResearch#1811 (Hindsight by benfrank241) and PR NousResearch#2933 (Mem0 by kartik-mem0) to the MemoryProvider interface as drop-in plugins. Hindsight plugin (plugins/hindsight-memory/): - 3 tools: hindsight_retain, hindsight_recall, hindsight_reflect - Cloud (API key) or local (embedded PostgreSQL) modes - Background prefetch with thread isolation for aiohttp - Auto-sync turns to knowledge graph Mem0 plugin (plugins/mem0-memory/): - 4 tools: mem0_profile, mem0_search, mem0_context, mem0_conclude - Server-side LLM fact extraction and deduplication - Semantic search with optional reranking - Verbatim fact storage via conclude (infer=False) Both require API keys (HINDSIGHT_API_KEY / MEM0_API_KEY) and the respective SDK packages (hindsight-client / mem0ai). is_available() gates on credentials so installing the plugin without a key is safe.

Introduces MemoryProvider ABC, MemoryManager orchestrator, and BuiltinMemoryProvider for the existing MEMORY.md/USER.md system. Key design decisions: - Built-in memory is ALWAYS active, never disabled by external providers - Multiple providers can be active simultaneously - Prefetch results from all providers are merged per-turn - Sync fans out to all providers after each turn - Each provider can expose its own tools Three registration paths: 1. Built-in (BuiltinMemoryProvider) — always first, not removable 2. First-party (Honcho stays as-is for now, migration in follow-up) 3. Plugin — ctx.register_memory_provider() in plugin system Files: - agent/memory_provider.py — ABC with core + optional lifecycle hooks - agent/memory_manager.py — orchestrator, single integration point - agent/builtin_memory_provider.py — wraps existing MemoryStore - hermes_cli/plugins.py — register_memory_provider() + accessor - run_agent.py — MemoryManager wired alongside existing Honcho code - tests/agent/test_memory_provider.py — 37 tests This establishes the interface for all pending memory backend PRs (NousResearch#1811 Hindsight, NousResearch#2732 RetainDB, NousResearch#2933 Mem0, NousResearch#3499 Byterover, NousResearch#3369 OpenViking, NousResearch#2351 Holographic, NousResearch#727 Cognitive) to implement as plugins rather than one-off integrations.

Adapts PR NousResearch#1811 (Hindsight by benfrank241) and PR NousResearch#2933 (Mem0 by kartik-mem0) to the MemoryProvider interface as drop-in plugins. Hindsight plugin (plugins/hindsight-memory/): - 3 tools: hindsight_retain, hindsight_recall, hindsight_reflect - Cloud (API key) or local (embedded PostgreSQL) modes - Background prefetch with thread isolation for aiohttp - Auto-sync turns to knowledge graph Mem0 plugin (plugins/mem0-memory/): - 4 tools: mem0_profile, mem0_search, mem0_context, mem0_conclude - Server-side LLM fact extraction and deduplication - Semantic search with optional reranking - Verbatim fact storage via conclude (infer=False) Both require API keys (HINDSIGHT_API_KEY / MEM0_API_KEY) and the respective SDK packages (hindsight-client / mem0ai). is_available() gates on credentials so installing the plugin without a key is safe.

benfrank241 force-pushed the feat/hindsight-memory-integration branch from a3031ed to 27a8b61 Compare March 17, 2026 20:08

nicoloboschi approved these changes Mar 17, 2026

View reviewed changes

benfrank241 closed this Mar 18, 2026

benfrank241 reopened this Mar 18, 2026

benfrank241 marked this pull request as draft March 18, 2026 14:43

benfrank241 added 4 commits March 18, 2026 15:06

chore: resolve merge conflict with origin/main in run_agent.py

92be1f4

Keep both _inject_hindsight_turn_context (our addition) and the new _should_parallelize_tool_batch / _extract_parallel_scope_path / _paths_overlap helpers from main.

docs: add hindsight_integration to project structure and packages.find

2b69e69

benfrank241 marked this pull request as ready for review March 18, 2026 21:14

benfrank241 changed the title ~~feat: add Hindsight memory integration~~ feat: add Hindsight memory integration (local memory option) Mar 20, 2026

benfrank241 mentioned this pull request Mar 20, 2026

Feature: Richer local memory — structured retrieval and fact extraction beyond flat files #2184

Closed

nicoloboschi mentioned this pull request Mar 24, 2026

feat: activate plugin lifecycle hooks (pre/post_llm_call, session start/end) #2823

Closed

3 tasks

teknium1 mentioned this pull request Mar 29, 2026

feat(memory): add pluggable memory provider interface #3825

Closed

This was referenced Mar 31, 2026

feat(memory): pluggable memory provider interface with profile isolation #4154

Closed

feat(memory): pluggable memory provider interface with profile isolation, review fixes, and honcho CLI restoration #4623

Merged

teknium1 closed this Apr 3, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Hindsight memory integration (local memory option)#1811

feat: add Hindsight memory integration (local memory option)#1811
benfrank241 wants to merge 5 commits into
NousResearch:mainfrom
benfrank241:feat/hindsight-memory-integration

benfrank241 commented Mar 17, 2026 •

edited

Loading

Uh oh!

nicoloboschi left a comment

Uh oh!

nicoloboschi commented Mar 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

benfrank241 commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nicoloboschi left a comment

Choose a reason for hiding this comment

Uh oh!

nicoloboschi commented Mar 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

benfrank241 commented Mar 17, 2026 •

edited

Loading