Skip to content

Feature: Shared Memory Pools Between Sub-Agents in Workflows (inspired by CAMEL-AI) #377

@teknium1

Description

@teknium1

Overview

In Hermes Agent's current delegate_task, sub-agents are fully isolated by design — they cannot see each other's work, access the parent's memory, or share any state. Each child gets a fresh conversation, its own terminal session, and no access to sibling results. This isolation is good for safety but creates a real problem for multi-step workflows: downstream tasks often need context from upstream tasks, and the only way to pass it is through the parent agent, which burns tokens and adds latency.

CAMEL-AI's Workforce system solves this with shared memory pools — workers in a coordinated workflow can read from (and write to) a shared knowledge base, so step 2 can directly access what step 1 discovered without routing through a coordinator.

Adding shared memory to Hermes's workflow delegation would make multi-step tasks dramatically more efficient and higher quality.

Depends on: #344 (Workflow Formulas + Multi-Agent Orchestration — provides the workflow DAG infrastructure)
Related: #375 (Inception Prompting), #376 (Debate Mode), #346 (Structured Memory System)


Research Findings

How CAMEL-AI's Shared Memory Works

In CAMEL's Workforce:

  • Workers operate in a shared context where they can access results from other workers
  • The Task object carries state (OPEN/RUNNING/DONE/FAILED) and results
  • When a downstream task's dependencies are satisfied, it receives the upstream results as additional context
  • Workers can also write to a shared memory pool that persists across the workflow

The Problem This Solves in Hermes

Current flow (wasteful):

Parent delegates "Research API" to Agent A
  → Agent A returns 5-page summary to parent
Parent reads summary, extracts relevant parts
Parent delegates "Implement client" to Agent B with extracted context
  → Agent B returns implementation to parent
Parent reads implementation
Parent delegates "Write tests" to Agent C with implementation context
  → Agent C returns tests

Every handoff goes through the parent, which must:

  1. Read the full output (consumes context window)
  2. Extract the relevant parts (risks losing important details)
  3. Reformat as context for the next agent (adds tokens)

With shared memory:

Parent creates workflow with shared scratchpad
Agent A writes research findings to shared scratchpad
Agent B reads research from scratchpad, writes implementation
Agent C reads implementation from scratchpad, writes tests
Parent gets final aggregate result

No token waste on parent relay. No information loss from summarization. Each agent gets exactly the context it needs.


Current State in Hermes Agent

Sub-agent isolation (delegate_tool.py):

  • Children get fresh AIAgent instances with empty conversation history
  • Only the goal and context strings from the parent are available
  • Children cannot access: parent memory, sibling results, shared files (beyond the filesystem)
  • Batch mode (up to 3 parallel) returns individual summaries — no cross-agent visibility

Filesystem as implicit shared state:

  • Sub-agents CAN read/write to the shared filesystem
  • In practice, this is how context gets passed today: Agent A writes to a file, Agent B reads it
  • But this is fragile — no guarantees about timing, naming conventions, or cleanup
  • And it only works for local/Docker backends, not Modal/SSH

Memory system (tools/memory_tool.py):


Implementation Plan

Skill vs. Tool Classification

This is a codebase change to tools/delegate_tool.py and potentially a new agent/shared_context.py module. It modifies core delegation mechanics — not a skill, not a standalone tool.

What We'd Need

  1. SharedContext object — A thread-safe key-value store scoped to a workflow execution
  2. Read/write tools for sub-agentscontext_read(key) and context_write(key, value) tools available to children in workflow mode
  3. Lifecycle management — Context created when workflow starts, cleaned up when it ends
  4. Scope control — Define which agents can read/write which keys (prevents all-to-all chaos)

Phased Rollout

Phase 1: Workflow Scratchpad (Depends on #344)

A simple shared scratchpad for workflow steps:

delegate_task(
    workflow=[
        {"id": "research", "goal": "Research the Stripe API", "writes": ["api_findings"]},
        {"id": "implement", "goal": "Build the payment client", "needs": ["research"], "reads": ["api_findings"]},
        {"id": "test", "goal": "Write integration tests", "needs": ["implement"], "reads": ["api_findings"]},
    ],
    shared_context=True  # Enable shared scratchpad
)
  • Each step can declare what it writes and reads
  • When step completes, its designated outputs are stored in the scratchpad
  • Downstream steps get scratchpad contents injected into their context automatically
  • Read-only by default — agents read upstream results but can't modify them

Implementation:

  • SharedContext class: thread-safe dict with get(key), set(key, value), keys()
  • Passed to child agents via a lightweight context_read tool (read-only)
  • Write happens automatically: step output is stored under the step's declared writes keys
  • Deliverable: Efficient context passing between workflow steps without parent relay

Phase 2: Active Shared Memory (Depends on Phase 1)

Allow agents to actively read and write during execution, not just at step boundaries:

# Agent A (during execution):
context_write("discovered_endpoints", ["/v1/charges", "/v1/customers", ...])
context_write("auth_method", "Bearer token via API key")

# Agent B (during execution, after A completes):
endpoints = context_read("discovered_endpoints")
auth = context_read("auth_method")

Phase 3: Cross-Workflow Memory & Persistent Shared Context


Pros & Cons

Pros

  • Eliminates token waste — No more routing all context through the parent agent
  • Reduces information loss — Downstream agents get full upstream output, not a parent's summary of it
  • Enables deeper workflows — Currently 3+ step workflows are impractical because context degrades with each relay. Shared memory removes this ceiling.
  • Thread-safe by design — Python's threading.Lock (already used in delegate_tool.py) makes concurrent access safe
  • Filesystem already provides a weak version — This formalizes what agents already do informally via file read/write

Cons / Risks

  • Breaks isolation guarantee — Current sub-agent isolation is a safety property. Shared memory creates a side-channel between agents.
  • Scope creep risk — "Shared memory" can grow into a distributed database if not constrained
  • Naming conflicts — Without namespacing, two agents might write to the same key
  • Size management — Large outputs (full file contents, long research reports) could bloat the shared context
  • Only useful with Feature: Multi-Agent Architecture — Orchestration, Cooperation, Specialized Roles & Resilient Workflows #344 — Without workflow DAGs, there's no structure to define who reads/writes what

Open Questions

  1. Should shared context be opt-in per workflow or always available? Opt-in is safer but adds friction.
  2. What's the storage backend? In-memory dict (simple, lost on crash) vs SQLite (persistent, ties into session DB) vs filesystem (visible to agents)?
  3. Size limits per key? Large outputs could bloat memory. Should there be a max size with auto-summarization?
  4. Should agents see the full scratchpad or only their declared reads? Full visibility is simpler; scoped reads are safer.
  5. How does this interact with Feature: Structured Memory System — Typed Nodes, Graph Edges, and Hybrid Search #346 (Structured Memory)? Should shared workflow context eventually merge into the structured memory graph?

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions