Cut your AI agent's token spend by 50–97%.
One command compresses your entire workspace — memory files, session transcripts, sub-agent context — using 6 layered techniques, from deterministic rule-engines to a real-time LLM-driven memory system called Engram.

"Cut your tokens. Keep your facts."

Running long-lived AI agents is expensive. Context windows fill up. Memory files bloat. Session transcripts grow to megabytes. Claw Compactor solves all three:

• 5 deterministic layers — zero LLM cost, instant, fully reversible
• Layer 6: Engram — the real magic: a live, LLM-powered memory engine that observes conversations as they happen and compresses them into structured, priority-annotated knowledge
• ~97% savings on raw session transcripts
• One command (full) to run everything

# 1. Clone
git clone https://github.com/aeromomo/claw-compactor.git
cd claw-compactor

# 2. Dry-run benchmark (non-destructive, no side effects)
python3 scripts/mem_compress.py /path/to/workspace benchmark

# 3. Compress everything
python3 scripts/mem_compress.py /path/to/workspace full

Requirements: Python 3.9+. No mandatory dependencies.
Optional: pip install tiktoken for exact token counts (falls back to CJK-aware heuristic).

For Engram (Layer 6): configure engram.yaml (or set ANTHROPIC_API_KEY / OPENAI_API_KEY env vars).

┌─────────────────────────────────────────────────────────────────────┐
│                    mem_compress.py                                  │
│              (unified CLI entry point / dispatcher)                 │
└───┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┬───────┘
    │      │      │      │      │      │      │      │      │
    ▼      ▼      ▼      ▼      ▼      ▼      ▼      ▼      ▼
estimate compress dict  dedup observe tiers audit optimize engram
    │      │      │      │      │      │      │      │      │
    └──────┴──────┴──────┴──────┴──────┴──────┴──────┘      │
                          │                                   │
                          ▼                                   ▼
             ┌────────────────────────┐        ┌─────────────────────────┐
             │   scripts/lib/         │        │   ENGRAM ENGINE         │
             │                        │        │  (Layer 6, real-time)   │
             │  tokens.py             │        │                         │
             │    ↳ tiktoken fallback │        │  engram.py              │
             │  markdown.py           │        │    ↳ EngramEngine       │
             │    ↳ section parsing   │        │    ↳ Observer Agent     │
             │  dedup.py              │        │    ↳ Reflector Agent    │
             │    ↳ shingle hashing   │        │                         │
             │  dictionary.py         │        │  engram_storage.py      │
             │    ↳ $XX codebook      │        │    ↳ atomic file I/O    │
             │  rle.py                │        │    ↳ JSONL pending log  │
             │    ↳ path/IP/enum      │        │                         │
             │  tokenizer_optimizer   │        │  engram_prompts.py      │
             │    ↳ format fixes      │        │    ↳ Observer prompt    │
             │  config.py             │        │    ↳ Reflector prompt   │
             │  exceptions.py         │        └─────────────────────────┘
             └────────────────────────┘

Engram Memory Layout (per thread / 每个会话线程):
  memory/engram/{thread_id}/
    ├── pending.jsonl     ← raw messages buffer (auto-cleared after observe)
    ├── observations.md   ← Observer output, append-only structured log
    ├── reflections.md    ← Reflector output, compressed long-term memory
    └── meta.json         ← timestamps, token counts

*Lossy techniques preserve all facts and decisions; only verbose formatting is removed.
†Layer 6 requires ANTHROPIC_API_KEY or OPENAI_API_KEY.

Engram is the flagship feature of Claw Compactor. While the first 5 layers batch-process static files, Engram works live — it watches conversations as they unfold and continuously compresses them into a structured, priority-annotated knowledge base.

  Incoming Messages
        │
        ▼
  ┌─────────────┐       threshold exceeded
  │  pending    │  ──────────────────────►  ┌─────────────────┐
  │  queue      │       (30K tokens)         │  Observer LLM   │
  │  (JSONL)    │                            │  Agent          │
  └─────────────┘                            │                 │
                                             │  Raw messages   │
                                             │  → structured   │
                                             │  observation    │
                                             │  log 🔴🟡🟢     │
                                             └────────┬────────┘
                                                      │
                                             append to observations.md
                                             clear pending queue
                                                      │
                                                      ▼
                                          observation_tokens > 40K?
                                                      │
                                                      ▼ yes
                                             ┌─────────────────┐
                                             │  Reflector LLM  │
                                             │  Agent          │
                                             │                 │
                                             │  All observations│
                                             │  → compressed   │
                                             │  long-term      │
                                             │  reflection     │
                                             └────────┬────────┘
                                                      │
                                             overwrite reflections.md

Auto-triggering happens every time add_message() is called:

1. Observer fires when pending.jsonl exceeds OM_OBSERVER_THRESHOLD (default: 30,000 tokens)
2. Reflector fires when observations.md exceeds OM_REFLECTOR_THRESHOLD (default: 40,000 tokens)

Both thresholds can be tuned per-session — set lower for faster compression, higher to batch more context.

For real-time streaming integration, run Engram as a daemon that reads JSONL from stdin:

export ANTHROPIC_API_KEY=sk-ant-...

# Start daemon — pipe messages in real-time
python3 scripts/engram_cli.py /workspace daemon --thread live-session

# In another process, pipe messages:
echo '{"role":"user","content":"Building OpenCompress today"}' | \
    python3 scripts/engram_cli.py /workspace daemon --thread live-session

# Control commands (also via stdin):
echo '{"__cmd":"observe"}'  # force observe now
echo '{"__cmd":"reflect"}'  # force reflect now
echo '{"__cmd":"status"}'   # print thread status
echo '{"__cmd":"quit"}'     # exit daemon

Observations use a priority-annotated, bilingual (EN/中文) format:

Date: 2026-03-05
- 🔴 12:10 User building OpenCompress; deadline within one week / 用户在构建 OpenCompress，deadline 一周内
  - 🔴 12:10 Using ModernBERT-large for inference / 使用 ModernBERT-large 做推理
  - 🟡 12:12 Discussed training data annotation strategy / 讨论了训练数据标注策略
  - 🟢 12:15 Mentioned benchmark results are promising / 提到 benchmark 结果不错
- 🟡 12:30 Switched to deployment pipeline discussion on M3 Ultra
- 🟢 12:45 User prefers concise, structured replies

Priority legend (优先级):

• 🔴 Critical — user goals, hard deadlines, blockers, key decisions (never dropped)
• 🟡 Important — technical details, ongoing work, significant context
• 🟢 Useful — background info, mentions, soft context (pruned after 7 days if unreferenced)

# Set API key (Anthropic preferred / OpenAI also supported)
export ANTHROPIC_API_KEY=sk-ant-...

# Check all thread statuses
python3 scripts/engram_cli.py /workspace status

# Add a message (auto-triggers observe/reflect at threshold)
python3 -c "
from scripts.lib.engram import EngramEngine
eng = EngramEngine('/path/to/workspace')
eng.add_message('my-thread', role='user', content='Building OpenCompress today')
ctx = eng.build_system_context('my-thread')
print(ctx)
"

# Force observe for a thread
python3 scripts/engram_cli.py /workspace observe --thread my-thread

# Force reflect
python3 scripts/engram_cli.py /workspace reflect --thread my-thread

# Import conversation from JSONL file
python3 scripts/engram_cli.py /workspace ingest \
    --thread my-thread --input conversation.jsonl

# Print injectable system context
python3 scripts/engram_cli.py /workspace context --thread my-thread

# Via unified entry point (mem_compress.py)
python3 scripts/mem_compress.py /workspace engram status
python3 scripts/mem_compress.py /workspace engram observe --thread my-thread

The recommended way to run Engram in production is via the auto-mode scripts, which automatically detect all active threads, process them concurrently, and apply observe/reflect based on threshold triggers.

# Single run — auto-detects all threads in workspace
python3 scripts/engram_auto.py --workspace ~/.openclaw/workspace

# Via shell wrapper (same, with logging)
bash scripts/engram-auto.sh

# CLI equivalents
python3 scripts/engram_cli.py <workspace> auto --config engram.yaml
python3 scripts/engram_cli.py <workspace> status --thread openclaw-main
python3 scripts/engram_cli.py <workspace> observe --thread openclaw-main
python3 scripts/engram_cli.py <workspace> reflect --thread openclaw-main

Auto-detected thread sources:

• Discord channels: #general, #open-compress, #aimm, and others
• OpenClaw internals: cron, subagent, openclaw-main
• Any JSONL session files found in sessions.scan_dir

Concurrency: Uses ThreadPoolExecutor with 4 workers (configurable via concurrency.max_workers in engram.yaml) for parallel thread processing.

Retry mechanism: All LLM HTTP calls use automatic exponential-backoff retry:

• Retries on: 429, 500, 502, 503, 504
• Delays: 2s → 4s → 8s (max 3 attempts)
• No retry on: 400, 401, 403 (configuration errors — fail fast)

The preferred way to configure Engram is via engram.yaml in the project root:

llm:
  provider: openai-compatible        # "anthropic" or "openai-compatible"
  base_url: http://localhost:8403    # endpoint for openai-compatible provider
  api_key_env: OPENAI_API_KEY        # env var holding the API key
  model: claude-code/sonnet          # model identifier
  max_tokens: 4096                   # max output tokens per LLM call

threads:
  default:
    observer_threshold: 30000        # pending-message tokens before Observer fires
    reflector_threshold: 40000       # observation tokens before Reflector fires

  # Optional per-thread overrides:
  # discord-general:
  #   observer_threshold: 20000

sessions:
  scan_dir: ~/.openclaw/agents/main/sessions   # where to find session JSONL files
  max_age_hours: 48                            # ignore sessions older than this

storage:
  base_dir: ~/.openclaw/workspace/memory/engram  # Engram memory root

concurrency:
  max_workers: 4                     # parallel thread workers

logging:
  level: INFO

Environment variables (alternative to engram.yaml):

ANTHROPIC_API_KEY=sk-ant-...        # Preferred LLM provider (Anthropic)
OPENAI_API_KEY=sk-...               # OpenAI-compatible API key
OPENAI_BASE_URL=https://...         # Custom endpoint (local LLM, etc.)
OM_OBSERVER_THRESHOLD=30000         # Tokens before auto-observe (default: 30000)
OM_REFLECTOR_THRESHOLD=40000        # Tokens before auto-reflect (default: 40000)
OM_MODEL=claude-opus-4-5            # Override LLM model

Thresholds control how often Observer/Reflector fire. Lower = more frequent (fresher memory, higher LLM cost). Higher = less frequent (lower cost, slightly stale context).

Real-world channel token production (measured):

LLM cost at different thresholds (each Observer call ≈ 2K output tokens, Sonnet):

Recommendation: Start at observer_threshold: 30000. Tune down if context feels stale; tune up to reduce LLM spend. Reflector threshold can stay at 40000 (fires less often).

Python API:

from scripts.lib.engram import EngramEngine

engine = EngramEngine(
    workspace_path="/path/to/workspace",
    observer_threshold=30_000,       # tune lower for fresher context
    reflector_threshold=40_000,
    anthropic_api_key="sk-ant-...",  # or set env var
    model="claude-opus-4-5",         # optional model override
)

# Add messages — auto-triggers observe/reflect as needed
status = engine.add_message("thread-1", role="user", content="Hello!")
# status = {"observed": bool, "reflected": bool, "pending_tokens": int, ...}

# Build injectable context string for system prompt
ctx_str = engine.build_system_context("thread-1")

python3 scripts/mem_compress.py <workspace> <command> [options]

Global options:

• --json — Machine-readable JSON output
• --dry-run — Preview changes without writing
• --since YYYY-MM-DD — Filter sessions by date
• --auto-merge — Auto-merge duplicates (dedup)

Real-world compression results on OpenClaw agent workspaces:

Combined effect: 50% token reduction + prompt caching (90% cost discount) = ~95% effective cost reduction on repeated context.

Averaged across 5 conversation samples (DevOps, trading, ML, mixed-long, sysadmin):

Key insights:

• Engram ROUGE-L is low because it semantically restructures rather than copying verbatim — it preserves intent, not wording
• RuleCompressor (L1–5) is best for instant prompt compression (zero latency, zero LLM cost)
• Engram is best for long-term memory compression (87.5% space savings, meaning you keep 8× more history in the same token budget)
• Full per-sample results → benchmark/RESULTS.md

Claw Compactor ships as an installable OpenClaw skill. When installed, the agent runs compression automatically:

python3 scripts/mem_compress.py <workspace> auto

This: compresses all workspace files, tracks token counts between runs, and reports savings at session start.

## Memory Maintenance (weekly)
- Run: python3 skills/claw-compactor/scripts/mem_compress.py <workspace> benchmark
- If savings > 5%: run full pipeline
- If pending transcripts: run observe

Cron example (weekly, Sunday 3am):

0 3 * * 0 cd /path/to/skills/claw-compactor && \
  python3 scripts/mem_compress.py /path/to/workspace full

Enable prompt caching for a 90% discount on cached tokens:

{
  "agents": {
    "defaults": {
      "models": {
        "anthropic/claude-opus-4-6": {
          "params": { "cacheRetention": "long" }
        }
      }
    }
  }
}

Compression reduces token count; caching reduces cost per token.
Together: 50% compression × 90% cache discount = 95% effective cost reduction.

Optional claw-compactor-config.json in workspace root:

{
  "chars_per_token": 4,
  "level0_max_tokens": 200,
  "level1_max_tokens": 500,
  "dedup_similarity_threshold": 0.6,
  "dedup_shingle_size": 3
}

All fields optional — sensible defaults apply when absent.

Q: Will compression lose my data?
A: Rule engine, dictionary, RLE, and tokenizer optimization are fully lossless. Observation compression (Layer 3) and Engram (Layer 6) are lossy but preserve all facts and decisions — only verbose formatting is removed.

Q: How does dictionary decompression work?
A: decompress_text(text, codebook) expands all $XX codes back. The codebook JSON must be present (memory/.codebook.json).

Q: Can I run individual steps?
A: Yes — every command is independent: compress, dict, observe, tiers, dedup, optimize, engram.

Q: What if tiktoken isn't installed?
A: Falls back to a CJK-aware heuristic (chars ÷ 4). Results are ~90% accurate.

Q: Does it handle Chinese/Japanese/Unicode?
A: Yes. Full CJK support including character-aware token estimation and Chinese punctuation normalization.

Q: How does Engram differ from just running observe?
A: observe (Layer 3) is a batch processor for static JSONL files. Engram is a live engine: it works on any messages, triggers automatically, adds a Reflector stage for long-term distillation, and outputs structured priority logs.

Q: Can I use Engram with a local LLM?
A: Yes — set OPENAI_BASE_URL to your local OpenAI-compatible endpoint (e.g., LM Studio, Ollama).

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Ensure tests pass: python3 -m pytest tests/ -v
4. Commit with a clear message
5. Open a Pull Request

• Inspired by claude-mem by thedotmack
• Built by Bot777 for OpenClaw

MIT — see LICENSE for details.