Skip to content

layers: add read_diary public API#3

Merged
jpwinans merged 3 commits into
mainfrom
feat/read-diary-api
May 11, 2026
Merged

layers: add read_diary public API#3
jpwinans merged 3 commits into
mainfrom
feat/read-diary-api

Conversation

@jpwinans

Copy link
Copy Markdown
Owner

Summary

Adds read_diary(agent, last_n=5, *, palace_path=None) -> list[DiaryEntry] as a public API in mempalace.layers. Encapsulates the chromadb where-filter + sort + slice logic that consumers (Vestige's runtime_orientation, primarily) were inlining via palace.get_collection.

Public surface

  • DiaryEntry (frozen dataclass) — date, filed_at, topic, content. Mirrors persistence shape of tool_diary_write / tool_diary_read in mcp_server.
  • DiaryUnavailable (exception) — raised when palace is unreachable or chromadb query fails. Distinguishes infrastructure failure from genuinely empty diary (returns []). Callers who care: DiaryUnavailable"(diary unavailable)", []"(no entries yet)".
  • read_diary — filters wing=wing_{agent.lower()} room=diary, sorts by filed_at descending, slices to last_n.

Why

Consumer (Vestige's runtime_orientation) had a # TODO: migrate to mempalace.layers.read_diary comment naming this exact API. Inlining the where-filter + metadata-key knowledge in every consumer means schema changes here cascade across consumers. Centralized in layers.py, schema knowledge stays in the package that owns the schema.

Safety

  • Side-effect-free: uses palace.get_collection (not mcp_server, which performs dup2(stderr, stdout) at import time as part of the MCP stdio protocol contract and would clobber consumer log streams).
  • Read-only: col.get(...). No writes, no HNSW touch (avoids the chromadb metadata-update pathology we hit on 2026-04-17).

Tests

10 new in tests/test_read_diary.py:

  • Sort by filed_at desc
  • last_n limit
  • Empty palace returns []
  • No entries for agent's wing returns []
  • Wing+room filter correctness
  • Case-insensitive agent name lowercase
  • last_n=0 returns []
  • DiaryUnavailable raised on get_collection failure
  • DiaryUnavailable raised on col.get failure
  • palace_path kwarg overrides config

10/10 pass. No regression to existing layers.py tests.

Consumer coordination

This PR is the upstream half of a coordinated change. The consumer migration (Vestige's runtime_orientation calling read_diary instead of inlining get_collection) ships in a separate Vestige PR — that PR depends on this one merging first so its import resolves against main. Merge order: mempalace first, then Vestige.

Vestige PR: jpwinans/vestige feat/mempalace-read-diary-api (commit e698d51).

jpwinans added 3 commits May 2, 2026 15:44
Replaces the fixed-width 800-char hard-cut chunker that produced
mid-word fragments (59% of drawers in production palace) and
indexed tool-output noise (logs, ps listings, line-numbered diffs,
file listings, truncation messages) as memory.

New chunker.py module provides:
- smart_split(text, target, ceiling): boundary-aware splitting that
  scans backward from ceiling for paragraph -> sentence -> newline ->
  word boundaries; punctuation stays with the previous chunk.
- is_excluded_content(text): line-ratio heuristics that drop chunks
  dominated by log lines, ps rows, line-numbered diff/grep output,
  arrow-redirected tool output, or standalone truncation markers.
- Code-block atomicity: triple-backtick fences are never split mid-
  listing; oversized blocks emit as one atomic chunk rather than
  fragmenting.

convo_miner._chunk_by_exchange now preserves the response's original
newlines (paragraph and code-fence structure) instead of stripping
each line and joining with single spaces.

Real-data validation on production transcripts: mid-line starts
8% -> 0%, length-cliff at 800 chars 50% -> 0%, ~75% drawer reduction
from excluding tool-output noise. 991 tests pass (+20 new chunker
tests).

Forward-only fix: existing 95K drawers are unchanged. Re-mining
those source files would apply the new chunker; deferred pending
recall-quality telemetry from new mining cycles.
Audit of the 7b27608 chunker fix found that mid-line drawers
appeared on post-commit production data at 40% rather than the
claimed 0%. Root cause: 96% of bad post-commit drawers came from a
single tool-results .txt file under .claude/projects/, where Claude
Code spills oversized tool outputs. That file contained a Python
traceback with embedded Claude Code session JSONL inlined as the
"filename" of an OSError. normalize() returned it verbatim because
none of the JSON parsers extracted messages, then chunk_exchanges
fell through to paragraph-mode and smart_split hard-cut at ceiling
because the JSON blob has no natural-language boundaries.

Two-layer fix:

(1) Source-set hygiene — palace.SKIP_DIRS gains "tool-results".
    Claude Code per-session tool-results subdirectories contain raw
    tool artifacts (JSON dumps, log captures, error tracebacks with
    inlined session metadata), not conversation content. Skipping
    the subtree at the walker level prevents the failure mode at
    the source.

(2) Defense-in-depth — chunker.is_excluded_content gains JSON-blob
    detection. Two new heuristics:
    - >=50% of non-empty lines start with `{"key":` (the canonical
      JSONL session-log shape)
    - High `,"key":` density (>=1 per 200 chars) combined with a
      transcript marker (uuid/sessionId/requestId/messageId/
      parentUuid/timestamp:YYYY-) is decisive
    Real prose mentioning a uuid or showing one inline JSON example
    survives — only blob-level density triggers exclusion.

Also: convo_miner._emit_chunks and miner.chunk_text now apply
is_excluded_content per-chunk in addition to the whole-content
check. A largely-prose response with one embedded log/JSON
paragraph drops just that chunk rather than the whole exchange.

Validation against real production sources:
- tool-results/b5k5gj8gj.txt (the actual 82-bad-drawer culprit):
  whole-file excluded, 0 chunks produced (was 82 mid-line drawers)
- real Claude Code .jsonl session: 79K -> 13K transcript via
  normalize, 11 prose chunks, 0% length-cliff @800
- prose with inline JSON example: kept as one clean chunk

Tests: 997 pass (was 991), +5 chunker exclusion cases, +1 walker
skip-dir case. Forward-only — existing 95K drawers untouched. The
82 noise drawers from the b5k5gj8gj.txt artifact remain in the
palace pending a separate selective-delete decision.
Adds a clean public API for reading an agent's diary entries from the
palace, encapsulating the chromadb where-filter + sort + slice logic
that consumers (Vestige's runtime_orientation, primarily) were
inlining via mempalace.palace.get_collection.

Surface:

  - DiaryEntry dataclass (frozen): date, filed_at, topic, content.
    Mirrors the persistence shape used by tool_diary_write /
    tool_diary_read in mcp_server.

  - DiaryUnavailable exception: raised when the palace is unreachable
    or the diary collection cannot be queried. Distinguishes
    infrastructure failure (palace missing, chromadb error) from a
    genuinely empty diary (returns []). Callers who care can render
    differently: DiaryUnavailable -> "(diary unavailable)", [] ->
    "(no entries yet)".

  - read_diary(agent, last_n=5, *, palace_path=None) -> list[DiaryEntry]:
    Filters wing=wing_{agent.lower()} room=diary, sorts by filed_at
    descending, slices to last_n.

Side-effect-free: uses palace.get_collection (not mcp_server, which
performs dup2(stderr, stdout) at import time as part of the MCP stdio
protocol contract and would clobber consumer log streams).

Consumer migration: Vestige's runtime_orientation moves from inline
chromadb logic to this API in a coordinated PR. The TODO at
runtime_orientation.py:196 was the load-bearing comment that named
this exact API; that comment goes away in the consumer migration.

Tests: 10 new in tests/test_read_diary.py covering happy-path sort/
last_n/empty/wing-filter/case-insensitive-agent, plus DiaryUnavailable
raised on get_collection failure + col.get failure, plus palace_path
override semantics.

10/10 pass. No regression to existing layers tests.
@jpwinans jpwinans merged commit 2e9005c into main May 11, 2026
0 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant