layers: add read_diary public API#3
Merged
Merged
Conversation
Replaces the fixed-width 800-char hard-cut chunker that produced mid-word fragments (59% of drawers in production palace) and indexed tool-output noise (logs, ps listings, line-numbered diffs, file listings, truncation messages) as memory. New chunker.py module provides: - smart_split(text, target, ceiling): boundary-aware splitting that scans backward from ceiling for paragraph -> sentence -> newline -> word boundaries; punctuation stays with the previous chunk. - is_excluded_content(text): line-ratio heuristics that drop chunks dominated by log lines, ps rows, line-numbered diff/grep output, arrow-redirected tool output, or standalone truncation markers. - Code-block atomicity: triple-backtick fences are never split mid- listing; oversized blocks emit as one atomic chunk rather than fragmenting. convo_miner._chunk_by_exchange now preserves the response's original newlines (paragraph and code-fence structure) instead of stripping each line and joining with single spaces. Real-data validation on production transcripts: mid-line starts 8% -> 0%, length-cliff at 800 chars 50% -> 0%, ~75% drawer reduction from excluding tool-output noise. 991 tests pass (+20 new chunker tests). Forward-only fix: existing 95K drawers are unchanged. Re-mining those source files would apply the new chunker; deferred pending recall-quality telemetry from new mining cycles.
Audit of the 7b27608 chunker fix found that mid-line drawers appeared on post-commit production data at 40% rather than the claimed 0%. Root cause: 96% of bad post-commit drawers came from a single tool-results .txt file under .claude/projects/, where Claude Code spills oversized tool outputs. That file contained a Python traceback with embedded Claude Code session JSONL inlined as the "filename" of an OSError. normalize() returned it verbatim because none of the JSON parsers extracted messages, then chunk_exchanges fell through to paragraph-mode and smart_split hard-cut at ceiling because the JSON blob has no natural-language boundaries. Two-layer fix: (1) Source-set hygiene — palace.SKIP_DIRS gains "tool-results". Claude Code per-session tool-results subdirectories contain raw tool artifacts (JSON dumps, log captures, error tracebacks with inlined session metadata), not conversation content. Skipping the subtree at the walker level prevents the failure mode at the source. (2) Defense-in-depth — chunker.is_excluded_content gains JSON-blob detection. Two new heuristics: - >=50% of non-empty lines start with `{"key":` (the canonical JSONL session-log shape) - High `,"key":` density (>=1 per 200 chars) combined with a transcript marker (uuid/sessionId/requestId/messageId/ parentUuid/timestamp:YYYY-) is decisive Real prose mentioning a uuid or showing one inline JSON example survives — only blob-level density triggers exclusion. Also: convo_miner._emit_chunks and miner.chunk_text now apply is_excluded_content per-chunk in addition to the whole-content check. A largely-prose response with one embedded log/JSON paragraph drops just that chunk rather than the whole exchange. Validation against real production sources: - tool-results/b5k5gj8gj.txt (the actual 82-bad-drawer culprit): whole-file excluded, 0 chunks produced (was 82 mid-line drawers) - real Claude Code .jsonl session: 79K -> 13K transcript via normalize, 11 prose chunks, 0% length-cliff @800 - prose with inline JSON example: kept as one clean chunk Tests: 997 pass (was 991), +5 chunker exclusion cases, +1 walker skip-dir case. Forward-only — existing 95K drawers untouched. The 82 noise drawers from the b5k5gj8gj.txt artifact remain in the palace pending a separate selective-delete decision.
Adds a clean public API for reading an agent's diary entries from the
palace, encapsulating the chromadb where-filter + sort + slice logic
that consumers (Vestige's runtime_orientation, primarily) were
inlining via mempalace.palace.get_collection.
Surface:
- DiaryEntry dataclass (frozen): date, filed_at, topic, content.
Mirrors the persistence shape used by tool_diary_write /
tool_diary_read in mcp_server.
- DiaryUnavailable exception: raised when the palace is unreachable
or the diary collection cannot be queried. Distinguishes
infrastructure failure (palace missing, chromadb error) from a
genuinely empty diary (returns []). Callers who care can render
differently: DiaryUnavailable -> "(diary unavailable)", [] ->
"(no entries yet)".
- read_diary(agent, last_n=5, *, palace_path=None) -> list[DiaryEntry]:
Filters wing=wing_{agent.lower()} room=diary, sorts by filed_at
descending, slices to last_n.
Side-effect-free: uses palace.get_collection (not mcp_server, which
performs dup2(stderr, stdout) at import time as part of the MCP stdio
protocol contract and would clobber consumer log streams).
Consumer migration: Vestige's runtime_orientation moves from inline
chromadb logic to this API in a coordinated PR. The TODO at
runtime_orientation.py:196 was the load-bearing comment that named
this exact API; that comment goes away in the consumer migration.
Tests: 10 new in tests/test_read_diary.py covering happy-path sort/
last_n/empty/wing-filter/case-insensitive-agent, plus DiaryUnavailable
raised on get_collection failure + col.get failure, plus palace_path
override semantics.
10/10 pass. No regression to existing layers tests.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
read_diary(agent, last_n=5, *, palace_path=None) -> list[DiaryEntry]as a public API inmempalace.layers. Encapsulates the chromadb where-filter + sort + slice logic that consumers (Vestige'sruntime_orientation, primarily) were inlining viapalace.get_collection.Public surface
DiaryEntry(frozen dataclass) —date,filed_at,topic,content. Mirrors persistence shape oftool_diary_write/tool_diary_readinmcp_server.DiaryUnavailable(exception) — raised when palace is unreachable or chromadb query fails. Distinguishes infrastructure failure from genuinely empty diary (returns[]). Callers who care:DiaryUnavailable→"(diary unavailable)",[]→"(no entries yet)".read_diary— filterswing=wing_{agent.lower()} room=diary, sorts byfiled_atdescending, slices tolast_n.Why
Consumer (Vestige's
runtime_orientation) had a# TODO: migrate to mempalace.layers.read_diarycomment naming this exact API. Inlining the where-filter + metadata-key knowledge in every consumer means schema changes here cascade across consumers. Centralized inlayers.py, schema knowledge stays in the package that owns the schema.Safety
palace.get_collection(notmcp_server, which performsdup2(stderr, stdout)at import time as part of the MCP stdio protocol contract and would clobber consumer log streams).col.get(...). No writes, no HNSW touch (avoids the chromadb metadata-update pathology we hit on 2026-04-17).Tests
10 new in
tests/test_read_diary.py:filed_atdesclast_nlimit[][]last_n=0returns[]DiaryUnavailableraised onget_collectionfailureDiaryUnavailableraised oncol.getfailurepalace_pathkwarg overrides config10/10 pass. No regression to existing
layers.pytests.Consumer coordination
This PR is the upstream half of a coordinated change. The consumer migration (Vestige's
runtime_orientationcallingread_diaryinstead of inliningget_collection) ships in a separate Vestige PR — that PR depends on this one merging first so its import resolves againstmain. Merge order: mempalace first, then Vestige.Vestige PR: jpwinans/vestige feat/mempalace-read-diary-api (commit e698d51).