What does this PR do?

Closes #1539. Fixes four sites that fed unbounded content to collection.upsert(documents=[...]) or collection.add(documents=[...]). On the current mempalace/develop 95caf80 the upstream embedder accepts the oversized input and silently truncates it at its 256-token window, so the stored drawer's search vector represents only the first ~512 characters and the rest of the content becomes undiscoverable by semantic search. The same code path still hits RuntimeError: Invalid buffer size on multi-MB single-document inputs (the crash class #1534 fixed in convo_miner._chunk_by_paragraph); the four sites here were catalogued in #1539 during that audit.

Site 1: general_extractor.extract_memories (mempalace/general_extractor.py:426-444)

A 200,000-char single segment with classification markers used to become one oversized memory whose search vector represented only the first ~512 characters of the segment (and crashed embedding upsert outright on the pre-#1534 attention-buffer behavior). The fix adds a chunk_size: int = DEFAULT_CHUNK_SIZE parameter and a post-classification slicer that propagates memory_type to every slice. Reclassifying sub-slices would risk dropping data (markers may live in only one slice), so the label is approximated for sub-slices rather than re-derived. convo_miner.py:552 (the only in-tree caller) now passes chunk_size=cfg_chunk_size so MempalaceConfig.chunk_size reaches this path.

Site 2: diary_ingest.ingest_diaries (mempalace/diary_ingest.py:230-273)

A multi-MB daily diary file used to upsert as ONE document. The fix introduces per-entry drawers via the existing _split_entries(text) call, with character-chunk fallback for any single entry larger than chunk_size. New _diary_drawer_id_entry(wing, date_str, entry_idx, entry_chunk_idx) helper carries the per-entry/per-chunk position.

chunk_index in drawer metadata is a global counter across the file so searcher._expand_with_neighbors (which queries by source_file + chunk_index) stitches sibling chunks back together regardless of entry boundary.

Upsert is batched atomic per file: one drawers_col.upsert(ids=..., documents=..., metadatas=...) call carries every entry/chunk for the file, so a mid-pass embedding failure either commits the day in full or leaves no partial drawers behind. The state file is updated only after the upsert returns, so a failed batch is retried on the next run rather than silently skipped.

Auto-purge on full rebuild deletes prior-pass drawers via where={"source_file": ...}, which also migrates pre-#1539 legacy drawer_diary_ IDs as a side effect of normal use. No separate migration command needed; existing palaces auto-clean on the first ingest_diaries call that triggers a full rebuild (force=True or detected content change).

Site 3: mcp_server.tool_diary_write (mempalace/mcp_server.py:1599-1710)

The MCP tool accepted entries up to 100 KB (the sanitize_content cap) and filed them as one document. The fix splits oversized entries into bounded per-chunk drawers via a single batched col.add(ids=..., documents=..., metadatas=...) so the embedding pass is atomic (no half-written palace if the embedding model fails mid-loop). Each chunk carries parent_entry_id linkage and chunk_index metadata. Return shape is additive: chunks field is always present, chunk_ids is added on the chunked path. Single-chunk entries return chunks=1 and an entry_id that matches the stored drawer id (preserves the pre-#1539 contract).

Site 4: mcp_server.tool_add_drawer (mempalace/mcp_server.py:1103-1215)

The MCP mempalace_add_drawer tool accepted content up to 100 KB through sanitize_content and filed it as one document, reproducing the Site 3 design defect on the more common add_drawer surface. The fix mirrors Site 3: content above chunk_size is split into bounded per-chunk drawers via a single batched col.upsert(ids=..., documents=..., metadatas=...). Each chunk carries parent_drawer_id linkage and chunk_index metadata. Chunk ids use a 6-digit {drawer_id}_chunk_NNNNNN suffix so even a low-chunk_size config (down to single-digit) cannot lex-sort chunks out of order.

Idempotency on the chunked path probes both the last chunk id (its presence implies the whole batched upsert committed atomically) and the legacy logical drawer_id (so a pre-#1539 single-row write of oversized content under the same id is detected and skipped rather than getting co-resident chunk siblings on the next call).

Return shape is additive: chunks is always present; chunk_ids is added on the chunked path. Single-chunk content returns chunks=1 and a drawer_id that matches the stored drawer id (preserves the pre-#1539 contract). tool_get_drawer(drawer_id) and tool_delete_drawer(drawer_id) against the logical handle return "not found" on the chunked path; callers must iterate chunk_ids or query by parent_drawer_id metadata. The docstring on tool_add_drawer documents this contract.

Adjacent known issue, NOT closed by this PR

#390 (open) reports that the default CHUNK_SIZE = 800 exceeds the default embedding model's 256-token / ~512-char window, causing silent truncation in embeddings. That is a different failure mode than the crash this PR fixes: crash on multi-MB input vs. silent quality degradation on chunks above ~512 chars. The PR honours whatever chunk_size is configured (MempalaceConfig.chunk_size, via #1024), so users hitting the silent-truncation case today can drop chunk_size in their config. Default change and per-model binding (#442) are out of scope here.

Out-of-scope sites surfaced during the audit (filed as follow-ups, not fixed here)

• mempalace/miner.py:add_drawer legacy shim: no in-tree caller passes unbounded content; recommend a size-cap if a future external caller wires raw documents through.
• mempalace/sources/context.py:upsert_drawer interface (PalaceContext): RFC 002 adapter surface, no in-tree adapter currently constructs records of unbounded size; recommend a separate issue to enforce len(record.content) <= chunk_size at the boundary before the adapter surface stabilises.
• mempalace/sweeper.py:parse_claude_jsonl per-message content: a transcript message carrying a multi-MB tool_result block reaches collection.upsert(documents=[...]) at line 251 unbounded. Separate code path (Stop hook only), file a dedicated follow-up issue.

Thanks to @davidglidden for the upstream #1534 report that surfaced the audit.

How to test

Manual reproducers (all four exit 0 and emit CHUNK_SIZE-bounded drawers on this branch; on mempalace/develop 95caf80 the same calls also exit 0 but file the input as a single oversized drawer whose search vector represents only the first ~512 characters. Observe chunks field absent or equal to 1 on develop, versus multiple bounded chunks on this branch):

# Site 1: extract_memories
TMPDIR=$(mktemp -d)
uv run python -c "open('$TMPDIR/long.txt','w').write('We decided to migrate to PostgreSQL because performance matters. ' * 200)"
uv run python -m mempalace mine "$TMPDIR" --mode convos --extract general --wing repro-1539-extract

# Site 2: diary_ingest with multi-MB single entry
DIARY=$(mktemp -d)/2026-05-18.md
PALACE=$(mktemp -d)
uv run python -c "
content = '# 2026-05-18\n\n## 10:00\nshort entry one.\n\n## 11:00\n' + ('A' * 50000) + '\n\n## 12:00\nshort entry two.\n'
open('$DIARY','w').write(content)
"
uv run python -c "
from mempalace.diary_ingest import ingest_diaries
print(ingest_diaries('$(dirname $DIARY)', '$PALACE', wing='repro-1539-diary', force=True))
"

# Site 3: tool_diary_write via direct invocation
uv run python -c "
from mempalace.mcp_server import tool_diary_write
result = tool_diary_write(agent_name='audit', entry='Z' * 5000, wing='repro-1539-mcp')
assert result['success'] and result['chunks'] > 1
print(result)
"

# Site 4: tool_add_drawer via direct invocation
uv run python -c "
from mempalace.mcp_server import tool_add_drawer
result = tool_add_drawer(wing='repro-1539-add', room='paste', content='X' * 10000)
assert result['success'] and result.get('chunks', 1) > 1
print(result)
"

Unit suite (21 new tests across the four sites, including atomicity coverage for Site 2 and contract coverage for Site 4):

uv run pytest tests/test_general_extractor.py tests/test_closets.py::TestDiaryIngest tests/test_mcp_server.py::TestWriteTools tests/test_mcp_server.py::TestDiaryTools -v

Coverage added:

• extract_memories oversized-segment slicing with memory_type propagation + custom chunk_size plumbing
• diary_ingest per-entry drawer count, oversized-entry character chunking, file-shrink orphan purge, header-only entry, global chunk_index contiguity, incremental append idempotence, batched-upsert atomicity on embedding failure
• tool_diary_write normal vs. chunked return shape, chunk_index metadata, parent_entry_id linkage, joined verbatim equality
• tool_add_drawer normal vs. chunked return shape, chunk_index + parent_drawer_id metadata, idempotency on re-call with identical oversized content (last-chunk + legacy drawer_id probe), boundary at len == chunk_size, documented contract that tool_get_drawer / tool_delete_drawer against the logical drawer_id report "not found" on the chunked path

Checklist

• Tests pass (uv run pytest tests/ -v --ignore=tests/benchmarks)
• No hardcoded paths
• Linter passes (uvx --from 'ruff==0.15.9' ruff check . && uvx --from 'ruff==0.15.9' ruff format --check . per CI ruff pin)