Symptom

On a freshly patched install where hnsw:num_threads: 1 is passed via metadata= to get_or_create_collection / create_collection (the fix in #991 and part of #976), the guard still catches sparse link_lists.bin bloat within an hour, and the kernel log records segfaults in chromadb_rust_bindings.abi3.so whenever multiple processes touch the palace concurrently:

kernel: tokio-rt-worker[229244]: segfault at 88 ip ... in chromadb_rust_bindings.abi3.so[32d4...]
kernel: tokio-rt-worker[230410]: segfault at 88 ip ... in chromadb_rust_bindings.abi3.so[32d4...]

mempalace-disk-monitor.sh: HNSW sparse bloat: .../link_lists.bin claims 1337GB. Truncating (chroma rebuilds).

Environment

• OS: Ubuntu 24.04 (GCP VM)
• Python: 3.12
• chromadb: 1.5.8
• mempalace: 3.3.2 (pip --user --break-system-packages)
• Used concurrently by: Claude Code Stop/PreCompact hooks (per turn), a cron health-check (every 30 min), an hourly disk-guard timer, and the MCP server.

Reproduction

Restore a palace from backup SQLite, then run these concurrently (e.g. via threading.Thread spawning subprocess.run):

subprocess.run(["python3", "-m", "mempalace", "mine", small_dir_A])
subprocess.run(["python3", "-m", "mempalace", "mine", small_dir_B])
subprocess.run(["python3", "-m", "mempalace", "mine", small_dir_C])

Observe:

• link_lists.bin grows to sparse >100GB apparent size within a few minutes
• journalctl -k | grep segfault.*chromadb shows crashes
• Each subsequent mine may fail with InternalError: Error constructing hnsw segment reader: Error deserializing pickle file: eval error at offset 0: EOF while parsing (truncated by the guard)

Root causes

1. chromadb 1.5.x does NOT persist modified configuration_json["hnsw"] across PersistentClient reopens. Setting hnsw:num_threads: 1 at collection-creation time is effectively a one-shot — the next process that opens the palace reads the persisted schema (which lacks the override) and chromadb reverts to the default parallel HNSW insert path. This is the observation that PR fix: HNSW graph corruption, PreCompact deadlock, mine fan-out (closes #974, #965, #955) #976 documents in its _pin_hnsw_threads docstring, and it matches the behavior I see on 3.3.2 after patches 1-3.

2. Metadata-only retrofit at the SQLite layer isn't enough for the same reason: I attempted a direct INSERT into collection_metadata to record hnsw:num_threads=1, and chromadb read the key back fine (collection.metadata shows it), but HNSW sparse growth still occurred — the runtime config is what governs the actual worker pool, and that lives in memory, not in the persisted schema.

3. Concurrent-writer races. Even when pinning is applied correctly, having multiple processes open the palace simultaneously still triggers tokio-rt-worker segfaults during chromadb shutdown (visible in kernel log). A palace-wide serialization mechanism is needed on top of the thread pinning.

What's needed

• Runtime retrofit for num_threads=1 on every get_collection / create_collection, not just creation — covers existing palaces restored from backup. (Same approach as _pin_hnsw_threads in fix: HNSW graph corruption, PreCompact deadlock, mine fan-out (closes #974, #965, #955) #976.)
• Palace-wide write lock to serialize concurrent writers. The hook path (mempalace hook run --hook stop) is latency-sensitive and data-sensitive: a non-blocking skip (as in fix: HNSW graph corruption, PreCompact deadlock, mine fan-out (closes #974, #965, #955) #976) means every turn whose hook happens to collide silently drops the conversation save. A blocking queue with per-process timeout keeps data integrity while still failing fast on pathological contention.
• Informative logging so operators can see which op is currently holding the lock and how long waiters should expect.

Proposed implementation

Filing a PR shortly. Delta from #976:

• Queue pattern (blocking fcntl.flock with smart wait) instead of non-blocking skip — avoids losing hook-driven saves.
• Status file ~/.mempalace/palace/.mine.status.json with {pid, pid_start_jiffies, op, started_at, est_seconds} so waiters can (a) budget their wait, (b) break stale locks if the owner's PID+start-time are gone.
• Sleep cap (2s per iteration) so waiters don't over-sleep when the holder finishes earlier than est_seconds.
• MEMPAL_LOCK_TIMEOUT env var (default 120s).
• Tests covering acquire/release, queue semantics, fast-exit on est > timeout, 3-way serialization, sleep cap, stale-lock break after SIGKILL.

Related issues / PRs

• Critical Data write loop, HNSW compaction bug #965 — Critical Data write loop, HNSW compaction bug (disk bloat report)
• SIGSEGV in HNSW parallel inserts — missing hnsw:num_threads on collection creation #974 — HNSW SIGSEGV parallel inserts
• fix: HNSW graph corruption, PreCompact deadlock, mine fan-out (closes #974, #965, #955) #976 — felipetruman's fix (non-blocking skip + precompact attempt cap) — currently CONFLICTING with main
• fix: set hnsw:num_threads to 1 for thread-safe HNSW inserts #991 — narrower chroma.py-only patch — superseded by fix: HNSW graph corruption, PreCompact deadlock, mine fan-out (closes #974, #965, #955) #976

Happy to adjust scope / naming if a maintainer prefers — the core ask is just (a) runtime retrofit and (b) a palace-level write lock with a policy that preserves hook data.