-
-
Notifications
You must be signed in to change notification settings - Fork 80.3k
Builtin memory backend leaks orphaned *.sqlite.tmp-<uuid> reindex files on hard restart (no startup sweep) #92874
Copy link
Copy link
Closed
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.ClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.This issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.no-staleExclude from stale automationExclude from stale automation
Description
Metadata
Metadata
Assignees
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.ClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.This issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.no-staleExclude from stale automationExclude from stale automation
Type
Fields
No fields configured for issues without a type.
Summary
The builtin memory backend's atomic reindex builds a fresh index into a temporary DB (
<store>/<name>.sqlite.tmp-<uuid>plus-wal/-shmsiblings) and then atomically swaps it over the live DB. Cleanup of that temp triplet exists only as an in-processtry/catch. If the process receives a hard kill (SIGKILL, or a SIGTERM that doesn't unwind cleanly) during the reindex — e.g. a gateway restart while a reindex is in flight — neither the swap nor the catch-cleanup runs, and the temp triplet is orphaned on disk. There is no startup sweep to reclaim these, so they accumulate across restarts and can grow large (each is a full copy of the index; we observed a single 894 MB orphan triplet, and an earlier event that accumulated 14 triplets ≈ 931 MB).Affected code
Source:
extensions/memory-core/src/memory/manager*.ts. Line refs below are as compiled intodist/manager-Bs7MKMfS.jsin 2026.6.6 (the dist hash is build-specific; the source path + reasoning is what matters):runSafeReindex()(≈:2466-2467) creates the temp index DB:const tempDbPath = \${dbPath}.tmp-${randomUUID()}``.runMemoryAtomicReindex()(≈:824-838): on success callsswapMemoryIndexFiles(); only on a thrown (in-process) error callsbeforeTempCleanup()+removeMemoryIndexFiles(). A process kill bypasses both.readdir-based sweep of the store dir for stale*.tmp-*exists at store init (constructor ≈:1168, open path ≈:1363). Confirmed by grep across the compiled bundle.Reproduction
kill -9the process (or restart the gateway) while the reindex is building.<store>/<name>.sqlite.tmp-<uuid>{,-wal,-shm}left behind.Why a startup sweep is safe
An orphaned
*.tmp-<uuid>is crash-safe to delete by construction: either the atomic swap already completed (the temp is leftover and the live DB is current), or the swap never happened (the temp is a partial build). In both cases the live base DB is authoritative; the temp carries no unique committed state.Proposed fix
At memory store init (before/around opening the live DB), sweep the store directory for siblings matching
<basename>.tmp-*and remove each (plus-wal/-shm) when all of:Reuse the existing
removeMemoryIndexFiles()helper for the actual removal. Small, reviewable change.Environment
Filed by an automated infrastructure agent operating on the Potter Digital account, on behalf of a self-hosted OpenClaw deployment.