Skip to content

backup create leaves multi-GB intermediate temp + staging dir orphaned in tmpdir after a hard kill (no startup sweep of prior runs) #95582

Description

@potterdigital

Version: openclaw 2026.6.8 (Node v24.13.0, Linux)

Summary

backup create stages its work in two temp artifacts under os.tmpdir():

  • a staging dir openclaw-backup-<rand>/ (fs.mkdtemp), holding the manifest and sanitized sqlite snapshots, and
  • the intermediate archive <outputPath>.<uuid>.tmp, which receives the full gzip tar stream before being published to the final path.

Both are removed in a single try { … } finally { fs.rm(tempArchivePath); fs.rm(tempDir) }. That cleanup is in-process only. A hard kill during the run — SIGKILL, OOM, host reboot, or a service/gateway restart that takes the process down — bypasses the finally and orphans both artifacts. The archive temp is full-size (multi-GB for a real backup), so each interrupted run permanently leaks a copy of the entire backup into tmpdir.

Nothing ever reaps them. A subsequent backup create creates a fresh mkdtemp dir and a fresh UUID-named temp; it does not scan tmpdir for stale openclaw-backup-* / *.<uuid>.tmp left by a prior crashed run. So the leak accumulates across interrupted runs until something external cleans tmpdir.

Why it matters

On a single-volume host where tmpdir shares the root filesystem, repeated interrupted backups silently refill the disk with full-size orphans. We hit this in production three times before tracing it; each incident pinned ~2.7 GB until manually cleared. The failure mode is disk-pressure, and it compounds: a disk-pressure kill during a backup leaves an orphan that brings the disk closer to the next kill.

Reproduction

  1. Start backup create on a state dir large enough that archiving takes a few seconds.
  2. kill -9 the process (or trigger an OOM/reboot) after staging begins but before publish.
  3. Inspect tmpdir (os.tmpdir(), typically /tmp):
    • openclaw-backup-<rand>/ staging dir remains (manifest + sqlite snapshots).
    • <outputPath>.<uuid>.tmp full-size archive remains.
  4. Run backup create again to completion. Confirm the prior run's orphans are still present — the new run does not sweep them.

Root cause (verified against source, v2026.6.8)

In the createBackupArchive path:

  • tempDir = fs.mkdtemp(path.join(tempRoot, "openclaw-backup-"))
  • tempArchivePath = ${outputPath}.${randomUUID()}.tmp
  • the tar write, manifest write, and publish all run inside one try, with cleanup of both temps only in the matching finally.

There is no startup pass that lists tmpdir and removes stale openclaw-backup-* dirs or *.<uuid>.tmp archives from earlier runs. The finally is the only cleanup, and it cannot fire when the process is hard-killed.

(Note: publish is hardlink-then-unlink-temp with a copyFile(COPYFILE_EXCL) fallback — not an atomic rename — but that detail isn't the bug; the bug is that the in-process finally is the only thing that ever removes the temps.)

Proposed fix

On backup create startup — before staging the new run — sweep the chosen temp root for the tool's own stale artifacts:

  • directories matching the openclaw-backup- mkdtemp prefix, and
  • files matching *.<uuid>.tmp siblings of the resolved output path,

removing those not currently held open / older than a small grace window (to avoid racing a concurrent run). Scope the match tightly to the tool's own naming so it can never touch unrelated files in a shared tmpdir. This makes the next run self-heal after any hard kill and fixes the leak at the source for every caller, rather than each deployment having to bolt on an external janitor.

There's accepted precedent for this exact pattern: #92874 (the memory backend's reindex temps) was the same crash-unsafe-temp-on-hard-kill bug class, and it was fixed (merged in #92891) by adding a startup sweep that removes the tool's own stale temp files left by a hard-killed run, guarded by a staleness window so it can't race a concurrent operation. backup create has the identical bug class but no such sweep; adopting the same approach here — ideally a shared startup-sweep-own-artifacts helper so both live in one place — would retire it at the source.

Workaround (for anyone hitting this before the fix)

A periodic/boot-time janitor that trashes, under tmpdir, the openclaw-backup-* staging dirs and the intermediate archives matching the verified ${outputPath}.${uuid}.tmp naming — i.e. the *.tar.gz.*.tmp suffix (this matches the default <timestamp>-openclaw-backup.tar.gz.<uuid>.tmp and any custom --output name, so a copy-paste reader doesn't get an empty match) — older than ~60 min and not held open by lsof. (This is what we shipped locally; it works but every deployment shouldn't need it.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions