Summary

• Problem: Issue [Bug]: Session file lock not released properly by watchdog #87483 reports that a session write-lock file held by a live gateway process persists for hours past maxHoldMs/staleMs; the watchdog never reclaims it, so later requests fail with session file locked (timeout 60000ms): pid=…. The reporter's holder process was alive and busy (83% CPU) while holding the lock for 8+ hours despite the lock recording maxHoldMs: 1020000 (17 min).
• Root Cause: There are two reclaim mechanisms and both fail for a live-but-stuck holder:
  1. The per-process watchdog (ensureWatchdogStarted) is a setInterval running inside the holding process. If that process is stuck in a synchronous CPU-bound loop, its event loop is blocked and the watchdog timer never fires, so the holder never self-releases.
  2. A contending process inspects the lock and, at acquireSessionWriteLock time (with respectMaxHold on), correctly flags it hold-exceeded (age past the holder's own recorded maxHoldMs). But shouldRemoveContendedLockFile (the shouldRemoveStaleLock callback the lock library consults under staleRecovery: "remove-if-unchanged") treats both too-old and hold-exceeded as report-only: REPORT_ONLY_STALE_LOCK_REASONS = {"too-old", "hold-exceeded"}. When a live holder's only stale reasons are these, the function returns false (don't remove), the library throws file_lock_stale, and the acquire eventually fails — the lock is never reclaimed even though it is past the holder's own declared max hold. So a live-but-stuck holder pins the lock forever: it cannot self-release (event loop blocked) and contenders are forbidden from reclaiming it.
• Fix: Remove "hold-exceeded" from REPORT_ONLY_STALE_LOCK_REASONS. A holder past its own recorded maxHoldMs is overdue by its own contract and is reclaimable by a contender. "too-old" (past the global staleMs heuristic) stays report-only — a live holder may still be working within its own declared maxHoldMs, so we do not steal its lock on the global heuristic alone. The acquire path's remove-if-unchanged recovery still skips a lock whose file has changed (e.g. a concurrent release), so only a genuinely stalled holder loses the lock.
• What changed:
  • src/agents/session-write-lock.ts — drop "hold-exceeded" from REPORT_ONLY_STALE_LOCK_REASONS (one element + an explanatory comment).
  • src/agents/session-write-lock.test.ts — add an integration regression: a live OpenClaw-owned lock past its own maxHoldMs (but within staleMs) is now reclaimed by acquireSessionWriteLock.
  • src/config/schema.help.ts and docs/reference/session-management-compaction.md — document that session.writeLock.maxHoldMs is the hard hold limit enforced both by the holder's watchdog and by a contending reclaim (help string + reference prose only).
• What did NOT change (scope boundary):
  • "too-old" stays report-only, so a live holder past the global staleMs but within its own maxHoldMs is still respected (covered by the existing "reports live OpenClaw-owned stale locks without removing them" test).
  • The cleanStaleLockFiles doctor sweep does not pass respectMaxHold and therefore never produces hold-exceeded; its behavior is unchanged (it still force-removes only dead/recycled/orphan locks, never a live holder's).
  • No new config key, schema field, default, env var, or doctor migration; the only config/docs touch is the existing session.writeLock.maxHoldMs help string and its reference doc, updated to match the behavior. No plugin SDK surface, no dependency changes. The watchdog interval, timeout, staleMs, and maxHoldMs resolution are untouched.

Reproduction

1. A long-running gateway holds a session write-lock and gets stuck in a synchronous CPU loop (event loop blocked).
2. The holder's in-process watchdog can't fire, so the lock is never self-released past maxHoldMs.
3. Another request tries to write the same session; it sees the lock past the holder's maxHoldMs but cannot reclaim it.
4. Before this PR: the contender fails with a stale/session file locked error; the lock persists until the gateway restarts or the file is deleted manually.
5. After this PR: the contender reclaims the lock (the stalled holder's overdue lock is removed-if-unchanged) and proceeds.

Deterministic source reproduction (added as a regression test): a live OpenClaw-owned lock file with maxHoldMs: 1000 and createdAt 30s ago, acquired with a large staleMs. Before the fix the acquire throws SessionWriteLockStaleError: session file lock stale (hold-exceeded): … alive=true; after the fix it reclaims the lock.

Real behavior proof

Behavior addressed (#87483 — a stalled live holder's lock is reclaimable): a session write-lock held past the holder's own maxHoldMs by a live OpenClaw process is reclaimed by a contender instead of pinning the session indefinitely.

Real environment tested (Linux, Node — Vitest against the production acquireSessionWriteLock with a real spawned OpenClaw-recognized owner process; no live gateway): verified via session-write-lock.test.ts, plus core type-check, lint, and format on the changed files. A live overnight gateway lockup was not reproduced.

Exact steps or command run after this patch (repo-root): node scripts/run-vitest.mjs src/agents/session-write-lock.test.ts; node scripts/run-tsgo.mjs -p tsconfig.core.json; oxfmt --check and oxlint on the changed files.

Evidence after fix (Vitest output for the touched test file):

 session-write-lock.test.ts   Tests  54 passed (54)

The new case fails on current main exactly as the issue describes — SessionWriteLockStaleError: session file lock stale (hold-exceeded): pid=… alive=true ageMs=30005 — and passes after dropping "hold-exceeded" from the report-only set, while all 53 existing cases (including the conservative "report live too-old locks without removing them") stay green.

Observed result after fix (overdue live holder reclaimed; conservative live holder respected): a live holder past its own maxHoldMs is reclaimed; a live holder merely past the global staleMs (within its maxHoldMs) is still only reported, not removed.

What was not tested (live overnight gateway lockup): the multi-hour real-gateway lock retention was not reproduced live; the proof is the deterministic module-level reproduction.

Risk / Mitigation

• Risk: reclaiming a live holder's lock could race with a holder still writing the transcript. Mitigation: no new race is introduced. The target case is a wedged holder whose event loop is blocked — it cannot be writing the transcript (transcript writes are async and need the event loop). A holder that is still making progress is already released at maxHoldMs by its own in-process watchdog (runLockWatchdogCheck force-releases without aborting in-flight work), so maxHoldMs was already the hard hold limit before this change; the cross-process reclaim only restores that limit for the wedged case where the watchdog cannot fire. remove-if-unchanged additionally skips a lock whose file changed.
• Risk: behavior change in session-state concurrency. Mitigation: the change narrows report-only to the global staleMs heuristic; it does not touch thresholds, the watchdog, or the doctor sweep, and the conservative live-too-old behavior is preserved (covered by an existing test).

Change Type (select all)

• Bug fix

Scope (select all touched areas)

• Agents / session write-lock

Linked Issue/PR

Fixes #87483