Bug type

Behavior bug (recurring event-loop block on main thread; no crash)

Beta release blocker

No

Summary

configureSqliteWalMaintenance in src/infra/sqlite-wal.ts schedules PRAGMA wal_checkpoint(TRUNCATE) — a synchronous, blocking disk I/O operation — on the Node.js main event loop every 30 minutes via setInterval. Because node:sqlite's DatabaseSync.exec() runs synchronously on the calling thread, this operation blocks the event loop for the full duration of the WAL checkpoint. The stall fires regardless of user traffic (active=0, waiting=0, queued=0) and affects every deployment using any SQLite-backed store: task registry, plugin state, memory host SDK, and proxy capture — not only deployments with the Hindsight memory plugin.

Steps to reproduce

1. Run the OpenClaw gateway on any deployment where a SQLite-backed store is active (task registry, plugin state, or memory store — all gateways qualify).
2. Leave the gateway idle (zero active sessions) for 30 minutes.
3. Observe the diagnostic log: eventLoopDelayMaxMs spikes to 8,000–12,000 ms at a 30-minute cadence.

[diagnostic] liveness warning: reasons=event_loop_delay eventLoopDelayMaxMs=10678ms eventLoopUtilization=0.484 active=0 waiting=0 queued=0

The stall fires even when the gateway is fully idle because setInterval fires the checkpoint callback on the main event loop thread regardless of session activity.

Expected behavior

WAL maintenance should not block the event loop. A periodic WAL checkpoint should incur ≤ a few milliseconds of overhead on the main thread regardless of WAL size.

Actual behavior

PRAGMA wal_checkpoint(TRUNCATE) on a node:sqlite DatabaseSync connection blocks the main thread for the entire duration of the WAL I/O — up to 8–12 seconds on slow disks or after a high-write period when the WAL has grown large. All WebSocket events, HTTP responses, and gateway timers are delayed for the duration.

Code location

• src/infra/sqlite-wal.ts:4: DEFAULT_SQLITE_WAL_TRUNCATE_INTERVAL_MS = 30 * 60 * 1000 — 30-minute periodic interval
• src/infra/sqlite-wal.ts:50: db.exec(\PRAGMA wal_checkpoint(${checkpointMode});`)— synchronous call onDatabaseSync`; blocks the calling thread
• src/infra/sqlite-wal.ts:60: timer = setInterval(checkpoint, checkpointIntervalMs) — fires the synchronous checkpoint on the main event loop thread
• src/infra/sqlite-wal.ts:61: timer.unref?.() — prevents the timer from keeping the process alive at shutdown, but does not make the callback non-blocking when it fires

Affected production call sites (7 total):

All gateways with a task registry (the default) are affected, not only Hindsight/memory-host deployments.

Hypothesis (analysis, not observed in production)

The 30-minute event-loop stall pattern reported in #80820 (active=0, waiting=0, queued=0 at 30-minute cadence) matches this timer precisely. TRUNCATE mode blocks until all WAL frames are checkpointed into the main database file and the WAL is truncated to zero bytes — a full read-then-write I/O pass over the WAL. On deployments where the task registry or plugin-state store accumulates writes between checkpoints (e.g., during heavy cron or heartbeat periods), the WAL can grow large enough to cause multi-second stalls.

The timer.unref?.() call is a common pattern to prevent process hang-at-exit, but it is unrelated to event-loop blocking: when an unref'd timer fires, its callback still runs synchronously on the main thread.

Proposed fix options

Option A — Change default checkpoint mode from TRUNCATE to PASSIVE (minimal change, non-blocking):

PASSIVE mode writes as many WAL frames as possible without waiting for active readers; it never blocks the event loop. The trade-off is that PASSIVE does not guarantee WAL truncation, but the existing autocheckpoint threshold (1,000 pages, PRAGMA wal_autocheckpoint = 1000) already limits WAL growth. Change in src/infra/sqlite-wal.ts:43:

- const checkpointMode = options.checkpointMode ?? "TRUNCATE";
+ const checkpointMode = options.checkpointMode ?? "PASSIVE";

Option B — Hybrid: PASSIVE timer + TRUNCATE on close (recommended):

Use PASSIVE for the 30-minute periodic timer (non-blocking, controls WAL size) and keep TRUNCATE only in the close() path where blocking is acceptable (gateway shutdown). This avoids WAL accumulation while eliminating the main-thread stall.

  const checkpointMode = options.checkpointMode ?? "TRUNCATE";
+ const timerCheckpointMode: SqliteWalCheckpointMode = checkpointMode === "TRUNCATE" ? "PASSIVE" : checkpointMode;

  // in the setInterval callback:
- timer = setInterval(checkpoint, checkpointIntervalMs);
+ const timerCheckpoint = () => {
+   try { db.exec(`PRAGMA wal_checkpoint(${timerCheckpointMode});`); } catch (e) { options.onCheckpointError?.(e); }
+ };
+ timer = setInterval(timerCheckpoint, checkpointIntervalMs);

Option C — Move periodic checkpoint to a node:worker_threads worker (complete isolation):

Open a separate DatabaseSync connection in a worker thread dedicated to WAL maintenance. The worker can block freely without affecting the main event loop. Requires MessageChannel for coordination and a second SQLite connection. Most correct, most complex.

OpenClaw version

Confirmed on current main branch (commit 81b239dc984f, inspected src/infra/sqlite-wal.ts directly).

Operating system

N/A — analysis is code-level; reproducible on any OS where SQLite I/O takes >1ms.

Install method

N/A

Model

N/A

Provider / routing chain

N/A

Additional information

Related symptom report: #80820 ("Gateway event loop stalls 8–12s every ~30 minutes at idle") — that report attributed the stall to the Hindsight memory plugin, but the root cause is in the core configureSqliteWalMaintenance call used by all SQLite-backed stores, including the task registry that is active in every default gateway deployment.

Issue prepared with AI assistance. Analysis is sourced from static inspection of src/infra/sqlite-wal.ts and a search of its call sites on the current main branch. The 30-minute stall has not been independently reproduced in a live environment by the issue author; the root cause is inferred from the code and corroborated by the symptom pattern in #80820.