Skip to content

Bug: channel stop timeout leaves channel permanently dead — running: true with stale store entries #70024

Description

@garnetlyx

Description

When stopChannel times out waiting for a channel task to settle, it sets running: true in the runtime snapshot without cleaning up store.aborts or store.tasks. Three code paths then combine to make the channel silently dead with no automatic recovery.

Root Cause — Three-Link Chain

Link 1: stop timeout lies about state and skips cleanup

src/gateway/server-channels.ts:574-584

if (!stoppedCleanly) {
  setRuntime(channelId, id, {
    running: true,          // ← lie: channel is actually dead
    restartPending: false,
    lastError: `channel stop timed out ...`,
  });
  return;                   // ← skips cleanup
}
// normal path (586-593):
store.aborts.delete(id);
store.tasks.delete(id);
setRuntime(channelId, id, { running: false, ... });

The timeout path sets running: true and returns early, leaving the dead promise in store.tasks and the stale AbortController in store.aborts.

Link 2: startChannel blocked by stale store entry

src/gateway/server-channels.ts:306

if (store.tasks.has(id)) {
  return;  // ← dead promise still in map → silently skipped
}

Map.has() only checks key existence, not promise state. The dead promise from Link 1 blocks all future starts permanently.

Link 3: health monitor fooled by running: true

src/gateway/channel-health-policy.ts:80:

if (!snapshot.running) {
  return { healthy: false, reason: "not-running" };
}
// running: true → skips this, assumes healthy

src/gateway/channel-health-monitor.ts:148:

if (now - record.lastRestartAt <= cooldownMs) {
  continue;  // cooldown window blocks retry
}

Since running is true, the health monitor never flags the channel as unhealthy. Even if it did, cooldown would suppress restart attempts.

Combined Effect

Channel dies → system thinks it's alive → no automatic recovery. Silent permanent death.

Steps to Reproduce

  1. Start a channel (e.g. Telegram, Discord)
  2. Trigger a stopChannel where the underlying task does not settle within 5 seconds (CHANNEL_STOP_ABORT_TIMEOUT_MS)
  3. Observe runtime snapshot: running: true, restartPending: false
  4. Observe store.tasks still contains the dead promise
  5. Attempt startChannel → silently skipped due to store.tasks.has(id)
  6. Health monitor never detects the channel as unhealthy

Expected Behavior

  • Stop timeout should set running: false
  • Stop timeout should still clean up store.aborts and store.tasks
  • Subsequent startChannel calls should succeed
  • Health monitor should detect and recover the channel

Suggested Fix

In the timeout branch (server-channels.ts:574-584):

  1. Set running: false instead of true
  2. Clean up store.aborts.delete(id) and store.tasks.delete(id) before returning
 if (!stoppedCleanly) {
   log.warn?.(
     `[${id}] channel stop exceeded ${CHANNEL_STOP_ABORT_TIMEOUT_MS}ms after abort; continuing shutdown`,
   );
+  store.aborts.delete(id);
+  store.tasks.delete(id);
   setRuntime(channelId, id, {
     accountId: id,
-    running: true,
+    running: false,
     restartPending: false,
+    lastStopAt: Date.now(),
     lastError: `channel stop timed out after ${CHANNEL_STOP_ABORT_TIMEOUT_MS}ms`,
   });
   return;
 }

Environment

  • OpenClaw: latest main (fd2c883)
  • Channels affected: all (Telegram, Discord, Slack, etc.)
  • Node: 22+

[AI-assisted]

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.maturity:stableIssue affects a taxonomy feature currently scored M4/M5.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions