-
-
Notifications
You must be signed in to change notification settings - Fork 80.3k
Bug: channel stop timeout leaves channel permanently dead — running: true with stale store entries #70024
Copy link
Copy link
Open
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.maturity:stableIssue affects a taxonomy feature currently scored M4/M5.Issue affects a taxonomy feature currently scored M4/M5.
Description
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.maturity:stableIssue affects a taxonomy feature currently scored M4/M5.Issue affects a taxonomy feature currently scored M4/M5.
Type
Fields
No fields configured for issues without a type.
Description
When
stopChanneltimes out waiting for a channel task to settle, it setsrunning: truein the runtime snapshot without cleaning upstore.abortsorstore.tasks. Three code paths then combine to make the channel silently dead with no automatic recovery.Root Cause — Three-Link Chain
Link 1: stop timeout lies about state and skips cleanup
src/gateway/server-channels.ts:574-584The timeout path sets
running: trueand returns early, leaving the dead promise instore.tasksand the staleAbortControllerinstore.aborts.Link 2: startChannel blocked by stale store entry
src/gateway/server-channels.ts:306Map.has()only checks key existence, not promise state. The dead promise from Link 1 blocks all future starts permanently.Link 3: health monitor fooled by
running: truesrc/gateway/channel-health-policy.ts:80:src/gateway/channel-health-monitor.ts:148:Since
runningistrue, the health monitor never flags the channel as unhealthy. Even if it did, cooldown would suppress restart attempts.Combined Effect
Channel dies → system thinks it's alive → no automatic recovery. Silent permanent death.
Steps to Reproduce
stopChannelwhere the underlying task does not settle within 5 seconds (CHANNEL_STOP_ABORT_TIMEOUT_MS)running: true,restartPending: falsestore.tasksstill contains the dead promisestartChannel→ silently skipped due tostore.tasks.has(id)Expected Behavior
running: falsestore.abortsandstore.tasksstartChannelcalls should succeedSuggested Fix
In the timeout branch (
server-channels.ts:574-584):running: falseinstead oftruestore.aborts.delete(id)andstore.tasks.delete(id)before returningif (!stoppedCleanly) { log.warn?.( `[${id}] channel stop exceeded ${CHANNEL_STOP_ABORT_TIMEOUT_MS}ms after abort; continuing shutdown`, ); + store.aborts.delete(id); + store.tasks.delete(id); setRuntime(channelId, id, { accountId: id, - running: true, + running: false, restartPending: false, + lastStopAt: Date.now(), lastError: `channel stop timed out after ${CHANNEL_STOP_ABORT_TIMEOUT_MS}ms`, }); return; }Environment
main(fd2c883)[AI-assisted]