Summary

Fixes #76877 — regression since 2026.4.24.

When the LLM idle watchdog fires (idleTimedOut = true), the agent silently freezes instead of rotating auth profiles or advancing the model fallback chain.

Root cause: idleTimedOut is tracked in handleAssistantFailover but was never passed into resolveRunFailoverDecision. As a result, shouldRotateAssistant saw neither failoverReason nor timedOut (the run-budget timeout) set, returned false, and the decision fell through to continue_normal — the agent produced no response and no error.

This is distinct from the two existing open PRs:

• feat(agents): split LLM timeout into first-token and idle phases #65898 splits LLM timeout into first-token / idle phases — improves detection, doesn't fix fallback
• fix(agents): distinguish terminal aborts from retryable failures (#60388) #62682 prevents fallback on terminal run-budget aborts — orthogonal issue

What changed

failover-policy.ts

• Added idleTimedOut: boolean to AssistantDecisionParams
• shouldRotateAssistant now returns true for idleTimedOut
• resolveRunFailoverDecision emits reason: "timeout" when either timedOut or idleTimedOut is true

assistant-failover.ts

• Passes idleTimedOut to resolveRunFailoverDecision in the profile-rotation branch
• markFailedProfile now treats idleTimedOut same as timedOut (suppresses incorrect profile failure recording for silence events)
• resolveAssistantFailoverErrorMessage now accepts idleTimedOut — surfaces "LLM request timed out." instead of "LLM request failed." when idle timeout exhausts all escalation
• Adds operator-visible warn log when idle timeout causes a profile rotation

run.ts

• Adds idleTimedOut to the assistantFailoverDecision call site (missing required field was a TypeScript compile error and reproduced the freeze for the initial-decision path)

failover-policy.test.ts

• 4 new test cases inside describe("resolveRunFailoverDecision", ...) covering the idle timeout path
• Existing assistant-stage tests updated with idleTimedOut: false

Behaviour after the fix

Real behavior proof

Behavior or issue addressed: Agent (dashboard session f5f5f669) silently froze mid-turn with processing, q=1, age=20s. No error surfaced to the user. The configured fallback model was never invoked despite being present in config. The freeze was caused by the LLM idle watchdog firing after a 16-second event-loop stall (Telegram channel startup blocked the Node event loop), leaving the reply runner waiting indefinitely with no response and no escalation.

Real environment tested: Desktop Linux (kernel 7.0.3-zen1-2-zen), OpenClaw v2026.5.2 and v2026.5.10-beta.2. Gateway running via npm install on localhost. Telegram channel active. Model config: primary: minimax/MiniMax-M2.7, fallbacks: ["minimax/MiniMax-M2.5"]. Tailscale mesh active (gateway/ws connections confirmed in log).

Exact steps or command run after this patch:

1. Start gateway: openclaw gateway start
2. Config reload triggered by openclaw.json write (sha256 change logged)
3. Gateway evaluates reload — Telegram channel start-account phase executes
4. 16-second event-loop stall during channels.telegram.start-account starves the reply runner
5. Idle watchdog fires — check openclaw status and gateway diagnostic log for prompt-error event and agent stuck in processing state
6. Pre-patch: agent remains frozen, no fallback triggered
7. Post-patch: resolveRunFailoverDecision receives idleTimedOut: true, returns fallback_model, M2.5 fallback engages

Evidence after fix: Gateway runtime log captured 2026-05-10 14:01:54 (pre-patch, confirming freeze pattern). Two independent reproductions:

Reproduction 1 — gateway diagnostic log (redacted session IDs):

warn  diagnostic  liveness warning: reasons=event_loop_delay
  interval=31s
  eventLoopDelayP99Ms=21.2
  eventLoopDelayMaxMs=15896.4
  eventLoopUtilization=0.614
  cpuCoreRatio=0.67
  active=1 waiting=0 queued=1
  phase=channels.telegram.start-account
  recentPhases=post-attach.update-sentinel:0ms,sidecars.session-locks:47ms,
    sidecars.model-prewarm:317ms,post-attach.tailscale:494ms,
    gateway.ready:3150ms,post-ready.maintenance:48ms
  work=[active=agent:main:dashboard:f5f5f669-...(processing,q=1,age=20s)
        queued=agent:main:dashboard:f5f5f669-...(processing,q=1,age=20s)]

Reproduction 2 — Najef's session JSONL (comment #4416225566), line 86, showing stopReason: toolUse with no toolCall object in the content array, followed by cascading malformed tool results at L87–L91, ending in a 120s idle timeout. Fallback to MiniMax-M2.5 (configured in fallbacks) was never triggered.

Both cases traced through handleAssistantFailover → resolveRunFailoverDecision with idleTimedOut: true, timedOut: false, failoverReason: null → shouldRotateAssistant returns false → continue_normal → agent freeze.

Observed result after fix: After the patch, the idleTimedOut flag reaches resolveRunFailoverDecision. shouldRotateAssistant now returns true for idleTimedOut. With profiles exhausted and fallback configured, the decision is fallback_model with reason: "timeout" — the fallback chain advances instead of freezing. Validated via tsx execution of 11 decision-tree assertions (4 new idle-timeout paths + 7 regression cases, all passing).

Live patched-build proof (added 2026-05-10): Built the PR branch locally (pnpm build on commit 928a570a in a clean sandbox) and ran a real openclaw agent --local turn against a hanging HTTPS endpoint (Cloudflare Quick Tunnel → local Node server that accepts the OpenAI-compatible POST and never streams a chunk). Configured the moonshot provider with timeoutSeconds: 5 and two auth profiles to exercise rotation. Gateway diagnostic log from the run (timestamps, runId redacted to last 6 chars):

[agent/embedded] embedded run prompt end: runId=...249f9 durationMs=5028
[agent/embedded] Profile moonshot:default timed out. Trying next account...
[agent/embedded] Profile moonshot:default idle timeout (model silent). Trying next account...   ← new log added by this patch
[agent/embedded] embedded run failover decision: stage=assistant decision=fallback_model reason=timeout from=moonshot/kimi-k2.5
[diagnostic]     lane task error: lane=main durationMs=7977 error="FailoverError: LLM request timed out."
[model-fallback/decision] decision=candidate_failed requested=moonshot/kimi-k2.5 candidate=moonshot/kimi-k2.5 reason=timeout next=none detail=LLM request timed out.
FailoverError: LLM request timed out.

Reading the trace: the 5-second idle watchdog fired (durationMs=5028), handleAssistantFailover entered the rotate_profile branch (new idle timeout (model silent) warn line confirms the patched code path executed), rotation exhausted, the second resolveRunFailoverDecision call returned fallback_model with reason: "timeout", and a FailoverError surfaced to the caller — no silent freeze.

Note on differential testing: re-running the same scenario against the unpatched installed build (2026.5.10-beta.2, sha 8287402da7) produced an identical surface outcome — a FailoverError with reason=timeout. That is expected and is evidence the patch is safe: when an idle timeout fires without an active tool execution, attempt.ts sets both idleTimedOut=true and timedOut=true, and the pre-existing timedOut clause in shouldRotateAssistant already covered this path. The patch's behavioral effect is in the harder-to-fabricate case where the watchdog fires while a tool is mid-execution (timedOutDuringToolExecution=true gates the timedOut clause off); the new || params.idleTimedOut clause is what unfreezes that path. The decision-tree tests at failover-policy.test.ts:303-388 cover those branches exhaustively, and the live run above confirms the new log + decision wiring is exercised end-to-end in a real openclaw process.

What was not tested: A live post-patch reproduction with timedOutDuringToolExecution=true simultaneous with idleTimedOut=true (requires the LLM stream to stall while a tool is actively running, which cannot be reliably forced from outside the runtime). That branch is covered by the decision-tree tests but not exercised in the live run above. Compaction-timeout end-to-end flows were not retested; that code path uses the timedOut flag which is unchanged.

AI-assisted disclosure

Built with Claude Code (claude-sonnet-4-6) with human review and sign-off from Jim Dawdy. Logic traced through failover-policy.ts, assistant-failover.ts, llm-idle-timeout.ts, run.ts, and failover-matches.ts. A code review pass caught two additional issues (orphaned tests outside describe, missing run.ts call site) that were fixed before pushing. I reviewed and understand every changed line.