fix: treat HTTP 502/503/504 as failover-eligible (timeout reason) by taw0002 · Pull Request #21017 · openclaw/openclaw

taw0002 · 2026-02-19T15:11:51Z

Problem

When the primary model's API returns 502, 503, or 504, resolveFailoverReasonFromError() in failover-error.ts doesn't match any status-code branch (only 402/429/401/403/408/400 are handled). The error falls through to message-based classification via classifyFailoverReason(), which relies on extractLeadingHttpStatus() — this only works if the error message starts with the numeric status code (e.g. "503 Service Unavailable ...").

Many API SDKs (Google, Anthropic, OpenAI) set err.status = 503 as a property without prefixing the message string with 503, so the message-based classifier never matches and model failover never triggers. The run retries the same unavailable model indefinitely.

Fix

Add 502 || 503 || 504 to the status-code branch in resolveFailoverReasonFromError(), returning "timeout" — consistent with the existing behavior of isTransientHttpError() in the message-based classifier (which already includes TRANSIENT_HTTP_ERROR_CODES = new Set([500, 502, 503, 521, 522, 523, 524, 529])). 504 is also added since it represents a gateway timeout.

Test assertions added for all three status codes.

Why "timeout" and not "rate_limit"?

The message-based classifier (classifyFailoverReason) already maps transient HTTP errors to "timeout" via isTransientHttpError() → return "timeout". Using the same reason ensures consistent behavior regardless of whether the error is classified by status code or message text.

Fixes #20999

Greptile Summary

Fixed model failover not triggering for HTTP 502/503/504 errors by adding explicit status-code branches in resolveFailoverReasonFromError(). These transient server errors now return "timeout" as the failover reason, consistent with the existing message-based classification for other transient errors.

Added status checks for 502, 503, and 504 in src/agents/failover-error.ts:164-166
Added test coverage for all three status codes
Fix prevents runs from retrying the same unavailable model indefinitely when API SDKs set err.status without prefixing the message string

Minor inconsistency: 504 is now treated as transient in the status-code branch but TRANSIENT_HTTP_ERROR_CODES (used by message-based classification) excludes it. This means classification could differ based on error format, though the practical impact is limited since most SDKs set err.status.

Confidence Score: 4/5

This PR is safe to merge with low risk
The fix is narrowly scoped, well-tested, and addresses a clear bug where failover wasn't triggering for common transient HTTP errors. The logic is straightforward and consistent with existing patterns. Score is 4 (not 5) due to a minor inconsistency between status-code and message-based classification paths for 504 errors, though this is unlikely to cause issues in practice.
No files require special attention

_{Last reviewed commit: c932dac}

greptile-apps

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

src/agents/failover-error.ts

nikolasdehor

This is a clean, well-scoped fix. I commented on #20999 about exactly this problem — my fallback chain is Codex → Opus, and when the primary returns 502/503/504 during a provider outage, the gateway should fail over to the next model instead of retrying the same unavailable one indefinitely.

The diff is minimal and correct:

Status-code mapping is right. Adding 502 || 503 || 504 to resolveFailoverReasonFromError() closes the gap where err.status is set by the SDK but the error message doesn't start with the numeric code, so the message-based classifier never matches.
"timeout" is the correct reason. My original comment on #20999 suggested "rate_limit", but after looking at the code more carefully, "timeout" is the right choice — the message-based classifier in classifyFailoverReason already maps transient HTTP errors to "timeout" via isTransientHttpError(), so this keeps both classification paths consistent.
Tests cover all three codes. Simple and sufficient.

One minor note: Greptile flagged that 504 is now handled in the status-code path but is absent from TRANSIENT_HTTP_ERROR_CODES (used by message-based classification). Practically this doesn't matter — the status-code path takes priority when err.status is set, and if it isn't set, a 504 message with the right prefix would still need to be caught. But it might be worth a follow-up to add 504 to TRANSIENT_HTTP_ERROR_CODES for full consistency. Not a blocker.

Also worth noting: this PR covers HTTP status codes only, not connection-level errors (ECONNREFUSED, ETIMEDOUT, etc.). Those are a separate concern and shouldn't block this fix — getting 5xx failover working is the priority.

Tested mentally against my own setup: Codex primary returns 503 during outage → resolveFailoverReasonFromError now returns "timeout" → failover kicks in → Opus handles the request. Exactly what should happen.

vincentkoc · 2026-02-23T05:32:30Z

This is now the canonical PR for #20999 after closing #21049.

One blocker before merge: check is failing on the merge ref due a formatting drift (src/agents/tools/cron-tool.ts) from the current base branch. Please rebase onto latest main and rerun CI so check is green on this PR head.

When a model API returns 502 Bad Gateway, 503 Service Unavailable, or 504 Gateway Timeout, the error object carries the status code directly. resolveFailoverReasonFromError() only checked 402/429/401/403/408/400, so 5xx server errors fell through to message-based classification which requires the status code to appear at the start of the error message. Many API SDKs (Google, Anthropic) set err.status = 503 without prefixing the message with '503', so the message classifier never matched and failover never triggered — the run retried the same broken model. Add 502/503/504 to the status-code branch, returning 'timeout' (matching the existing behavior of isTransientHttpError in the message classifier). Fixes openclaw#20999

…enclaw#21017) * fix: treat HTTP 502/503/504 as failover-eligible (timeout reason) When a model API returns 502 Bad Gateway, 503 Service Unavailable, or 504 Gateway Timeout, the error object carries the status code directly. resolveFailoverReasonFromError() only checked 402/429/401/403/408/400, so 5xx server errors fell through to message-based classification which requires the status code to appear at the start of the error message. Many API SDKs (Google, Anthropic) set err.status = 503 without prefixing the message with '503', so the message classifier never matched and failover never triggered — the run retried the same broken model. Add 502/503/504 to the status-code branch, returning 'timeout' (matching the existing behavior of isTransientHttpError in the message classifier). Fixes openclaw#20999 * Changelog: add failover 502/503/504 note with credits * Failover: classify HTTP 504 as transient in message parser * Changelog: credit taw0002 and vincentkoc for failover fix --------- Co-authored-by: Vincent Koc <vincentkoc@ieee.org>

…enclaw#21017) * fix: treat HTTP 502/503/504 as failover-eligible (timeout reason) When a model API returns 502 Bad Gateway, 503 Service Unavailable, or 504 Gateway Timeout, the error object carries the status code directly. resolveFailoverReasonFromError() only checked 402/429/401/403/408/400, so 5xx server errors fell through to message-based classification which requires the status code to appear at the start of the error message. Many API SDKs (Google, Anthropic) set err.status = 503 without prefixing the message with '503', so the message classifier never matched and failover never triggered — the run retried the same broken model. Add 502/503/504 to the status-code branch, returning 'timeout' (matching the existing behavior of isTransientHttpError in the message classifier). Fixes openclaw#20999 * Changelog: add failover 502/503/504 note with credits * Failover: classify HTTP 504 as transient in message parser * Changelog: credit taw0002 and vincentkoc for failover fix --------- Co-authored-by: Vincent Koc <vincentkoc@ieee.org> (cherry picked from commit 3c57bf4) # Conflicts: # src/agents/pi-embedded-helpers.isbillingerrormessage.test.ts # src/agents/pi-embedded-helpers/errors.ts

…enclaw#21017) * fix: treat HTTP 502/503/504 as failover-eligible (timeout reason) When a model API returns 502 Bad Gateway, 503 Service Unavailable, or 504 Gateway Timeout, the error object carries the status code directly. resolveFailoverReasonFromError() only checked 402/429/401/403/408/400, so 5xx server errors fell through to message-based classification which requires the status code to appear at the start of the error message. Many API SDKs (Google, Anthropic) set err.status = 503 without prefixing the message with '503', so the message classifier never matched and failover never triggered — the run retried the same broken model. Add 502/503/504 to the status-code branch, returning 'timeout' (matching the existing behavior of isTransientHttpError in the message classifier). Fixes openclaw#20999 * Changelog: add failover 502/503/504 note with credits * Failover: classify HTTP 504 as transient in message parser * Changelog: credit taw0002 and vincentkoc for failover fix --------- Co-authored-by: Vincent Koc <vincentkoc@ieee.org>

…enclaw#21017) * fix: treat HTTP 502/503/504 as failover-eligible (timeout reason) When a model API returns 502 Bad Gateway, 503 Service Unavailable, or 504 Gateway Timeout, the error object carries the status code directly. resolveFailoverReasonFromError() only checked 402/429/401/403/408/400, so 5xx server errors fell through to message-based classification which requires the status code to appear at the start of the error message. Many API SDKs (Google, Anthropic) set err.status = 503 without prefixing the message with '503', so the message classifier never matched and failover never triggered — the run retried the same broken model. Add 502/503/504 to the status-code branch, returning 'timeout' (matching the existing behavior of isTransientHttpError in the message classifier). Fixes openclaw#20999 * Changelog: add failover 502/503/504 note with credits * Failover: classify HTTP 504 as transient in message parser * Changelog: credit taw0002 and vincentkoc for failover fix --------- Co-authored-by: Vincent Koc <vincentkoc@ieee.org> (cherry picked from commit 3c57bf4) # Conflicts: # src/agents/pi-embedded-helpers.isbillingerrormessage.test.ts # src/agents/pi-embedded-helpers/errors.ts

…enclaw#21017) * fix: treat HTTP 502/503/504 as failover-eligible (timeout reason) When a model API returns 502 Bad Gateway, 503 Service Unavailable, or 504 Gateway Timeout, the error object carries the status code directly. resolveFailoverReasonFromError() only checked 402/429/401/403/408/400, so 5xx server errors fell through to message-based classification which requires the status code to appear at the start of the error message. Many API SDKs (Google, Anthropic) set err.status = 503 without prefixing the message with '503', so the message classifier never matched and failover never triggered — the run retried the same broken model. Add 502/503/504 to the status-code branch, returning 'timeout' (matching the existing behavior of isTransientHttpError in the message classifier). Fixes openclaw#20999 * Changelog: add failover 502/503/504 note with credits * Failover: classify HTTP 504 as transient in message parser * Changelog: credit taw0002 and vincentkoc for failover fix --------- Co-authored-by: Vincent Koc <vincentkoc@ieee.org>

openclaw-barnacle bot added agents Agent runtime and tooling size: XS labels Feb 19, 2026

greptile-apps bot reviewed Feb 19, 2026

View reviewed changes

src/agents/failover-error.ts Show resolved Hide resolved

nikolasdehor approved these changes Feb 19, 2026

View reviewed changes

vincentkoc mentioned this pull request Feb 23, 2026

fix(failover): treat HTTP 5xx as rate-limit for model fallback #21049

Closed

vincentkoc added the dedupe:parent Primary canonical item in dedupe cluster label Feb 23, 2026

taw0002 and others added 2 commits February 23, 2026 00:43

Changelog: add failover 502/503/504 note with credits

90858ab

vincentkoc force-pushed the fix/failover-502-503-504 branch from c932dac to 90858ab Compare February 23, 2026 05:48

Failover: classify HTTP 504 as transient in message parser

d792ba4

vincentkoc force-pushed the fix/failover-502-503-504 branch from 1986812 to d792ba4 Compare February 23, 2026 07:57

Changelog: credit taw0002 and vincentkoc for failover fix

4360ee0

vincentkoc merged commit 3c57bf4 into openclaw:main Feb 23, 2026
23 of 25 checks passed

github-actions bot mentioned this pull request Feb 23, 2026

📡 Upstream Digest — 2026-02-23 08:39 UTC curtismercier/openclaw-mods#108

Open

vabole mentioned this pull request Feb 23, 2026

ops: decision needed for upstream sync (3 disputed files) vabole/openclaw#3

Merged

3 tasks

ZPTDclaw mentioned this pull request Feb 24, 2026

fix: classify Google 503 UNAVAILABLE as transient failover [AI-assisted] #21491

Closed

taw0002 mentioned this pull request Feb 26, 2026

fix: silent model failover with fallback notification #13658

Closed

github-actions bot mentioned this pull request Mar 1, 2026

cherry-pick: upstream bugfix commits (2026-03-01-0443) hughdidit/DAISy-Agency#140

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: treat HTTP 502/503/504 as failover-eligible (timeout reason)#21017

fix: treat HTTP 502/503/504 as failover-eligible (timeout reason)#21017
vincentkoc merged 4 commits intoopenclaw:mainfrom
taw0002:fix/failover-502-503-504

taw0002 commented Feb 19, 2026 •

edited by greptile-apps bot

Loading

Uh oh!

greptile-apps bot left a comment

Uh oh!

Uh oh!

nikolasdehor left a comment

Uh oh!

vincentkoc commented Feb 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

taw0002 commented Feb 19, 2026 • edited by greptile-apps bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Why "timeout" and not "rate_limit"?

Greptile Summary

Confidence Score: 4/5

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nikolasdehor left a comment

Choose a reason for hiding this comment

Uh oh!

vincentkoc commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

taw0002 commented Feb 19, 2026 •

edited by greptile-apps bot

Loading

vincentkoc commented Feb 23, 2026 •

edited

Loading