Summary

When the Codex Responses API returns HTTP 200 with response.status = "failed" or "cancelled" (e.g. quota exhaustion), the runtime correctly detects the soft failure but only attempts cross-provider fallback via _try_activate_fallback(). Same-provider credential pool rotation is never invoked, so the agent stays on one exhausted account and eventually falls through to a different provider entirely.

Root cause: The response_invalid block (added in PR #15104) detects codex soft failures but never calls _recover_with_credential_pool(). The exception handler path (429/402) does call pool recovery, but these soft failures bypass the exception handler entirely — the HTTP response is 200, not an error code.

Changes

• New method _classify_codex_soft_failure() on AIAgent — pattern-matches the response.error message against billing and rate-limit signals, returning a FailoverReason or None for non-quota errors.
• Early pool-recovery block in the response_invalid path for codex_responses mode: classify the error, attempt credential rotation, and continue on the new credential before falling through to cross-provider fallback.
• Billing signals checked first: insufficient, credits have been exhausted, billing, payment required, exceeded your current quota, plan does not include, account is deactivated, usage limit, credit balance, top up your credits, quota.
• Rate-limit signals checked second: rate limit, rate_limit, too many requests, throttl, try again, retry, slow down, quota exceeded, requests per, tokens per.
• Non-quota soft failures (content policy, safety, cancelled by user) return None and do not trigger pool rotation.

Test plan

• 25 new tests in tests/agent/test_codex_soft_failure_pool_rotation.py:
  • 8 parametrized billing signal tests
  • 6 parametrized rate-limit signal tests
  • 4 non-quota error tests (content policy, safety, cancelled, empty)
  • 1 billing-priority-over-rate-limit test
  • 6 integration tests (pool rotation on billing, rate-limit, no-pool skip, content-policy skip, non-codex skip, pool exhaustion fallback)
• All 51 existing credential pool tests still pass
• pytest tests/agent/test_credential_pool.py tests/agent/test_credential_pool_routing.py tests/agent/test_codex_soft_failure_pool_rotation.py — 76/76 passed

Risks

• Low risk: Changes are additive — no existing code paths are modified, only new recovery logic inserted before the existing fallback path.
• Pattern matching is best-effort: Error messages from the Codex Responses API are free-form strings. The pattern lists cover known error phrasing but may need extension as new error messages are observed. The fallback path (cross-provider) still fires if pool recovery fails or returns None.
• Billing vs rate-limit ambiguity: Some messages (e.g. "exceeded your quota") could be either billing or rate-limit. The classifier errs on the side of billing (immediate rotation, no backoff wait) since quota exhaustion is more severe than temporary throttling. This matches the behavior users expect when they have multiple paid plans.

Related

• Fixes [Bug] Codex Responses API soft failures (status=failed) bypass credential pool rotation #24159
• Related to auth remove re-seeds manually added OAuth credentials from singleton auth store #13967 (source field mismatch in credential pool)
• Related to PR fix(codex): consolidated OAuth error parsing + failed-status fallback routing + reauth UX #15104 (landed soft-failure detection without pool rotation)
• Related to platform retry: rotate auth profiles within a single retry sequence before failing the request #22212 (partial rotation in python platform layer)