Summary

• Problem: When an API key is revoked, resolveFailoverReasonFromError() maps HTTP 401/403 to "auth" with a transient cooldown (max 1h). The key retries forever instead of being permanently disabled.
• Why it matters: During key rotation, a revoked key causes agents to retry indefinitely with exponential backoff capped at 1 hour, wasting tokens and blocking failover.
• What changed: Added "auth_permanent" as a new FailoverReason variant. Provider-specific permanent auth error signals (e.g. invalid_api_key, key has been revoked) are detected in the error message and routed through the disabledUntil path (5h default, 24h max — same as billing) instead of the transient cooldownUntil path.
• What did NOT change (scope boundary): Transient auth errors ("unauthorized", "forbidden", "invalid token", expired tokens) still use the existing "auth" reason with short cooldown. No changes to billing error handling. No changes to UI/display code.

Change Type (select all)

• Bug fix
• Feature
• Refactor
• Docs
• Security hardening
• Chore/infra

Scope (select all touched areas)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Closes Auth failover: distinguish revoked tokens from transient auth errors #25689

User-visible / Behavior Changes

• Revoked/deleted API keys are now disabled for 5h (exponential backoff to 24h max) instead of retrying every ≤1h forever.
• models status will show disabledUntil + disabledReason: "auth_permanent" for affected profiles.
• Profile recovers automatically after the disable window expires, or immediately on successful use (e.g. after key replacement).

Security Impact (required)

• New permissions/capabilities? No
• Secrets/tokens handling changed? No
• New/changed network calls? No
• Command/tool execution surface changed? No
• Data access scope changed? No

Repro + Verification

Environment

• OS: any
• Runtime/container: Node 22+
• Model/provider: any provider returning 401/403 with permanent auth error messages

Steps

1. Configure a profile with a revoked API key
2. Trigger an LLM request that fails with 401 + "invalid_api_key" in the error message
3. Check usageStats for the profile

Expected

• Profile gets disabledUntil set (5h+ window), disabledReason: "auth_permanent"
• Profile does NOT get cooldownUntil (transient path)

Actual

• Before: profile gets cooldownUntil (max 1h), retries forever
• After: profile gets disabledUntil (5h default, 24h max), matching billing error behavior

Evidence

• Failing test/log before + passing after
  • New tests in failover-error.test.ts (6 tests), auth-profiles.markauthprofilefailure.test.ts (1 test), pi-embedded-helpers.isbillingerrormessage.test.ts (3 test blocks)
  • 164 tests across 7 agent-area test files pass
  • pnpm check (oxfmt + tsgo + oxlint) clean

Human Verification (required)

• Verified scenarios: backward compat (bare 401/403 still returns "auth"), permanent auth detection from status+message, pattern separation ("invalid api key" → auth vs "invalid_api_key" → auth_permanent), disabledUntil path for auth_permanent, clearExpiredCooldowns handles auth_permanent generically, markAuthProfileUsed clears disabled state on success
• Edge cases checked: empty message on 401/403 (falls through to "auth"), FAILURE_REASON_SET/FAILURE_REASON_ORDER auto-updated, as const type inference, pi-embedded-runner/run.ts propagation, list.probe.ts/list.status-command.ts fallthrough behavior
• What you did not verify: live end-to-end with a real revoked key (would need a deliberately revoked provider key to trigger the 401 + "invalid_api_key" flow in a running instance)

Compatibility / Migration

• Backward compatible? Yes
• Config/env changes? No
• Migration needed? No

Failure Recovery (if this breaks)

• How to disable/revert this change quickly: revert this single commit; the "auth_permanent" reason will simply never be produced, falling back to "auth" transient cooldown
• Files/config to restore: none (no config changes)
• Known bad symptoms reviewers should watch for: profiles being permanently disabled when they shouldn't be (false positive on the authPermanent patterns), or profiles NOT being disabled when they should be (pattern too narrow)

Risks and Mitigations

• Risk: authPermanent patterns could false-positive on messages that contain "invalid_api_key" as substring in a longer non-permanent error
  • Mitigation: The patterns are specific (exact string "invalid_api_key", regexes requiring revoked|deactivated|deleted after api key). Transient auth keywords ("unauthorized", "forbidden", "invalid token") are NOT in the permanent list. The permanent check runs only when HTTP status is already 401/403 or when message-only classification is used.

Greptile Summary

This PR adds a new "auth_permanent" failover reason to distinguish permanently revoked/invalid API keys from transient auth errors. Revoked keys now trigger the longer disabledUntil path (5h default, 24h max) instead of the shorter cooldownUntil path (max 1h), preventing indefinite retry loops during key rotation.

The implementation follows existing patterns:

• Pattern detection checks authPermanent before auth in classifyFailoverReason() to handle overlapping patterns correctly
• auth_permanent reuses the billing backoff logic (disabledUntil + exponential backoff)
• FAILURE_REASON_PRIORITY places auth_permanent at highest priority (before auth)
• Test coverage includes pattern matching, backoff behavior, and backward compatibility

Confidence Score: 5/5

• Safe to merge with high confidence
• The change is well-contained with comprehensive test coverage (10 new tests), follows existing patterns for failover handling, maintains backward compatibility (bare 401/403 still returns "auth"), and includes proper ordering to handle pattern overlaps. The implementation correctly reuses billing backoff logic for permanent auth failures.
• No files require special attention

_{Last reviewed commit: c66546c}