Summary

Add a STATUS_DEAD terminal state to the credential pool for permanently-failed OAuth credentials, and auto-prune dead manual entries after a 24-hour quiet window. Closes #32849.

Background

When OpenAI Codex returns a 401 with token_invalidated or token_revoked, the credential is broken upstream — retrying after a TTL cooldown cannot fix it. The current code marks every 401/429/402 the same way: STATUS_EXHAUSTED with a TTL cooldown (5 min for 401, 1 hour for 429/default). After the TTL elapses the credential re-enters rotation and immediately fails again with the same 401, surfacing on long sessions as Failed to generate context summary and breaking compaction every hour all day.

Reporter observed 7 separate 401 token_invalidated failures from the same revoked Codex OAuth credential between 10:27 and 17:50 UTC on a single day. Manual hermes auth → remove was the only way to stop it.

Fix

New STATUS_DEAD state alongside STATUS_OK and STATUS_EXHAUSTED:

• What goes DEAD: 401 responses whose error.code / error.reason matches a known terminal OAuth state — token_invalidated, token_revoked, invalid_token, invalid_grant, unauthorized_client, refresh_token_reused. Sourced from OpenAI Codex, Anthropic, xAI, and OAuth 2.0 RFCs 6749/6750/7009.
• What stays EXHAUSTED: 429 rate limits, 402 billing, 5xx, generic 401 with no specific reason (might be a transient server-side glitch worth retrying after TTL).
• DEAD entries are excluded from rotation unconditionally — no TTL clears them.
• DEAD entries only clear via an explicit re-auth write-side sync — _sync_codex_pool_entries and the _sync_*_entry_from_auth_store paths clear last_status to None, so a fresh device-code login or other re-auth automatically resurrects the credential.
• Auth.json read-side sync extended to fire on DEAD too — if another process already has fresh tokens in auth.json, an in-flight DEAD entry adopts them on next pool selection without needing explicit re-auth.

Auto-prune DEAD manual entries after 24h (the addition you asked about):

• After 24 hours with no recovery, manual:* DEAD entries are removed from the pool so dead state doesn't accumulate. The user can always re-add via hermes auth add.
• Singleton-seeded DEAD entries (device_code, loopback_pkce, claude_code) are NOT pruned because _seed_from_singletons would just recreate them on next load with the same stale singleton tokens. Pruning them would be a pointless dance. The DEAD marker stays visible so the audit trail (label, last_error_reason, timestamps) is preserved.

Why mark-dead-then-prune instead of remove-immediately:

• Recoverability — within the 24h window, an explicit re-auth (write-side sync) resurrects the entry in-place with the same ID/priority.
• False-positive safety — if OpenAI hiccups during their own outage and returns a spurious token_invalidated on a valid token (we have no evidence this happens, but the codex backend has been flaky), the user still has a window to recover without re-adding.
• Audit trail — the user sees last_error_reason: token_invalidated in hermes auth list instead of an unexplained disappearance.
• refresh_token_reused from inter-process races — losing process's pool entry transitions back to OK on next selection via the auth.json sync.

Changes

• agent/credential_pool.py:
  • new STATUS_DEAD constant + _TERMINAL_AUTH_REASONS set + DEAD_MANUAL_PRUNE_TTL_SECONDS (24h)
  • new _is_terminal_auth_failure() helper
  • _mark_exhausted() routes terminal reasons to STATUS_DEAD
  • _available_entries() excludes DEAD entries and prunes DEAD manual entries past the 24h TTL
  • the four _sync_*_entry_from_auth_store calls now fire on {EXHAUSTED, DEAD} so a write-side re-auth clears DEAD too
  • mark_exhausted_and_rotate() logs DEAD distinctly at warning level
• tests/agent/test_credential_pool.py: seven new tests
  • token_invalidated marks DEAD and still rotates to a healthy entry
  • DEAD never re-enters rotation within the 24h window
  • 429 rate limits still use STATUS_EXHAUSTED (transient)
  • 401 without a specific terminal reason still uses STATUS_EXHAUSTED
  • DEAD manual entry pruned after 24h
  • DEAD manual entry kept within 24h (audit trail preserved)
  • DEAD singleton-seeded entry NOT pruned (would just re-seed)

Validation

• 77/77 pass in tests/agent/test_credential_pool.py (was 70 — 7 new)
• 338/338 pass across credential_pool + auxiliary_client + codex auth tests

Closes

• Credential pool: token_invalidated / token_revoked should be terminal failures, not 1-hour cooldowns #32849