Summary

• 429 on a credential with multi-key pool now rotates immediately instead of first burning a retry slot retrying the same key.
• Single-credential pools keep the original retry-same-then-rotate behavior so a transient 429 on the only available key isn't doubly punished.
• Pool walking is transparent to api_max_retries — only once every credential is exhausted does retry_count start incrementing.

The bug

Fixes #16830. With a 10-key credential pool and the default api_max_retries: 3, only 2-3 keys ever get tried before the agent gives up. The remaining 7+ keys go unused even though they would have served the request.

Tracing _recover_with_credential_pool (in run_agent.py):

Iter 1: 429 on key #1 → returns (False, True)              → retry_count=1
Iter 2: 429 on key #1 → rotates to key #2, returns (True,…) → retry_count=1
Iter 3: 429 on key #2 → returns (False, True)              → retry_count=2
Iter 4: 429 on key #2 → rotates to key #3, returns (True,…) → retry_count=2
Iter 5: 429 on key #3 → returns (False, True)              → retry_count=3 → exit

Each credential consumed at least one retry slot before being exchanged, because the first 429 returned (False, True) and the outer loop did retry_count += 1 (line 11315) before the second attempt that actually rotates.

The fix

agent/credential_pool.py — new has_unexhausted_alternates(): a read-only query (no _persist, no refresh) that reports whether the pool has at least one entry besides the current one that is either unexhausted or whose cooldown has elapsed.

run_agent.py — rate-limit branch of _recover_with_credential_pool:

has_alternates = False
try:
    has_alternates = bool(pool.has_unexhausted_alternates())
except AttributeError:
    has_alternates = False  # older / hand-rolled mock pools
if not has_retried_429 and not has_alternates:
    return False, True  # single credential — keep original behavior
next_entry = pool.mark_exhausted_and_rotate(...)
...

Multi-credential pool: first 429 → rotate immediately (no retry_count cost, fresh key likely has fresh quota).
Single-credential pool: first 429 → retry once before counting, preserving the safety net for transient blips on a lone key.

The AttributeError guard keeps existing tests with hand-rolled fake pools working without modification.

Test plan

• tests/agent/test_credential_pool_routing.py — TestPoolRotationCycle updated:
  • test_first_429_with_alternates_rotates_immediately (NEW): multi-credential → rotation on first 429
  • test_first_429_no_alternates_retries_same_credential (NEW): single-credential → original behavior
  • test_full_pool_walk_does_not_burn_retry_slots (NEW): 10-credential pool walks ~all keys before recovery returns False
• tests/run_agent/test_run_agent.py::TestCredentialPoolRecovery::test_recover_with_pool_rotates_immediately_when_alternates_available (NEW): direct repro of the issue scenario.
• tests/agent/test_credential_pool.py — four new direct unit tests for has_unexhausted_alternates: two ok entries, single entry, other-in-cooldown, other-cooldown-elapsed.
• Regression guard: reverted run_agent.py + agent/credential_pool.py, the 3 new behavior tests failed with recovered=False where they now expect True. Restored fix → all 60 tests in tests/agent/test_credential_pool* and TestCredentialPoolRecovery pass.
• CI's pre-existing baseline failures in test_anthropic_adapter.py, test_bedrock_adapter.py, test_copilot_acp_client.py reproduce on clean origin/main (8269f90) and are unrelated to this change.

Related

• Fixes Credential pool rotation should not count toward api_max_retries #16830