Skip to content

[Bug]: Auxiliary auto fallback fails on OpenRouter 403 credit/key-limit errors and may not skip the concrete failed backend #13887

@AllardQuek

Description

@AllardQuek

Bug Description

When auxiliary tasks run with provider=auto, and face 403 responses due to OpenRouter key/spend limits, auxiliary requests may raise immediately instead of triggering provider fallback to the next backend.”

A related issue is that the fallback can skip the wrong backend (or skip nothing useful), causing ineffective retries, as there does not seem to be a concrete label passed by the caller

Steps to Reproduce

  1. Configure auxiliary resolution as auto (default).
  2. Set an OpenRouter key/account in a depleted/key-limited state so requests return 403 with text like Key limit exceeded, spending limit, or total limit.
  3. Ensure another provider is available (for example Codex OAuth, Nous, or custom endpoint) so fallback is possible.
  4. Trigger an auxiliary call (for example compression/session search/web extract).
  5. Observe request handling in call_llm/async_call_llm.

Expected Behavior

  1. 403 spend/key-limit responses should be treated as fallback-eligible payment exhaustion (same class as 402 for this purpose).
  2. Auxiliary fallback should skip the actual failed backend and continue to the next available provider in the chain.
  3. Auxiliary task should succeed on a downstream provider when one is available.

Actual Behavior

  1. 403 OpenRouter key-limit/spend-limit errors do not seem to be classified as payment exhaustion, and thus not triggering fallback.
  2. In fallback-triggered paths, failed-backend identity may be too coarse if concrete backend metadata is not propagated, which can lead to poor skip behavior.
  3. Auxiliary calls can fail even when another provider is configured and healthy.

Screenshot below shows 3 tries with the same model and same error instead of trying a fallback provider:
Image

Affected Component

Agent Core (conversation loop, context compression, memory)

Messaging Platform (if gateway-related)

No response

Debug Report

Debug report uploaded:
  Report     https://paste.rs/JyerH
  agent.log  https://paste.rs/z2GhQ

Operating System

MacOS 26.4

Python Version

No response

Hermes Version

No response

Additional Logs / Traceback (optional)

Root Cause Analysis (optional)

  1. Payment-error classification was too narrow and focused on 402/generic patterns, missing common OpenRouter 403 key-limit wording.
  2. _try_payment_fallback() itself only skips what it is told (failed_provider string); it does not discover backend identity on its own.
  3. Call sites previously could pass a non-concrete provider value (for example coarse auto/provider context) instead of the concrete resolved backend label, so skip logic could be imprecise.
  4. Because auxiliary fallback is chain-based, incorrect failed-backend identity undermines the main purpose of “advance past the backend that just failed.”

Proposed Fix (optional)

  1. Extend _is_payment_error() to treat OpenRouter 403 key/spend-limit responses as payment exhaustion signals.
  2. Persist concrete resolved backend metadata on clients and thread it through cache/call paths.
  3. Use that resolved provider label in call_llm and async_call_llm when invoking _try_payment_fallback().
  4. Keep fallback scoped to auto mode and to backend-unavailable classes (payment exhaustion + connection errors), so we do not hide request-shape or model/config bugs.
  5. Add regression tests for:
  • OpenRouter 403 key-limit path triggers fallback
  • fallback receives concrete failed backend label
  • async path mirrors sync behavior

Are you willing to submit a PR for this?

  • I'd like to fix this myself and submit a PR

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High — major feature broken, no workaroundcomp/agentCore agent loop, run_agent.py, prompt builderprovider/openrouterOpenRouter aggregatortype/bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions