Skip to content

fix(vision): apply non-aggregator provider fix to vision routing (follow-up to #5091)#5376

Closed
Mibayy wants to merge 1 commit into
NousResearch:mainfrom
Mibayy:fix/vision-non-aggregator-provider
Closed

fix(vision): apply non-aggregator provider fix to vision routing (follow-up to #5091)#5376
Mibayy wants to merge 1 commit into
NousResearch:mainfrom
Mibayy:fix/vision-non-aggregator-provider

Conversation

@Mibayy

@Mibayy Mibayy commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

Root cause

Closes #5366.

PR #5091 fixed _resolve_auto() so that users on non-aggregator providers (Alibaba, DeepSeek, ZAI, Gemini Direct, etc.) have their main provider used for auxiliary tasks instead of silently failing with a generic error.

However, resolve_vision_provider_client() was not updated at the same time. It has its own "auto" routing path that only iterates _VISION_AUTO_PROVIDER_ORDER:

_VISION_AUTO_PROVIDER_ORDER = (
    "openrouter",
    "nous",
    "openai-codex",
    "anthropic",
    "custom",
)

_preferred_main_vision_provider() only promotes the user's main provider when it already belongs to that tuple. Providers like Alibaba, DeepSeek, ZAI, Gemini, etc. are absent → the entire loop runs, all backends return None, and vision silently returns (None, None, None) → vision tool fails every time.

Fix

Mirror the exact same logic from the _resolve_auto() fix in #5091, inserted before the _VISION_AUTO_PROVIDER_ORDER loop in the "auto" branch of resolve_vision_provider_client():

# ── Non-aggregator main provider: try first ──
main_provider = _read_main_provider()
main_model = _read_main_model()
if (main_provider
        and main_model
        and main_provider not in _AGGREGATOR_PROVIDERS
        and main_provider not in _VISION_AUTO_PROVIDER_ORDER
        and main_provider not in ("auto", "")):
    sync_client, resolved_model = resolve_provider_client(main_provider, main_model)
    if sync_client is not None:
        return _finalize(main_provider, sync_client, resolved_model or main_model)

All existing paths (resolved_base_url, explicit provider, _VISION_AUTO_PROVIDER_ORDER loop) are unchanged.

Why this fixes the report

Users experiencing "vision tool keep failing" who ask "why can't it just use the default model?" are on providers not in _VISION_AUTO_PROVIDER_ORDER. With this fix, vision routing tries their configured main provider first, exactly as _resolve_auto() does for compression/web-extract/etc.

Test plan

  • User on Alibaba/DeepSeek/ZAI: send an image → vision tool uses their main provider model
  • User on OpenRouter: unchanged behaviour — openrouter still heads the priority list
  • User with no provider configured: main_provider = "" → condition is false → unchanged fallback chain

🤖 Generated with Claude Code

@Mibayy @claude
fix(vision): apply non-aggregator provider fix to vision routing (Nou…
22b7c91 
…sResearch#5091 follow-up)

PR NousResearch#5091 fixed _resolve_auto() so that users on non-aggregator providers
(Alibaba, DeepSeek, ZAI, Gemini Direct, etc.) have their main provider
used for auxiliary tasks instead of silently failing.

However, resolve_vision_provider_client() has its own "auto" routing
path that was NOT updated. It only checks _VISION_AUTO_PROVIDER_ORDER
(openrouter, nous, openai-codex, anthropic, custom) and promotes the
main provider only when it already belongs to that list — so non-aggregator
providers were skipped entirely, causing the vision tool to always fail
for those users ("vision tool keep failing").

Fix: mirror the same non-aggregator check that _resolve_auto() received
in NousResearch#5091, inserted before the _VISION_AUTO_PROVIDER_ORDER loop in the
"auto" branch of resolve_vision_provider_client(). If the user's main
provider is not an aggregator and not already in _VISION_AUTO_PROVIDER_ORDER,
we attempt it via resolve_provider_client() first.

Closes NousResearch#5366

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@teknium1 teknium1 mentioned this pull request Apr 8, 2026
Merged
teknium1 pushed a commit that referenced this pull request Apr 8, 2026
@Mibayy @teknium1
fix(vision): try main provider first in vision auto-detection
be7973a 
Users on non-aggregator providers (DeepSeek, Alibaba, ZAI, Gemini, etc.)
had vision silently fail because resolve_vision_provider_client()'s auto
path only iterates _VISION_AUTO_PROVIDER_ORDER (openrouter, nous, codex,
anthropic, custom). Their main provider was never tried.

Mirror the same non-aggregator main-provider-first check that
_resolve_auto() already has for non-vision tasks. Combined with the
named custom provider fix in #5978, this also covers users whose main
provider is a named custom endpoint.

Cherry-picked from PR #5376 by Mibay. Closes #5366.
@teknium1 teknium1 mentioned this pull request Apr 8, 2026
Merged
teknium1 pushed a commit that referenced this pull request Apr 8, 2026
@Mibayy @teknium1
fix(vision): try main provider first in vision auto-detection with fa…
c43c155 
…llback

Users on non-aggregator providers (DeepSeek, Alibaba, ZAI, Gemini, etc.)
had vision silently fail because resolve_vision_provider_client()'s auto
path only iterates _VISION_AUTO_PROVIDER_ORDER (openrouter, nous, codex,
anthropic, custom). Their main provider was never tried.

Mirror the same non-aggregator main-provider-first check that
_resolve_auto() already has for non-vision tasks. Combined with the
named custom provider fix in #5978, this also covers users whose main
provider is a named custom endpoint.

Add vision-specific API fallback in call_llm: if the auto-resolved main
provider returns 400/422 (model doesn't support vision), fall back through
the strict vision backends which use known vision-capable models. This
handles the case where the user's main model is text-only.

Cherry-picked from PR #5376 by Mibay. Closes #5366.
teknium1 pushed a commit that referenced this pull request Apr 8, 2026
@Mibayy @teknium1
fix(vision): simplify vision auto-detection to openrouter → nous → ac…
e1a9d12 
…tive provider

Simplify the vision auto-detection chain from 5 backends (openrouter,
nous, codex, anthropic, custom) down to 3:

  1. OpenRouter  (known vision-capable default model)
  2. Nous Portal (known vision-capable default model)
  3. Active provider + model (whatever the user is running)
  4. Stop

This is simpler and more predictable. The active provider step uses
resolve_provider_client() which handles all provider types including
named custom providers (from #5978).

Removed the complex preferred-provider promotion logic and API-level
fallback — the chain is short enough that it doesn't need them.

Based on PR #5376 by Mibay. Closes #5366.
teknium1 pushed a commit that referenced this pull request Apr 8, 2026
@Mibayy @teknium1
fix(vision): simplify vision auto-detection to openrouter → nous → ac…
ab271eb 
…tive provider

Simplify the vision auto-detection chain from 5 backends (openrouter,
nous, codex, anthropic, custom) down to 3:

  1. OpenRouter  (known vision-capable default model)
  2. Nous Portal (known vision-capable default model)
  3. Active provider + model (whatever the user is running)
  4. Stop

This is simpler and more predictable. The active provider step uses
resolve_provider_client() which handles all provider types including
named custom providers (from #5978).

Removed the complex preferred-provider promotion logic and API-level
fallback — the chain is short enough that it doesn't need them.

Based on PR #5376 by Mibay. Closes #5366.
@teknium1

teknium1 commented Apr 8, 2026

Copy link
Copy Markdown
Contributor

Merged via PR #6041 with a simplified vision auto-detection chain (openrouter → nous → active provider → stop). Your commit was cherry-picked with authorship preserved. The chain was streamlined from 5 backends to 3 per Teknium's direction — codex/anthropic/custom removed, active provider added as final fallback. Thanks for identifying the gap, @Mibayy!

@teknium1 teknium1 closed this Apr 8, 2026
Tommyeds pushed a commit to Tommyeds/hermes-agent that referenced this pull request Apr 12, 2026
@Mibayy
fix(vision): simplify vision auto-detection to openrouter → nous → ac…
ef74e63 
…tive provider

Simplify the vision auto-detection chain from 5 backends (openrouter,
nous, codex, anthropic, custom) down to 3:

  1. OpenRouter  (known vision-capable default model)
  2. Nous Portal (known vision-capable default model)
  3. Active provider + model (whatever the user is running)
  4. Stop

This is simpler and more predictable. The active provider step uses
resolve_provider_client() which handles all provider types including
named custom providers (from NousResearch#5978).

Removed the complex preferred-provider promotion logic and API-level
fallback — the chain is short enough that it doesn't need them.

Based on PR NousResearch#5376 by Mibay. Closes NousResearch#5366.
angelburgosrosado pushed a commit to angelburgosrosado/hermes-agent that referenced this pull request Apr 27, 2026
@Mibayy @teknium1
fix(vision): simplify vision auto-detection to openrouter → nous → ac…
02851b9 
…tive provider

Simplify the vision auto-detection chain from 5 backends (openrouter,
nous, codex, anthropic, custom) down to 3:

  1. OpenRouter  (known vision-capable default model)
  2. Nous Portal (known vision-capable default model)
  3. Active provider + model (whatever the user is running)
  4. Stop

This is simpler and more predictable. The active provider step uses
resolve_provider_client() which handles all provider types including
named custom providers (from NousResearch#5978).

Removed the complex preferred-provider promotion logic and API-level
fallback — the chain is short enough that it doesn't need them.

Based on PR NousResearch#5376 by Mibay. Closes NousResearch#5366.
angelburgosrosado pushed a commit to angelburgosrosado/hermes-agent that referenced this pull request Apr 28, 2026
@Mibayy @teknium1
fix(vision): simplify vision auto-detection to openrouter → nous → ac…
b97c817 
…tive provider

Simplify the vision auto-detection chain from 5 backends (openrouter,
nous, codex, anthropic, custom) down to 3:

  1. OpenRouter  (known vision-capable default model)
  2. Nous Portal (known vision-capable default model)
  3. Active provider + model (whatever the user is running)
  4. Stop

This is simpler and more predictable. The active provider step uses
resolve_provider_client() which handles all provider types including
named custom providers (from NousResearch#5978).

Removed the complex preferred-provider promotion logic and API-level
fallback — the chain is short enough that it doesn't need them.

Based on PR NousResearch#5376 by Mibay. Closes NousResearch#5366.
02356abc pushed a commit to 02356abc/hermes-agent that referenced this pull request May 14, 2026
@Mibayy @teknium1
fix(vision): simplify vision auto-detection to openrouter → nous → ac…
40c8440 
…tive provider

Simplify the vision auto-detection chain from 5 backends (openrouter,
nous, codex, anthropic, custom) down to 3:

  1. OpenRouter  (known vision-capable default model)
  2. Nous Portal (known vision-capable default model)
  3. Active provider + model (whatever the user is running)
  4. Stop

This is simpler and more predictable. The active provider step uses
resolve_provider_client() which handles all provider types including
named custom providers (from NousResearch#5978).

Removed the complex preferred-provider promotion logic and API-level
fallback — the chain is short enough that it doesn't need them.

Based on PR NousResearch#5376 by Mibay. Closes NousResearch#5366.
olympus-terminal pushed a commit to olympus-terminal/hermes-agent that referenced this pull request May 16, 2026
@Mibayy @teknium1
fix(vision): simplify vision auto-detection to openrouter → nous → ac…
f0ccf27 
…tive provider

Simplify the vision auto-detection chain from 5 backends (openrouter,
nous, codex, anthropic, custom) down to 3:

  1. OpenRouter  (known vision-capable default model)
  2. Nous Portal (known vision-capable default model)
  3. Active provider + model (whatever the user is running)
  4. Stop

This is simpler and more predictable. The active provider step uses
resolve_provider_client() which handles all provider types including
named custom providers (from NousResearch#5978).

Removed the complex preferred-provider promotion logic and API-level
fallback — the chain is short enough that it doesn't need them.

Based on PR NousResearch#5376 by Mibay. Closes NousResearch#5366.
gweeteve pushed a commit to gweeteve/hermes-agent that referenced this pull request Jun 2, 2026
@Mibayy @teknium1
fix(vision): simplify vision auto-detection to openrouter → nous → ac…
1fd8214 
…tive provider

Simplify the vision auto-detection chain from 5 backends (openrouter,
nous, codex, anthropic, custom) down to 3:

  1. OpenRouter  (known vision-capable default model)
  2. Nous Portal (known vision-capable default model)
  3. Active provider + model (whatever the user is running)
  4. Stop

This is simpler and more predictable. The active provider step uses
resolve_provider_client() which handles all provider types including
named custom providers (from NousResearch#5978).

Removed the complex preferred-provider promotion logic and API-level
fallback — the chain is short enough that it doesn't need them.

Based on PR NousResearch#5376 by Mibay. Closes NousResearch#5366.
Egavasyug pushed a commit to Egavasyug/hermes-agent that referenced this pull request Jun 10, 2026
@Mibayy @teknium1
fix(vision): simplify vision auto-detection to openrouter → nous → ac…
64ff91f 
…tive provider

Simplify the vision auto-detection chain from 5 backends (openrouter,
nous, codex, anthropic, custom) down to 3:

  1. OpenRouter  (known vision-capable default model)
  2. Nous Portal (known vision-capable default model)
  3. Active provider + model (whatever the user is running)
  4. Stop

This is simpler and more predictable. The active provider step uses
resolve_provider_client() which handles all provider types including
named custom providers (from NousResearch#5978).

Removed the complex preferred-provider promotion logic and API-level
fallback — the chain is short enough that it doesn't need them.

Based on PR NousResearch#5376 by Mibay. Closes NousResearch#5366.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Vision tool failed

2 participants

@Mibayy @teknium1