Skip to content

Vision auxiliary client drops api_mode for custom providers, causing crash #8857

@fuyizheng3120

Description

@fuyizheng3120

Bug Description

resolve_vision_provider_client() in agent/auxiliary_client.py resolves api_mode from config via _resolve_task_provider_model("vision", ...) but never passes it downstream to resolve_provider_client() or _get_cached_client(). This causes vision_analyze and browser_vision to crash when the main provider uses api_mode: anthropic_messages (e.g., a custom corporate Anthropic proxy).

Root Cause

Two issues combine:

1. _normalize_aux_provider() strips named custom provider for vision

Line 84:

normalized = suffix if not for_vision else "custom"

When for_vision=True, custom:corp-anthropic becomes just "custom", losing the named provider identity.

2. resolve_vision_provider_client() discards resolved_api_mode

The variable resolved_api_mode is computed at line 1696 but never used in any of the three downstream paths:

  • Line 1711-1716 (explicit base_url path): resolve_provider_client("custom", ...) — missing api_mode=
  • Line 1740-1741 (auto-detect exotic provider path): resolve_provider_client(main_provider, vision_model) — missing api_mode=
  • Line 1765 (explicit non-auto path): _get_cached_client(requested, resolved_model, async_mode) — missing api_mode=

Compare with the non-vision path in call_llm() (line 2328-2334) which correctly passes api_mode=resolved_api_mode.

What Happens

  1. User configures model.provider: custom:my-proxy with api_mode: anthropic_messages
  2. call_llm(task="vision") is invoked
  3. Vision resolution creates an OpenAI SDK client (no api_mode → defaults to OpenAI format)
  4. Client sends OpenAI chat.completions.create() request to an Anthropic-format endpoint
  5. Response comes back in wrong format → ChatCompletion with choices=None
  6. Crash: NoneType has no [0] / RuntimeError from response validation

Main model conversation and compression work fine because they use the non-vision code path that correctly propagates api_mode.

Reproduction

# config.yaml
model:
  default: claude-opus-4-6
  provider: custom:corp-anthropic
custom_providers:
  - name: corp-anthropic
    base_url: https://my-proxy.example.com/anthropic
    api_key: ${ANTHROPIC_API_KEY}
    api_mode: anthropic_messages

Send an image to the agent → vision_analyze crashes.

Suggested Fix

--- a/agent/auxiliary_client.py
+++ b/agent/auxiliary_client.py
@@ -81,7 +81,7 @@
-        normalized = suffix if not for_vision else "custom"
+        normalized = suffix  # preserve named custom provider for all tasks

@@ -1711,6 +1711,7 @@
             async_mode=async_mode,
             explicit_base_url=resolved_base_url,
             explicit_api_key=resolved_api_key,
+            api_mode=resolved_api_mode,
         )

@@ -1740,7 +1741,8 @@
-                    main_provider, vision_model)
+                    main_provider, vision_model,
+                    api_mode=resolved_api_mode)

@@ -1765,7 +1767,8 @@
-    client, final_model = _get_cached_client(requested, resolved_model, async_mode)
+    client, final_model = _get_cached_client(requested, resolved_model, async_mode,
+                                             api_mode=resolved_api_mode)

Environment

  • Hermes Agent v0.8.0
  • macOS, corporate Anthropic proxy with api_mode: anthropic_messages
  • Claude Opus 4.6 as main model

Related Issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions