Skip to content

fix(agent): propagate api_mode to vision provider resolution#8871

Closed
luyao618 wants to merge 1 commit into
NousResearch:mainfrom
luyao618:fix/vision-aux-client-api-mode
Closed

fix(agent): propagate api_mode to vision provider resolution#8871
luyao618 wants to merge 1 commit into
NousResearch:mainfrom
luyao618:fix/vision-aux-client-api-mode

Conversation

@luyao618

@luyao618 luyao618 commented Apr 13, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

resolve_vision_provider_client() in agent/auxiliary_client.py resolves api_mode from config via _resolve_task_provider_model("vision", ...) but never passes it downstream to resolve_provider_client() or _get_cached_client(). This causes vision_analyze and browser_vision to crash when the provider uses a non-default API mode (e.g., api_mode: anthropic_messages for a custom corporate Anthropic proxy).

Additionally, _normalize_aux_provider() had a for_vision branch that incorrectly stripped named custom provider identity (e.g., custom:corp-anthropiccustom) for vision tasks only.

Related Issue

Fixes #8857

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

Two independent bugs combine:

  1. api_mode not propagated: The variable resolved_api_mode is computed at line 1696 but never passed to any of the three downstream call sites. Compare with the non-vision path in call_llm() which correctly passes api_mode=resolved_api_mode. Fixed by adding api_mode=resolved_api_mode to all three downstream calls.

  2. Named custom provider lost for vision: _normalize_aux_provider(for_vision=True) returned "custom" instead of the named suffix, losing provider identity needed for config lookup. Fixed by removing the for_vision special case — named custom providers are now preserved for all tasks.

  3. Dead code cleanup: Removed the now-unused for_vision parameter from _normalize_aux_provider().

  4. Tests: Added 2 targeted tests — named custom provider preservation + api_mode forwarding verification.

How to Test

  1. Configure a custom provider with api_mode: anthropic_messages:
    model:
      default: claude-opus-4-6
      provider: custom:corp-anthropic
    custom_providers:
      - name: corp-anthropic
        base_url: https://my-proxy.example.com/anthropic
        api_mode: anthropic_messages
  2. Send an image to the agent — vision_analyze should work without crashing
  3. Run the test suite: pytest tests/agent/test_auxiliary_named_custom_providers.py -v — all 19 tests pass
  4. Run full suite: pytest tests/ -q --ignore=tests/integration --ignore=tests/e2e — passes

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS (Darwin 25.4.0, Apple Silicon), Python 3.11

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

resolve_vision_provider_client() computed resolved_api_mode from config
but never passed it to downstream resolve_provider_client() or
_get_cached_client() calls, causing custom providers with
api_mode: anthropic_messages to crash when used for vision tasks.

Also remove the for_vision special case in _normalize_aux_provider()
that incorrectly discarded named custom provider identifiers.

Fixes NousResearch#8857

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@teknium1

Copy link
Copy Markdown
Contributor

Merged via PR #9001. Your commit was cherry-picked onto current main with your authorship preserved in git log. Thanks for the clean fix, @luyao618!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vision auxiliary client drops api_mode for custom providers, causing crash

2 participants