Skip to content

fix(agent): call _prepare_messages_for_non_vision_model on chat_completions profile path#23818

Closed
gsskk wants to merge 1 commit into
NousResearch:mainfrom
gsskk:fix/api-server-non-vision-image-fallback
Closed

fix(agent): call _prepare_messages_for_non_vision_model on chat_completions profile path#23818
gsskk wants to merge 1 commit into
NousResearch:mainfrom
gsskk:fix/api-server-non-vision-image-fallback

Conversation

@gsskk

@gsskk gsskk commented May 11, 2026

Copy link
Copy Markdown

Fixes #23733.

What & Why

_build_api_kwargs (run_agent.py) had two parallel return paths for chat_completions:

  • Legacy branch (get_provider_profile()None, unknown provider): called _prepare_messages_for_non_vision_model to strip image_url parts before they reached non-vision models.
  • Profile branch (registered providers — opencode-go, deepseek, kimi, openrouter, gemini, etc.): did not call it, so image_url parts passed through to the upstream provider.

Result: text-only models on any registered provider 400'd with unknown variant 'image_url' whenever a multimodal request reached /v1/chat/completions (and the same path from any caller that lets the agent build messages from images). The gateway-adapter path (gateway/run.py_decide_image_input_mode_enrich_message_with_vision) and the codex_responses profile path both already handle this; the chat_completions profile branch was the lone gap.

The fix mirrors the legacy branch: one call to _prepare_messages_for_non_vision_model(api_messages) immediately before _ct.build_kwargs(...) in the profile branch. The function early-returns when no message contains image parts and again when _model_supports_vision() is true, so the added call is free on the common paths.

How to test

# The targeted regression tests (new + existing):
uv run python -m pytest tests/run_agent/test_vision_aware_preprocessing.py -v
# 13 passed (11 pre-existing + 2 new in TestBuildApiKwargsProfileBranch)

# Adjacent test directories — full pass, no regressions from this change:
uv run python -m pytest tests/run_agent/ tests/agent/test_image_routing.py \
    tests/agent/transports/test_chat_completions.py tests/providers/ -q

The added TestBuildApiKwargsProfileBranch covers both directions on the profile path:

  • test_profile_branch_strips_images_for_non_vision_models — fails on main, passes with the fix. Sets provider=opencode-go + model=deepseek-v4-pro, calls _build_api_kwargs with an image_url user message, and asserts the messages handed to the transport contain no image_url parts.
  • test_profile_branch_passes_through_when_model_supports_vision — already-passes guard so vision-capable models don't regress to text-only.

Platforms

Tested on Linux (Debian trixie). Change is pure Python message-shape manipulation; no file I/O, process management, or terminal handling.

Notes

The full root-cause analysis (including verification on origin/main 64145a199) is in the linked issue.

…etions profile path

Mirror the legacy branch: registered providers (opencode-go, deepseek, kimi,
openrouter, ...) were skipping the non-vision image fallback because the call
sat only in the legacy branch of _build_api_kwargs. Profile-path providers
forwarded image_url parts unchanged, and text-only models 400'd with
"unknown variant `image_url`".

The function short-circuits for vision-capable models and image-less turns,
so the added call is free on the common paths.

Regression test in test_vision_aware_preprocessing.py exercises the profile
branch end-to-end via _build_api_kwargs: a non-vision model on a registered
provider must receive text-only messages at the transport layer.

Fixes #23733
@gsskk

gsskk commented May 11, 2026

Copy link
Copy Markdown
Author

Closing as duplicate — #23743 (and #23750) cover the same fix on the chat_completions profile branch and were opened earlier today. Apologies for the noise; should have checked the open-PR list right after filing the issue. Leaving a pointer in case any of the regression test from this branch (TestBuildApiKwargsProfileBranch in tests/run_agent/test_vision_aware_preprocessing.py) is useful as additional coverage for whichever fix lands.

@gsskk gsskk closed this May 11, 2026
@gsskk gsskk deleted the fix/api-server-non-vision-image-fallback branch May 11, 2026 14:22
@daimon-nous daimon-nous Bot added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent loop, run_agent.py, prompt builder duplicate This issue or pull request already exists labels May 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent loop, run_agent.py, prompt builder duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: image routing bypassed on api_server /v1/chat/completions — non-vision models receive raw image_url (400)

1 participant