Skip to content

fix(api-server): apply non-vision image fallback#23743

Closed
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:codex/23733-api-server-nonvision-image-route
Closed

fix(api-server): apply non-vision image fallback#23743
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:codex/23733-api-server-nonvision-image-route

Conversation

@LeonSGP43

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes the provider-profile chat_completions path so non-vision models still run Hermes' shared image fallback before transport kwargs are built. Without this, /v1/chat/completions can forward raw image_url parts to non-vision models and trigger 400s even though the legacy path already strips or describes those images.

Related Issue

Fixes #23733

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • updated /run_agent.py so the provider-profile chat_completions branch calls _prepare_messages_for_non_vision_model(...) before build_kwargs(...)
  • added a regression test in /tests/run_agent/test_run_agent.py that proves the provider-profile path now forwards transformed non-vision messages instead of raw multimodal payloads
  • re-ran /tests/gateway/test_api_server_multimodal.py to verify the API server multimodal path still behaves correctly end to end

How to Test

  1. uv run --frozen ruff check run_agent.py tests/run_agent/test_run_agent.py
  2. uv run --frozen pytest -q -o addopts='' tests/run_agent/test_run_agent.py -k 'provider_profile_applies_non_vision_fallback or converts_multimodal_user_image_to_text or reuses_cached_image_analysis_for_duplicate_images'
  3. uv run --frozen pytest -q -o addopts='' tests/gateway/test_api_server_multimodal.py

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 15.5

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

  • Targeted regression pytest: 3 passed, 321 deselected
  • API server multimodal suite: 23 passed
  • Clean-env collect-only smoke: 347 tests collected

@teknium1

Copy link
Copy Markdown
Contributor

This looks implemented on current main now. Automated hermes-sweeper review found the provider-profile chat completions path already runs the shared non-vision image fallback before building transport kwargs.

Evidence:

  • agent/chat_completion_helpers.py:742 now calls agent._prepare_messages_for_non_vision_model(api_messages) inside the registered provider-profile branch before _ct.build_kwargs(...) receives messages=api_messages.
  • run_agent.py:4405 defines _prepare_messages_for_non_vision_model(...), which strips/replaces image parts only for non-vision models and is a no-op for vision-capable/no-image requests.
  • Commit 563b4d9e51a46cc421e327b351cb7efe1ccb151b re-applied the same fix after _build_api_kwargs was extracted from run_agent.py into agent/chat_completion_helpers.py.
  • The linked issue Bug: image routing bypassed on api_server /v1/chat/completions — non-vision models receive raw image_url (400) #23733 is already closed as resolved on current main.

Thanks for the focused fix and regression coverage here; the behavior this PR targeted has landed on main.

@teknium1 teknium1 closed this Jun 11, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jun 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent loop, run_agent.py, prompt builder comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: image routing bypassed on api_server /v1/chat/completions — non-vision models receive raw image_url (400)

3 participants