Skip to content

feat: allow custom provider capability overrides#8942

Closed
MestreY0d4-Uninter wants to merge 2 commits into
NousResearch:mainfrom
MestreY0d4-Uninter:bughunt/issue-8731-feature-request-allow-manual-cap
Closed

feat: allow custom provider capability overrides#8942
MestreY0d4-Uninter wants to merge 2 commits into
NousResearch:mainfrom
MestreY0d4-Uninter:bughunt/issue-8731-feature-request-allow-manual-cap

Conversation

@MestreY0d4-Uninter

Copy link
Copy Markdown
Contributor

Summary

  • allow custom_providers[].models.<model>.capabilities to override model metadata for private/custom endpoints
  • surface those overrides in the shared /model switch pipeline, including synthetic capability metadata when models.dev has no entry
  • let the runtime honor custom_providers reasoning overrides for custom endpoints, and document the new config shape

What this adds

Supported per-model capability keys under custom_providers:

  • vision
  • reasoning
  • tools
  • streaming

Example:

custom_providers:
  - name: my-local-vllm
    base_url: http://localhost:8000/v1
    models:
      my-llava-model:
        context_length: 8192
        capabilities:
          vision: true
          reasoning: false
          tools: true
          streaming: true

Behavior

  • capability resolution now prefers explicit custom_providers per-model overrides over catalog metadata
  • /model can show capability info even for custom/private models that are absent from models.dev
  • AIAgent._supports_reasoning_extra_body() now respects custom_providers per-model reasoning overrides matched by active base_url + model
  • config validation warns on unknown capability keys

Test plan

  • python3 -m py_compile agent/models_dev.py hermes_cli/config.py hermes_cli/model_switch.py run_agent.py tests/agent/test_models_dev.py tests/hermes_cli/test_config_validation.py tests/hermes_cli/test_model_switch_custom_providers.py tests/run_agent/test_run_agent.py
  • uv run pytest tests/agent/test_models_dev.py tests/hermes_cli/test_config_validation.py tests/hermes_cli/test_model_switch_custom_providers.py -o addopts='' -q
  • uv run pytest tests/run_agent/test_run_agent.py::TestBuildApiKwargs::test_reasoning_sent_for_custom_provider_capability_override tests/run_agent/test_run_agent.py::TestBuildApiKwargs::test_reasoning_override_matches_custom_provider_base_url_not_name -o addopts='' -q

Closes #8731.

@MestreY0d4-Uninter MestreY0d4-Uninter force-pushed the bughunt/issue-8731-feature-request-allow-manual-cap branch from 5efc0a0 to dba2384 Compare April 13, 2026 10:54
@MestreY0d4-Uninter

Copy link
Copy Markdown
Contributor Author

Validation / CI note:

Local validation for the actual change set is green:

  • python3 -m py_compile agent/models_dev.py hermes_cli/config.py hermes_cli/model_switch.py run_agent.py tests/agent/test_models_dev.py tests/hermes_cli/test_config_validation.py tests/hermes_cli/test_model_switch_custom_providers.py tests/run_agent/test_run_agent.py
  • uv run pytest tests/agent/test_models_dev.py tests/hermes_cli/test_config_validation.py tests/hermes_cli/test_model_switch_custom_providers.py -o addopts='' -q
  • uv run pytest tests/run_agent/test_run_agent.py::TestBuildApiKwargs::test_reasoning_sent_for_custom_provider_capability_override tests/run_agent/test_run_agent.py::TestBuildApiKwargs::test_reasoning_override_matches_custom_provider_base_url_not_name -o addopts='' -q

Current CI red does not appear to be evidence that this PR regressed the repo:

  1. test
  • The failure list is broad and mostly outside this diff (memory, gateway session race guards, auth/provider gate, voice mode, interrupt plumbing, browser camofox state, etc.).
  • The same clusters are already failing on main push runs as well (for example 24339048768).
  • This PR's failures do not cluster around the custom-provider capability override code path.
  1. build-and-push
  • Fails with the same Docker/npm error already seen on main:
    • npm ERR! syscall spawn git
    • ENOENT
  • That's the same class of Dockerfile failure we already observed on current mainline runs.
  1. docs-site-checks
  • The diagram linter is failing on unrelated docs files / existing ASCII diagram issues outside this PR's diff.
  • The file changed in this PR (website/docs/integrations/providers.md) is reported as clean by the same job.

So at this point the red checks look like broad-repo / operational noise, not a targeted regression from this custom-provider capability patch.

@MestreY0d4-Uninter

Copy link
Copy Markdown
Contributor Author

Refresh rebuilt from origin/main and pushed to refresh/8942. Validation: py_compile on changed .py files passed; focused pytest on changed test files ran and hit one existing failure in tests/run_agent/test_run_agent.py::TestStreamingApiCall::test_tool_call_accumulation (expected web_search, got search).

@MestreY0d4-Uninter

Copy link
Copy Markdown
Contributor Author

Additional validation note:

The one extra focused failure I hit while checking this refreshed branch is reproducible on clean origin/main too:

  • /home/ubuntu/hermes-all-venv/bin/python -m pytest -o addopts= -q tests/run_agent/test_run_agent.py::TestStreamingApiCall::test_tool_call_accumulation\n- current origin/main fails with the same "search" vs expected "web_search" assertion\n\nSo that remaining failure is baseline repo noise in current main, not a regression unique to the custom-provider capability override refresh.

@MestreY0d4-Uninter

Copy link
Copy Markdown
Contributor Author

Audit completed (2026-04-19)

  • Branch refreshed from current origin/main (SHA: 6af04474)
  • Validation: Focused tests passing (capability overrides)
  • Impact: Medium (allows manual provider capability overrides)
  • Risk: Medium (484 lines, 9 files, feature addition)
  • Age: Very old PR (5/5)

Baseline Noise Note: Test failures reproduce on clean origin/main — not a regression from this PR.

Recommendation: DECISION NEEDED — Feature adds custom provider capability overrides.

Question for maintainers: Is manual capability override still needed?

  • If YES → merge after review
  • If NO → close as deprecated

Feature value: Allows users to manually override provider capabilities (e.g., force a model to support vision or tools).


Part of batch audit: 23 PRs audited, 2 closed (absorbed), 21 refreshed
Batch 3 — Feature decision required

@MestreY0d4-Uninter

Copy link
Copy Markdown
Contributor Author

🔔 Ready for maintainer review

Esta PR foi validada como parte da auditoria completa de 2026-04-19.

Status:

Ação necessária: Review e merge (ou decisão de feature para #9209, #8942).


Audit batch 3: 4 PRs com validação mínima concluída

@MestreY0d4-Uninter MestreY0d4-Uninter force-pushed the bughunt/issue-8731-feature-request-allow-manual-cap branch from dba2384 to e6c3098 Compare April 23, 2026 23:39
@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/agent Core agent loop, run_agent.py, prompt builder comp/cli CLI entry point, hermes_cli/, setup wizard area/config Config system, migrations, profiles labels Apr 23, 2026
@MestreY0d4-Uninter MestreY0d4-Uninter force-pushed the bughunt/issue-8731-feature-request-allow-manual-cap branch 2 times, most recently from d0fe8aa to 161594c Compare April 25, 2026 15:06
@MestreY0d4-Uninter

Copy link
Copy Markdown
Contributor Author

Audit/update 2026-04-25:

  • Force-pushed d0fe8aaf -> 161594ca with the minimal get_model_info() fix and canonical author metadata.
  • Focused local validation: 358 passed in 26.55s after minimal get_model_info() fix.
  • Recommendation: Applied minimal custom-provider get_model_info() fix and canonical author rewrite; ready for re-review after CI.

This was part of the open-PR cleanup pass against current upstream/main.

@MestreY0d4-Uninter

Copy link
Copy Markdown
Contributor Author

CI follow-up from the 2026-04-25 cleanup pass:

  • Contributor Attribution now passes on the rewritten head (161594ca).
  • Remaining CI is red: Supply Chain Audit, test, and e2e.
  • I reproduced the supply-chain scanner trigger locally. It is firing on stale-branch diff noise (hermes_cli/setup.py and skills/productivity/google-workspace/scripts/setup.py) because the PR diff against current main spans ~490 files.
  • The minimal get_model_info() fix itself was validated locally with the focused suite: tests/agent/test_models_dev.py, tests/hermes_cli/test_config_validation.py, tests/hermes_cli/test_model_switch_custom_providers.py, and tests/run_agent/test_run_agent.py — 358 passed.

So this PR is not merge-ready as-is despite the minimal fix and attribution pass. Next step would be a clean rebase/reconstruction onto current main so CI evaluates only the custom-provider capability override change.

@MestreY0d4-Uninter MestreY0d4-Uninter force-pushed the bughunt/issue-8731-feature-request-allow-manual-cap branch from 161594c to 07a8f3f Compare April 25, 2026 15:41
@MestreY0d4-Uninter

Copy link
Copy Markdown
Contributor Author

Refreshed this PR onto current main with a normal non-force push.

Current PR scope remains limited to custom-provider capability overrides:

  • agent/models_dev.py
  • hermes_cli/config.py
  • hermes_cli/model_switch.py
  • run_agent.py
  • targeted tests and provider docs

Validation:

  • python3 -m py_compile agent/models_dev.py hermes_cli/config.py hermes_cli/model_switch.py run_agent.py tests/agent/test_models_dev.py tests/hermes_cli/test_config_validation.py tests/hermes_cli/test_model_switch_custom_providers.py tests/run_agent/test_run_agent.py -> passed
  • HERMES_TEST_WORKERS=2 scripts/run_tests.sh tests/agent/test_models_dev.py tests/hermes_cli/test_config_validation.py tests/hermes_cli/test_model_switch_custom_providers.py tests/run_agent/test_run_agent.py::TestBuildApiKwargs::test_reasoning_sent_for_custom_provider_capability_override tests/run_agent/test_run_agent.py::TestBuildApiKwargs::test_reasoning_override_matches_custom_provider_base_url_not_name -q -> 62 passed
  • Post-push PR diff file list verified: 9 scoped files only
  • Attribution, docs, check, e2e, Nix, and supply-chain checks are green

The remaining broad Tests workflow is still red. Its failure profile matches current main closely; the one PR-only failure observed (test_live_cap_applied_after_read_add) is outside this PR's custom-provider scope and passes locally in the refreshed worktree.

@MestreY0d4-Uninter

Copy link
Copy Markdown
Contributor Author

Closing this for now. Provider capability overrides are a broader config/model-switching feature, and this branch now conflicts in config/model-switch tests. It should be rebuilt from current main if still desired.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent loop, run_agent.py, prompt builder comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: Allow manual capability declaration (vision/reasoning/tools) in custom_providers

2 participants