fix(providers): treat custom:<name> as aggregator for vendor-prefixed slugs (#26578)#26594
Closed
briandevans wants to merge 1 commit into
Closed
Conversation
Contributor
Author
|
CI audit — both failing checks are pre-existing baselines on clean
Nothing in |
3 tasks
b70bf77 to
4c0740f
Compare
3207fb2 to
3ea80bf
Compare
… slugs (NousResearch#26578) `custom:<name>` providers front arbitrary upstream catalogs (LiteLLM, ZenMux, etc.) and commonly accept vendor-prefixed model slugs like `google/gemini-3.1-flash-lite` or `z-ai/glm-5.1`. Two places treated them as bare/unknown providers and broke that pattern: - `hermes_cli/providers.py:is_aggregator()` only returned True for providers registered in HERMES_OVERLAYS with `is_aggregator=True`. `custom:zenmux` fell through to `get_provider()` which returns None for unknown slugs, so `is_aggregator()` returned False — breaking model_switch alias resolution for vendor-prefixed model IDs. - `hermes_cli/doctor.py` vendor-prefix validation included bare `"custom"` in `providers_accepting_vendor_slugs` but not the `custom:<name>` namespace, producing a false-positive warning: "model.default 'z-ai/glm-5.1' is vendor-prefixed but model.provider is 'custom:zenmux'. Either set model.provider to 'openrouter', or drop the vendor prefix." — which is incorrect for aggregators. Fix: short-circuit on the `custom:` prefix in both call sites. Regression-guarded: with the production change reverted, all three new assertions fail (verified locally). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3ea80bf to
93836ee
Compare
Contributor
Author
|
Housekeeping: closing to keep my open-PR set focused on actively-reviewed work. This has been open ~17d without maintainer review and the surrounding code has continued to move, so it's unlikely to land as-is. The underlying fix still stands — happy to reopen and rebase if it would be useful. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two places in the CLI treated
custom:<name>providers as bare/unknown providers and broke vendor-prefixed model slugs (e.g.google/gemini-3.1-flash-lite,z-ai/glm-5.1) for users running throughcustom:zenmux,custom:litellm, etc.The bug
custom:<name>providers front arbitrary upstream catalogs (LiteLLM, ZenMux, etc.) and commonly accept vendor-prefixed model slugs, just likeopenrouter. But:hermes_cli/providers.py:is_aggregator()only returnedTruefor providers registered inHERMES_OVERLAYSwithis_aggregator=True.custom:zenmuxfell through toget_provider(), which returnsNonefor unknown slugs, sois_aggregator()returnedFalse. This broke model-alias resolution inmodel_switchfor vendor-prefixed model IDs when the current provider wascustom:<name>.hermes_cli/doctor.pyvendor-prefix validation included bare"custom"inproviders_accepting_vendor_slugsbut not thecustom:<name>namespace, producing a false-positive warning:which is incorrect —
custom:zenmuxIS an aggregator.The fix
Short-circuit on the
custom:prefix in both call sites:is_aggregator(provider)returnsTruewhenprovider.startswith("custom:"), before falling through to the registry lookup.run_doctor()'s vendor-prefix check skips whenprovider_for_policy.startswith("custom:").The custom: special-case is intentionally narrow: it does NOT make every unknown provider an aggregator (covered by a regression test).
Test plan
tests/hermes_cli/test_model_switch_custom_providers.py::test_is_aggregator_recognizes_custom_prefix—custom:zenmux,custom:ollama,custom:local-(127.0.0.1:4141)→ all Truetests/hermes_cli/test_model_switch_custom_providers.py::test_is_aggregator_unknown_provider_still_false— the custom: fast-path must not turn every unknown provider into an aggregatortests/hermes_cli/test_model_switch_custom_providers.py::test_is_aggregator_known_non_aggregator_unchanged—anthropicstill returns Falsetests/hermes_cli/test_doctor.py::test_run_doctor_does_not_warn_vendor_prefix_for_custom_named_provider—provider: custom:zenmux+default: z-ai/glm-5.1does not produce the vendor-prefix warningtest_doctor.py+test_model_switch_custom_providers.py+test_custom_provider_model_switch.py+test_custom_provider_context_length.py— 83 passed locallyAssertionError: assert False is Trueforis_aggregator("custom:zenmux")and the vendor-prefix substring still appears in doctor output; restoring the fix makes them pass.Related