Skip to content

fix(cli): probe /v1/models for custom providers in model picker#20760

Closed
dusterbloom wants to merge 1 commit into
NousResearch:mainfrom
dusterbloom:fix/model-picker-custom-providers
Closed

fix(cli): probe /v1/models for custom providers in model picker#20760
dusterbloom wants to merge 1 commit into
NousResearch:mainfrom
dusterbloom:fix/model-picker-custom-providers

Conversation

@dusterbloom

@dusterbloom dusterbloom commented May 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Custom providers now probe the live /v1/models endpoint and merge results into the model picker, matching existing behavior for user-defined providers (Section 3)
  • Resolves key_env/api_key_env from environment variables so env-backed credentials also trigger the probe
  • Config models are preserved as fallback; live models are appended (no duplicates)

Root cause

list_authenticated_providers() Section 4 (custom_providers) only read the static model: field and models: dict from config. Unlike Section 3 (user_providers), it never called fetch_api_models() to discover all available models from the server.

Changes

File What
hermes_cli/model_switch.py +20 lines: resolve key_env in grouping loop, unpack group key, add live probe block
tests/hermes_cli/test_model_switch_custom_providers.py +5 tests: live probe, dedup, no-key skip, error fallback, key_env resolution

Test plan

  • 22/22 custom provider tests pass (17 existing + 5 new)
  • Codex review passed — caught the key_env gap, now fixed
  • Manual test with local vLLM/Ollama server returning multiple models

Closes #20582

@dusterbloom
fix(cli): probe /v1/models for custom providers in model picker
be4f02e 
Custom providers only showed models from config (the singular model:
field and models: dict). Now probes the live /v1/models endpoint and
merges results, matching the existing behavior for user-defined
providers. Also resolves key_env/api_key_env from environment variables
so env-backed credentials trigger the probe.

Closes NousResearch#20582
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard labels May 6, 2026
@alt-glitch

alt-glitch commented May 6, 2026

Copy link
Copy Markdown
Collaborator

Duplicate of #19072 (and #20763) — all implement live /v1/models probe for custom_providers. Target issue: #20582.

@dusterbloom

dusterbloom commented May 6, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @alt-glitch — I see #19072 and #20763 cover the same core fix (adding fetch_api_models() to Section 4).

This PR adds two things the others don't:

  1. key_env/api_key_env resolution — when users store credentials as env var references instead of inline api_key, the grouping loop now resolves them via os.environ before building the group key. Without this, the probe silently skips env-backed providers.

  2. 5 dedicated tests: live probe, deduplication, no-key skip, error fallback, and key_env resolution.

Happy to contribute the key_env fix as a follow-up to #19072 if you'd prefer to merge that one. Just let me know what's easiest.

@dusterbloom

dusterbloom commented May 9, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by upstream — main now ships an equivalent fix in hermes_cli/model_switch.py (live /v1/models probe with replace-semantics) and includes the test_custom_providers_uses_live_models_for_multi_model_endpoint test. Verified by rebasing onto current main: the resulting diff is empty.

Will open a follow-up PR with the four edge-case tests this branch had that upstream doesn't cover yet (dedupe, no-key skip, probe-failure fallback, key_env resolution).

@dusterbloom dusterbloom closed this May 9, 2026
@dusterbloom dusterbloom deleted the fix/model-picker-custom-providers branch May 12, 2026 09:00
@alt-glitch alt-glitch mentioned this pull request May 16, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Model picker only shows one model for custom providers

2 participants

@dusterbloom @alt-glitch