Skip to content

/model shows providers and models not configured in config.yaml #6799

@ShuaiHui

Description

@ShuaiHui

Bug Description

The /model command displays providers and model lists that are inconsistent with the user's config.yaml configuration. Specifically:

  1. Providers without allowed_models in config.yaml still appear, using hardcoded model lists from _PROVIDER_MODELS or OPENROUTER_MODELS in models.py.
  2. Model lists for configured providers may differ between what /model shows and what is actually defined in config.yaml, because multiple code paths (_PROVIDER_MODELS, provider_model_ids(), OPENROUTER_MODELS) each maintain their own model lists independently of config.yaml.

Steps to Reproduce

  1. Set OPENROUTER_API_KEY in ~/.hermes/.env (or any provider's API key).
  2. Do NOT add an openrouter entry to config.yaml's providers: section.
  3. Run /model in the CLI or gateway.

Expected: OpenRouter should not appear in the provider list (since there is no config.yaml entry).

Actual: OpenRouter appears with 3 models from the hardcoded OPENROUTER_MODELS list in models.py:

OPENROUTER_MODELS = [
    ("anthropic/claude-opus-4.6", "recommended"),
    ("anthropic/claude-sonnet-4.6", ""),
    ("openai/gpt-5-mini", ""),
]

Root Cause

In model_switch.py, list_authenticated_providers() Step 1 (line 764):

for hermes_id, mdev_id in PROVIDER_TO_MODELS_DEV.items():
    # ...
    pconfig = PROVIDER_REGISTRY.get(hermes_id)
    if pconfig and pconfig.api_key_env_vars:
        env_vars = list(pconfig.api_key_env_vars)
    else:
        env_vars = pdata.get("env", [])  # falls back to models.dev

    has_creds = any(os.environ.get(ev) for ev in env_vars)
    if not has_creds:
        continue

    model_ids = curated.get(hermes_id, [])  # falls back to OPENROUTER_MODELS

Two issues here:

Issue A — Unconfigured providers appear when they should not

  • PROVIDER_REGISTRY does not contain "openrouter", so pconfig = None.
  • Falls back to pdata.get("env", [])["OPENROUTER_API_KEY"] from models.dev.
  • If OPENROUTER_API_KEY is set, has_creds = True, and the provider appears.
  • The curated dict is built from _PROVIDER_MODELS / OPENROUTER_MODELS first, then overridden by config.yaml allowed_models. But since there is no openrouter entry in config, the hardcoded list is used.
  • No mechanism to hide a provider — if credentials exist, it shows up regardless of config.

Issue B — Multiple data sources for model lists

The model list shown in /model comes from curated, which is correct when config.yaml has allowed_models. But other functions like provider_model_ids() in models.py read from _PROVIDER_MODELS (hardcoded) or live API directly, not from config.yaml. This means:

  • /model display → reads config.yaml (correct)
  • Model validation/switching → reads _PROVIDER_MODELS (may be stale/inconsistent)

Additionally, _PROVIDER_MODELS has model IDs like "gpt-5.4-mini" while config.yaml uses "gpt-5-mini" — these may not match.

Impact

  • Users see providers they did not configure
  • Model lists shown may not reflect what is actually available or configured
  • Confusing UX — config.yaml is not the single source of truth

Suggested Fix

  1. In list_authenticated_providers() Step 1: If a provider exists in PROVIDER_TO_MODELS_DEV but has no entry in config.yaml providers:, skip it (or require explicit opt-in).
  2. Unify model list sources: Either read from config.yaml everywhere, or ensure _PROVIDER_MODELS is always in sync with config.yaml. Consider making config.yaml the single source of truth for allowed models.
  3. Normalize model IDs: Ensure consistency between _PROVIDER_MODELS model IDs and what users put in config.yaml (e.g., gpt-5.4-mini vs gpt-5-mini).

Environment

  • hermes-agent: latest (NousResearch/hermes-agent)
  • OS: macOS
  • Config: ~/.hermes/config.yaml has providers for nous, copilot, deepseek, minimax-cn — but NOT openrouter
  • Env: OPENROUTER_API_KEY is set in ~/.hermes/.env

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existsarea/configConfig system, migrations, profilescomp/cliCLI entry point, hermes_cli/, setup wizardtype/bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions