Skip to content

Custom provider model validation fails when /v1/models endpoint is unavailable #12153

@happy5318

Description

@happy5318

Problem

When using a custom OpenAI-compatible provider whose /v1/models endpoint returns errors (e.g., HTTP 500 or malformed response), Hermes fails to validate and switch to models defined in custom_providers config, even though:

  1. The model IS listed in the config's models field
  2. The chat completions API works perfectly fine
  3. Only the /v1/models endpoint is broken

Root Cause

Two issues in hermes_cli/models.py:

Issue 1: Provider format mismatch

The condition if normalized == "custom" doesn't match custom:provider-name format:

# Current code (line ~2013)
if normalized == "custom":  # Only matches "custom", not "custom:some_provider"

Issue 2: No fallback to config models list

When /v1/models is unavailable, Hermes returns failure immediately instead of checking the user-defined models list in config:

# Current behavior: returns failure when api_models is None
# Expected: fallback to custom_providers[].models for validation

Proposed Fix

  1. Match all custom:* formats:
if normalized == "custom" or normalized.startswith("custom:"):
  1. Fallback to config models list when API is unavailable:
if api_models is None:
    # Try to validate from config's models list
    from hermes_cli.config import get_compatible_custom_providers
    custom_providers = get_compatible_custom_providers()
    provider_name = provider.replace("custom:", "")
    for cp in custom_providers:
        if cp.get("name") == provider_name:
            config_models = cp.get("models", {})
            if requested in config_models:
                return {"accepted": True, ...}

Why This Matters

Some OpenAI-compatible providers (especially regional/domestic cloud providers) have broken /v1/models endpoints but fully functional chat APIs. Users who have already defined their models in config should not be blocked by a broken discovery endpoint.

Workaround

Currently requires patching hermes_cli/models.py locally.

Environment

  • Hermes version: latest
  • Python: 3.11+

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High — major feature broken, no workaroundarea/configConfig system, migrations, profilescomp/cliCLI entry point, hermes_cli/, setup wizardtype/bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions