Skip to content

fix(model-switcher): custom endpoint persistence, pre-fill, and silent validation failures#3264

Open
Mibayy wants to merge 1 commit into
NousResearch:mainfrom
Mibayy:fix/model-switcher-custom-endpoint-3263
Open

fix(model-switcher): custom endpoint persistence, pre-fill, and silent validation failures#3264
Mibayy wants to merge 1 commit into
NousResearch:mainfrom
Mibayy:fix/model-switcher-custom-endpoint-3263

Conversation

@Mibayy

@Mibayy Mibayy commented Mar 26, 2026

Copy link
Copy Markdown
Contributor

Closes #3263

Three independent fixes for the custom endpoint experience with Ollama, LM Studio, and other local/custom OpenAI-compatible servers.


Fix 1 — Surface connection errors (silent validation failure)

File: hermes_cli/model_switch.py

The except Exception block in Step 5 was swallowing all validation errors and returning accepted: True, so switching to an unreachable http://localhost:11434/v1 would report "Switched successfully" even when Ollama wasn't running.

Before:

> /model llama3
Switched to llama3 (custom)
[chat proceeds to fail silently]

After:

> /model llama3
Error: Could not reach custom endpoint: Connection refused

The fix detects connection-class errors (refused, timeout, 401, 403, 404) specifically on custom/localhost targets. Cloud providers keep the previous lenient behaviour — a temporary OpenRouter outage should not block switching to a model that's likely fine.


Fix 2 — Pre-fill base_url and model from config.yaml

File: hermes_cli/main.py_model_flow_custom()

The interactive setup flow read OPENAI_BASE_URL from env but ignored model.base_url and model.default in config.yaml, presenting blank prompts every time even after a successful previous setup.

Before:

Custom OpenAI-compatible endpoint configuration:

API base URL [e.g. https://api.example.com/v1]: _
API key [optional]: _
Model name (e.g. gpt-4, llama-3-70b): _

After:

Custom OpenAI-compatible endpoint configuration:
  Current URL: http://localhost:11434/v1
  Current model: llama3

  Press Enter to keep the current value shown in [brackets].

API base URL [http://localhost:11434/v1]: _
API key [optional, press Enter to skip]: _
Model name [llama3]: _

Empty input → keeps the current value. The config is read from config.yaml first, with env vars as fallback.


Fix 3 — config.yaml custom endpoint survives OPENROUTER_API_KEY in .env

File: hermes_cli/runtime_provider.py

When requested_norm was "auto" (normal CLI startup) and cfg_provider was "custom", use_config_base_url was not set. The base_url resolution chain then fell through to OPENROUTER_BASE_URL, flipping the active provider back to OpenRouter on every restart.

The existing #1760 fix only handled requested_norm == "custom" explicitly, missing the "auto" path used at startup.

Fix: added elif cfg_provider == "custom" inside the "auto" block so that an explicitly saved custom endpoint always wins, regardless of what env vars are present.


Tests

8 new tests in tests/test_model_switcher_custom_3263.py:

Test Covers
test_connection_refused_surfaced_for_localhost Fix 1 — refused → failure
test_timeout_surfaced_for_custom_endpoint Fix 1 — timeout → failure
test_404_surfaced_for_custom_endpoint Fix 1 — 404 → failure
test_cloud_provider_validation_error_still_accepts Fix 1 — cloud keeps lenient path
test_model_flow_custom_prefills_from_config Fix 2 — hints and empty-input defaults
test_model_flow_custom_uses_config_base_url_when_empty_input Fix 2 — config URL used for probing
test_custom_config_wins_over_openrouter_key Fix 3 — OPENROUTER_API_KEY doesn't flip provider
test_openrouter_key_still_works_without_custom_config Fix 3 — OpenRouter path unaffected

44 total pass (36 existing + 8 new).

…t failures

Closes NousResearch#3263

Three distinct fixes for custom endpoint reliability:

1. Surface connection errors in model_switch.py (silent validation failure)
   The except block in Step 5 was returning accepted:True for all exceptions,
   including connection refused and timeouts on local endpoints. This reported
   a successful switch to an unreachable Ollama/LM Studio server.
   Fix: detect connection-class errors (refused, timeout, 401, 403, 404) on
   custom/localhost targets and return success:False with the actual error
   message. Cloud providers keep the previous lenient behaviour so temporary
   API outages don't block switching.

2. Pre-fill base_url and model from config.yaml in _model_flow_custom
   The interactive custom endpoint setup flow read OPENAI_BASE_URL from env
   but ignored model.base_url and model.default in config.yaml, presenting
   empty prompts even after a successful previous setup.
   Fix: load config.yaml first, fall back to env vars. Show existing values
   as bracket hints, and apply them as defaults when the user presses Enter
   without typing a new value (empty input = keep current).
   Also displays the current model name in the summary header.

3. Config.yaml custom endpoint survives OPENROUTER_API_KEY in .env
   When requested_norm was 'auto' and cfg_provider was 'custom', the
   use_config_base_url flag was not set, so the base_url resolution chain
   fell through to OPENROUTER_BASE_URL and flipped the active provider back
   to OpenRouter on CLI restart.
   Fix: add elif cfg_provider == 'custom' branch inside the 'auto' block so
   that an explicitly saved custom endpoint is always honoured regardless of
   which env vars are present.

8 new tests, 44 total pass.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard area/config Config system, migrations, profiles labels May 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Model Switcher: Configuration Persistence Loss & Missing Pre-fills for Custom Endpoints

2 participants