Problem

When configuring a custom endpoint via hermes setup or /model, the setup wizard collects URL, API key, model name, and context length — but never asks which API protocol to use. The api_mode is left to auto-detection from the URL.

This breaks when:

• A proxy/gateway serves multiple protocols on the same URL (e.g. LiteLLM proxy that can do both OpenAI Chat Completions and Anthropic Messages)
• The URL does not contain enough hints for auto-detection (e.g. a plain https://my-llm.internal/v1 that actually speaks Anthropic Messages)
• Users want to force codex_responses for an endpoint that would otherwise default to chat_completions

Related: #5718 (hardcoded api_mode for openai-codex provider), #3388 (api_mode drift causing routing failures)

Proposed Solution

Add an API protocol prompt to the custom endpoint setup flow. Default behavior: auto-detect (just press Enter to accept). Users who need to override type ? to see choices:

API protocol [chat_completions] (Enter to accept, ? to choose):

If ? is typed:

  1. OpenAI Chat Completions (most common)
  2. Anthropic Messages (Claude-native API)
  3. OpenAI Responses (Codex/GPT-5 reasoning)
  4. Auto-detect from URL (default)
Choice [1-4]:

The selected api_mode is:

• Saved to model.api_mode in config.yaml
• Persisted in custom_providers entries so it is restored on provider switch
• Applied in _model_flow_named_custom when activating a saved custom provider

Scope

• _model_flow_custom() — add api_mode prompt after context length
• _model_flow_named_custom() — read and apply saved api_mode
• _save_custom_provider() — persist api_mode in custom_providers entries
• New helper _prompt_api_mode_selection() — lightweight prompt with Enter-to-accept default

No changes to auto-detection logic itself — this only adds an explicit override path.