Skip to content

fix(context): respect custom provider per-model context_length overrides#24460

Open
andrewhosf wants to merge 3 commits into
NousResearch:mainfrom
andrewhosf:fix/custom-provider-context-override
Open

fix(context): respect custom provider per-model context_length overrides#24460
andrewhosf wants to merge 3 commits into
NousResearch:mainfrom
andrewhosf:fix/custom-provider-context-override

Conversation

@andrewhosf

@andrewhosf andrewhosf commented May 12, 2026

Copy link
Copy Markdown
Contributor

Bug Description

When using a custom provider with a per-model context_length, the global model.context_length (e.g. 200K) always wins — the custom provider's setting is silently ignored. The displayed context length jumps to the global default regardless of what's configured.

Root Cause

Two bugs stacked:

  1. model_metadata.py — resolution order bug: Step 0 of get_model_context_length() returns the global config_context_length before Step 0b ever checks custom_providers[].models[].context_length. The per-model override is unreachable.

  2. cli.py — missing custom_providers in display: Both CLI /model display paths call resolve_display_context_length() without passing custom_providers, so even if the resolution were correct internally, the displayed value would still show 200K.

(The gateway display paths in run.py already pass custom_providers correctly — this was a CLI-only omission.)

Fix

agent/model_metadata.py — In Step 0, before returning the global config_context_length, check for a matching custom provider entry. If the current provider is a custom provider with a models.<model>.context_length override, return that value instead.

cli.py — Both display paths now load custom_providers from config and pass them to resolve_display_context_length().

How to Verify

  1. Add a custom provider with a per-model context_length:
    custom_providers:
      - name: my-inference
        base_url: http://192.168.1.116:8080/v1
        model: my-model.gguf
        models:
          my-model.gguf:
            context_length: 131072
  2. Switch to this provider (/model custom:my-inference)
  3. CLI now displays 131,072 instead of 200,000

Risk Assessment

Low. Both changes are additive — they only enable a path that was previously blocked (custom_providers lookup inside existing guard clauses). For non-custom-provider users, behavior is identical: no custom_providers means no override, falls through to the global default as before.

Test Plan

  • Manual: /model display now shows correct context for custom providers
  • Python unit test: resolve_display_context_length() returns 131072 with custom_providers, 200000 without
  • Existing tests pass (no behavioral change for non-custom providers)

…lobal default

Step 0 of get_model_context_length() returns the global
model.context_length (e.g. 200K) before Step 0b ever gets
to check custom_providers per-model overrides. This means a
custom provider with a smaller context_length (e.g. 131K for
a local GGUF) always shows 200K — the per-model override is
silently ignored.

Fix: in Step 0, check for a matching custom provider entry
before returning the global default. If the current provider
is a custom provider with a models.<model>.context_length
override, return that value instead.

Also fixes a latent dead-key bug: the top-level context_length
field in a custom_providers entry is listed in
_VALID_CUSTOM_PROVIDER_FIELDS but get_custom_provider_context_length()
only reads from models[model_name].context_length. This change
doesn't wire the top-level field (that's a separate config-schema
fix), but the model_metadata resolution change means that
models[model_name].context_length — the format users should use —
now actually wins over the global default.
The two CLI /model display paths call resolve_display_context_length()
without passing custom_providers, so even though the per-model
context_length override is correctly resolved internally (via the
model_metadata.py fix), the displayed value still shows the global
model.context_length default.

Fix: load custom_providers from config and pass them to
resolve_display_context_length() in both display paths.
@alt-glitch alt-glitch added type/bug Something isn't working comp/cli CLI entry point, hermes_cli/, setup wizard comp/agent Core agent loop, run_agent.py, prompt builder area/config Config system, migrations, profiles P2 Medium — degraded but workaround exists labels May 12, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related: #11438 (same root cause — custom provider per-model context_length ignored in model_metadata.py). Also related to closed #13052, #7998, #15779 which addressed similar resolution-order bugs. This PR appears to be a more complete fix covering both model_metadata.py and cli.py display paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent loop, run_agent.py, prompt builder comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants