Skip to content

fix(config): resolve provider-level context_length on /model switch#37712

Closed
vanhoof wants to merge 1 commit into
NousResearch:mainfrom
vanhoof:fix/config-provider-context-length
Closed

fix(config): resolve provider-level context_length on /model switch#37712
vanhoof wants to merge 1 commit into
NousResearch:mainfrom
vanhoof:fix/config-provider-context-length

Conversation

@vanhoof

@vanhoof vanhoof commented Jun 2, 2026

Copy link
Copy Markdown

Problem

/model switch always shows Context: 256,000 tokens for custom providers configured via providers: in config.yaml, regardless of the actual context_length value set on the provider entry.

Observed behavior:

/model vertex-opus46
  ✓ Model switched: claude-opus-4.6
    Provider: vertex-opus46
    Context: 256,000 tokens    ← wrong, should be 1,000,000

This affects all custom provider setups (vertex-proxy, LiteLLM, or any providers: entry) using the common config shape:

providers:
  vertex-opus46:
    base_url: http://127.0.0.1:8788/anthropic
    model: claude-opus-4.6
    context_length: 1000000
    api_mode: anthropic_messages

Root Cause

Two bugs in the resolution chain:

  1. get_compatible_custom_providers() early-returns [] on dict-shaped custom_providers. When custom_providers: {} exists in config.yaml (an empty dict, not a list), the not isinstance(custom_providers, list) guard returns early, skipping the providers: dict loop below it. All user-configured providers become invisible to the context_length lookup.

  2. get_custom_provider_context_length() only checks models.<model>.context_length. The common config shape has model and context_length as siblings at the provider level, with no nested models: dict. The lookup returns None and the whole chain falls through to DEFAULT_FALLBACK_CONTEXT = 256_000.

Fix

config.py:

  • get_compatible_custom_providers(): gracefully handle dict-shaped custom_providers by normalizing it the same way as the providers: dict, instead of early-returning.
  • get_custom_provider_context_length(): after the existing models.<model>.context_length lookup, fall back to the provider-level context_length when the entry's default model matches the requested model.

agent_runtime_helpers.py:

  • Persist the resolved new_context_length back to agent._config_context_length after switch_model completes, so the display path in cli.py reads the correct value instead of None.

Testing

Manual: Tested with providers: config entries pointing at vertex-proxy (port 8788, Anthropic wire protocol).

Before fix:

/model vertex-opus46   → Context: 256,000 tokens
/model vertex-haiku    → Context: 256,000 tokens

After fix:

/model vertex-opus46   → Context: 1,000,000 tokens
/model vertex-haiku    → Context: 200,000 tokens

Automated: All existing tests pass:

  • test_config.py — 87 passed
  • test_config_validation.py — 21 passed (dict-shaped custom_providers detection still works)
  • test_model_switch_custom_providers.py — 19 passed
  • test_custom_provider_context_length — 12 passed, 2 skipped

Supersedes #36199.

Signed-off-by: Chris van Hoof vanhoof@ouwish.com

Two bugs prevented `providers.<name>.context_length` from being used
when switching models via /model:

1. `get_compatible_custom_providers()` returned `[]` when
   `custom_providers` was an empty dict (`{}`) instead of a list.
   The early return skipped the `providers:` dict loop entirely, so
   user-configured providers were invisible to the context_length
   lookup chain.

2. `get_custom_provider_context_length()` only checked the nested
   `models.<model>.context_length` path. The common config shape
   (`providers.<name>` with `model` + `context_length` as siblings,
   no nested `models` dict) was never matched, so the lookup returned
   `None` and fell through to the 256K default.

Fix (1) by gracefully handling dict-shaped `custom_providers` the same
way as the `providers:` dict. Fix (2) by falling back to the entry's
top-level `context_length` when the entry's default model matches.

Also persist the resolved context_length back to
`agent._config_context_length` after `switch_model` so the display
path reads the correct value on subsequent /model calls.

Supersedes NousResearch#36199.

Signed-off-by: Chris van Hoof <vanhoof@ouwish.com>
@vanhoof

vanhoof commented Jun 2, 2026

Copy link
Copy Markdown
Author

Closing -- fix doesn't take effect at runtime. Need to investigate the code path further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant