fix(config): resolve provider-level context_length on /model switch#37712
Closed
vanhoof wants to merge 1 commit into
Closed
fix(config): resolve provider-level context_length on /model switch#37712vanhoof wants to merge 1 commit into
vanhoof wants to merge 1 commit into
Conversation
Two bugs prevented `providers.<name>.context_length` from being used
when switching models via /model:
1. `get_compatible_custom_providers()` returned `[]` when
`custom_providers` was an empty dict (`{}`) instead of a list.
The early return skipped the `providers:` dict loop entirely, so
user-configured providers were invisible to the context_length
lookup chain.
2. `get_custom_provider_context_length()` only checked the nested
`models.<model>.context_length` path. The common config shape
(`providers.<name>` with `model` + `context_length` as siblings,
no nested `models` dict) was never matched, so the lookup returned
`None` and fell through to the 256K default.
Fix (1) by gracefully handling dict-shaped `custom_providers` the same
way as the `providers:` dict. Fix (2) by falling back to the entry's
top-level `context_length` when the entry's default model matches.
Also persist the resolved context_length back to
`agent._config_context_length` after `switch_model` so the display
path reads the correct value on subsequent /model calls.
Supersedes NousResearch#36199.
Signed-off-by: Chris van Hoof <vanhoof@ouwish.com>
Author
|
Closing -- fix doesn't take effect at runtime. Need to investigate the code path further. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
/modelswitch always showsContext: 256,000 tokensfor custom providers configured viaproviders:in config.yaml, regardless of the actualcontext_lengthvalue set on the provider entry.Observed behavior:
This affects all custom provider setups (vertex-proxy, LiteLLM, or any
providers:entry) using the common config shape:Root Cause
Two bugs in the resolution chain:
get_compatible_custom_providers()early-returns[]on dict-shapedcustom_providers. Whencustom_providers: {}exists in config.yaml (an empty dict, not a list), thenot isinstance(custom_providers, list)guard returns early, skipping theproviders:dict loop below it. All user-configured providers become invisible to the context_length lookup.get_custom_provider_context_length()only checksmodels.<model>.context_length. The common config shape hasmodelandcontext_lengthas siblings at the provider level, with no nestedmodels:dict. The lookup returnsNoneand the whole chain falls through toDEFAULT_FALLBACK_CONTEXT = 256_000.Fix
config.py:
get_compatible_custom_providers(): gracefully handle dict-shapedcustom_providersby normalizing it the same way as theproviders:dict, instead of early-returning.get_custom_provider_context_length(): after the existingmodels.<model>.context_lengthlookup, fall back to the provider-levelcontext_lengthwhen the entry's defaultmodelmatches the requested model.agent_runtime_helpers.py:
new_context_lengthback toagent._config_context_lengthafterswitch_modelcompletes, so the display path in cli.py reads the correct value instead ofNone.Testing
Manual: Tested with
providers:config entries pointing at vertex-proxy (port 8788, Anthropic wire protocol).Before fix:
After fix:
Automated: All existing tests pass:
test_config.py— 87 passedtest_config_validation.py— 21 passed (dict-shaped custom_providers detection still works)test_model_switch_custom_providers.py— 19 passedtest_custom_provider_context_length— 12 passed, 2 skippedSupersedes #36199.
Signed-off-by: Chris van Hoof vanhoof@ouwish.com