fix(context): respect custom provider per-model context_length overrides#24460
Open
andrewhosf wants to merge 3 commits into
Open
fix(context): respect custom provider per-model context_length overrides#24460andrewhosf wants to merge 3 commits into
andrewhosf wants to merge 3 commits into
Conversation
…lobal default Step 0 of get_model_context_length() returns the global model.context_length (e.g. 200K) before Step 0b ever gets to check custom_providers per-model overrides. This means a custom provider with a smaller context_length (e.g. 131K for a local GGUF) always shows 200K — the per-model override is silently ignored. Fix: in Step 0, check for a matching custom provider entry before returning the global default. If the current provider is a custom provider with a models.<model>.context_length override, return that value instead. Also fixes a latent dead-key bug: the top-level context_length field in a custom_providers entry is listed in _VALID_CUSTOM_PROVIDER_FIELDS but get_custom_provider_context_length() only reads from models[model_name].context_length. This change doesn't wire the top-level field (that's a separate config-schema fix), but the model_metadata resolution change means that models[model_name].context_length — the format users should use — now actually wins over the global default.
The two CLI /model display paths call resolve_display_context_length() without passing custom_providers, so even though the per-model context_length override is correctly resolved internally (via the model_metadata.py fix), the displayed value still shows the global model.context_length default. Fix: load custom_providers from config and pass them to resolve_display_context_length() in both display paths.
Collaborator
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug Description
When using a custom provider with a per-model
context_length, the globalmodel.context_length(e.g. 200K) always wins — the custom provider's setting is silently ignored. The displayed context length jumps to the global default regardless of what's configured.Root Cause
Two bugs stacked:
model_metadata.py— resolution order bug: Step 0 ofget_model_context_length()returns the globalconfig_context_lengthbefore Step 0b ever checkscustom_providers[].models[].context_length. The per-model override is unreachable.cli.py— missing custom_providers in display: Both CLI/modeldisplay paths callresolve_display_context_length()without passingcustom_providers, so even if the resolution were correct internally, the displayed value would still show 200K.(The gateway display paths in
run.pyalready passcustom_providerscorrectly — this was a CLI-only omission.)Fix
agent/model_metadata.py— In Step 0, before returning the globalconfig_context_length, check for a matching custom provider entry. If the current provider is a custom provider with amodels.<model>.context_lengthoverride, return that value instead.cli.py— Both display paths now loadcustom_providersfrom config and pass them toresolve_display_context_length().How to Verify
context_length:/model custom:my-inference)131,072instead of200,000Risk Assessment
Low. Both changes are additive — they only enable a path that was previously blocked (custom_providers lookup inside existing guard clauses). For non-custom-provider users, behavior is identical: no custom_providers means no override, falls through to the global default as before.
Test Plan
/modeldisplay now shows correct context for custom providersresolve_display_context_length()returns 131072 with custom_providers, 200000 without