fix: resolve context_length from custom_providers on model switch#13052
Closed
TroyMitchell911 wants to merge 1 commit into
Closed
fix: resolve context_length from custom_providers on model switch#13052TroyMitchell911 wants to merge 1 commit into
TroyMitchell911 wants to merge 1 commit into
Conversation
Three related bugs caused custom_providers per-model context_length overrides to be ignored, particularly when switching models at runtime: 1. __init__: self._config_context_length was assigned *before* the custom_providers lookup, so per-model overrides were never stored on the agent instance. Moved the assignment after the lookup. 2. switch_model: re-resolve config_context_length from config.yaml for the new (model, base_url) pair instead of reusing the stale value from __init__. Checks both top-level model.context_length and custom_providers per-model overrides. 3. get_model_context_length: when a custom endpoint's /models response omits context_length, fall through to models.dev / OpenRouter / hardcoded defaults instead of returning 128K immediately. Proxy endpoints (LiteLLM, NewAPI, etc.) often omit context_length but serve well-known models whose metadata is available downstream. Signed-off-by: Troy Mitchell <i@troy-y.org>
|
Good work on this fix. Key improvements:
The changes address:
This ensures models served through proxy endpoints correctly report their context lengths. Ready to merge. |
ca849be to
bc0ce12
Compare
Contributor
Author
The PR is ready to be merged. :) |
Collaborator
This was referenced Apr 24, 2026
Closed
Contributor
|
Thanks for the well-structured fix and the thorough test coverage, @TroyMitchell911! This is an automated hermes-sweeper review. After inspecting current Evidence:
The fix on main is broader in scope than this PR (five call sites vs three), so closing this as superseded. The tests added here may still be worth cherry-picking if they cover paths not already covered by the new |
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three related bugs caused
custom_providersper-modelcontext_lengthoverrides to be ignored, particularly when switching models at runtime via/model. This resulted in models likegemini-3.1-pro-preview(1M context) being incorrectly reported as 128K when served through proxy endpoints (LiteLLM, NewAPI, etc.).Root Cause
__init__assignment order (run_agent.py):self._config_context_lengthwas assigned before thecustom_providerslookup, so per-model overrides fromcustom_providerswere never stored on the agent instance.switch_modelstale value (run_agent.py): When switching models, the old model's_config_context_lengthwas reused instead of re-resolving fromcustom_providersfor the new(model, base_url)pair.Early return on unknown endpoints (
model_metadata.py): When a custom endpoint's/modelsresponse omittedcontext_length,get_model_context_length()returned 128K immediately instead of falling through to models.dev / OpenRouter / hardcoded defaults.Changes
run_agent.py: Moveself._config_context_lengthassignment after thecustom_providerslookup in__init__.run_agent.py: Inswitch_model(), re-resolveconfig_context_lengthfrom config.yaml (both top-levelmodel.context_lengthandcustom_providersper-model overrides) for the new model.agent/model_metadata.py: Remove the earlyreturn DEFAULT_FALLBACK_CONTEXTfor unknown custom endpoints when/modelslackscontext_length— fall through to downstream registries instead.test_switch_model_context.pyand added new test files for both fixes.Test Plan
gemini-3.1-pro-previewvia a proxy endpoint now correctly reports 1M context.