feat: read context_length from providers.<name>.context_length (Step 0c)#20847
Open
qwjzl wants to merge 1 commit into
Open
feat: read context_length from providers.<name>.context_length (Step 0c)#20847qwjzl wants to merge 1 commit into
qwjzl wants to merge 1 commit into
Conversation
Add Step 0c to get_model_context_length() that reads per-provider context_length overrides from config.yaml. Custom endpoints (e.g. llama-server) rarely expose context_length via /models probing, leading to artificially low fallback values (e.g. 32K). This reads the per-provider override from providers.<name>.context_length independently of the custom_providers list (which requires base_url matching — unavailable for cron jobs whose base_url is resolved later). Also handles the 'custom' provider alias — when the full provider name (e.g. 'custom:qwj-local') is normalized to bare 'custom' during AIAgent init, fall back to prefix-matching providers.custom:* keys.
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add Step 0c to
get_model_context_length()that reads per-providercontext_lengthoverrides fromproviders.<name>.context_lengthin config.yaml.Problem
Custom endpoints (e.g. llama-server) rarely expose
context_lengthvia/modelsprobing, causing Hermes to fall back to artificially low values (e.g. 32K). This blocks cron jobs and any scenario where the model's true context window (e.g. 64K) can't be auto-detected.The existing Step 0b (
custom_providersper-model override) requiresbase_urlmatching, which is unavailable for cron jobs — theirbase_urlis resolved later in the AIAgent init sequence.Solution
Step 0c reads
providers.<name>.context_lengthdirectly from config.yaml, independently of thecustom_providerslist. It also handles thecustomprovider alias — when the full provider name (e.g.custom:qwj-local) gets normalized to barecustomduring AIAgent init, it falls back to prefix-matchingproviders.custom:*keys.Test Plan
get_model_context_length()with provider=custom:qwj-localand config_context_length=None returns 65536 (fromproviders.custom:qwj-local.context_length: 65536)Configuration Example