Describe the bug
The max_tokens configuration set in custom_providers is not being properly passed to the AIAgent instance. Instead, the gateway always uses model.max_tokens from the global config, or falls back to provider-specific defaults (like 32k for Kimi).
To Reproduce
- Set
max_tokens in a custom_provider entry, e.g.:
custom_providers:
- name: ark
max_tokens: 131072
# ...
- Remove or don't set
model.max_tokens globally
- Start a conversation through the gateway
- Observe that max_tokens uses the provider default instead of 131072
Expected behavior
The max_tokens value from custom_providers should be used, with fallback to model.max_tokens if provider-specific config is not present.
Fix
This is fixed by PR #19991 which:
- Adds
max_tokens to _KNOWN_KEYS in config normalization
- Passes
max_tokens through runtime provider resolution
- Updates gateway to read max_tokens from runtime config first, then fallback to model config
Relates to #19991
Describe the bug
The
max_tokensconfiguration set incustom_providersis not being properly passed to theAIAgentinstance. Instead, the gateway always usesmodel.max_tokensfrom the global config, or falls back to provider-specific defaults (like 32k for Kimi).To Reproduce
max_tokensin a custom_provider entry, e.g.:model.max_tokensgloballyExpected behavior
The
max_tokensvalue from custom_providers should be used, with fallback tomodel.max_tokensif provider-specific config is not present.Fix
This is fixed by PR #19991 which:
max_tokensto_KNOWN_KEYSin config normalizationmax_tokensthrough runtime provider resolutionRelates to #19991