Skip to content

feat(config): add per-model max_tokens overlay#29705

Open
digitalbase wants to merge 1 commit into
NousResearch:mainfrom
digitalbase:feat/per-model-max-tokens-overlay
Open

feat(config): add per-model max_tokens overlay#29705
digitalbase wants to merge 1 commit into
NousResearch:mainfrom
digitalbase:feat/per-model-max-tokens-overlay

Conversation

@digitalbase

@digitalbase digitalbase commented May 21, 2026

Copy link
Copy Markdown

If you use providers like openrouter, bedrock the single max_token setting isn't enough. It needs to be dynamic based on the model.

Summary

Adds a model.models.<id>.max_tokens overlay so a single profile can switch between models with different output-token ceilings without mutating the flat model.max_tokens fallback.

model:
  default: anthropic/claude-opus-4.6
  max_tokens: 8192              # fallback for unmatched models
  models:
    anthropic/claude-opus-4.6:
      max_tokens: 32768
    gpt-5.5:
      max_tokens: 65536

Resolution order is:

  1. Explicit constructor/programmatic max_tokens
  2. model.models.<active_model>.max_tokens
  3. Flat model.max_tokens
  4. Provider/model default

Why this is not a duplicate

Related PRs solve adjacent scopes, but not this specific config shape:

Changes

  • agent/agent_init.py: resolve model.models.<active>.max_tokens before flat model.max_tokens, preserving constructor precedence and positive-int validation.
  • gateway/run.py: include model.models in gateway agent cache-busting keys so edits to overlays rebuild cached agents.
  • cli-config.yaml.example: document the per-model output-token overlay.
  • Tests cover override wins, unmatched fallback, invalid per-model fallback, constructor precedence, and gateway cache invalidation.

@alt-glitch alt-glitch added type/feature New feature or request comp/agent Core agent loop, run_agent.py, prompt builder area/config Config system, migrations, profiles P3 Low — cosmetic, nice to have labels May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent loop, run_agent.py, prompt builder P3 Low — cosmetic, nice to have type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants