feat(gateway): opt-in per-platform model via platform_models (#34612)#34620
Open
Bartok9 wants to merge 1 commit into
Open
feat(gateway): opt-in per-platform model via platform_models (#34612)#34620Bartok9 wants to merge 1 commit into
Bartok9 wants to merge 1 commit into
Conversation
…earch#34612) api_server._create_agent always resolved model.default; _resolve_gateway_model() had no platform dimension, so the HTTP API server could not run a cheaper/faster model than CLI/Discord/Telegram. Add an opt-in platform param to _resolve_gateway_model(): when supplied and platform_models.<platform> exists in config.yaml, that model wins over model.default. Every existing call site omits the arg and is byte-for-byte unchanged; only api_server._create_agent opts in. Override accepts a bare string or a {default|model} mapping; provider/credentials still come from the global runtime config. Closes NousResearch#34612
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
platformparameter to_resolve_gateway_model()so a single gateway platform can run a different default model.api_server._create_agent) to opt in viaplatform_models.api_server, letting operators run it on a cheaper/faster model than CLI/Discord/Telegram.Motivation
Closes #34612.
gateway/platforms/api_server.py::APIServerAdapter._create_agentresolved its model with a bare_resolve_gateway_model(), which only ever readsmodel.default/model.model. There was no platform dimension, so every gateway platform that builds a temporary agent shared one model — no supported knob to put, say, the API server on Sonnet while everything else stays on Opus.This implements the issue author's proposed design faithfully:
platform_models.<platform>(additive, optional) wins over the global default only when a caller passesplatform=. The override accepts a bare model string or a{default|model}mapping. Provider/credentials still come from the global runtime config, so the override must name a model compatible with the active provider.Config shape:
Verification
python3 -m pytest tests/test_empty_model_fallback.py— 20 passed (12 existing + 8 new inTestResolveGatewayModelPlatformOverride: opt-in isolation, matching/non-matching platform, bare-string + dict shapes, empty/missing/malformedplatform_models).python3 -m pytest tests/gateway/test_api_server.py tests/gateway/test_api_server_toolset.py— 168 passed.python3 -m pytest tests/gateway/test_{fast_command,compress_command,discord_channel_prompts,session_info,compress_plugin_engine,compress_focus}.py— 31 passed. Confirms the other_resolve_gateway_model()consumers (compress/fast/discord/session_info) are unaffected.platform=and is byte-for-byte unchanged. One existing monkeypatch intests/gateway/test_api_server.pywas widened fromlambda:tolambda *a, **k:to accept the new optional arg.Differential value (related open PRs)
Two older PRs propose per-platform model routing but neither closes #34612:
platforms.<platform>.extra.{model,provider,...}and threads provider/credentials through the session-creation path (_resolve_session_agent_runtime). It adds aplatformparam to_resolve_gateway_model()too, but does not update theapi_server._create_agentcall site, which still calls_resolve_gateway_model()with no args — so the API server gap in API server ignores per-platform model config (no way to run api_server on a different model than the global default) #34612 remains open under that PR.platform_modelsfor IM platforms in_create_agent_for_session(), also not the api_server temporary-agent path.This PR is intentionally minimal and surgical to the issue's scope: it adds the opt-in dimension and wires only the API server. If the maintainers prefer the broader provider-aware design, the cleaner consolidation is to land #11439 and add the one-line api_server opt-in from here. Happy to rebase onto whichever direction is chosen.