fix(cli): correct model precedence — argv > settings > auth env vars#3645
Conversation
| | undefined; | ||
| const candidates = [ | ||
| argv.model, | ||
| settings.model?.name, |
There was a problem hiding this comment.
[Critical] When settings.model.name is set but does not match any configured provider, this loop falls through to OPENAI_MODEL / QWEN_MODEL and passes that env-matched modelProvider into resolveModelConfig. Core resolution still prioritizes modelProvider over settings, so the env provider overrides the configured settings model entirely. A user with settings.model.name = "custom-model" and a leftover OPENAI_MODEL matching a provider can unexpectedly run the env/provider model instead of the settings-selected model.
Please only use env-based provider lookup when neither argv.model nor settings.model.name is set, or adjust the resolver call so an env-matched provider can supply metadata without overriding the higher-precedence model. Also add an unmocked/integration test for settings.model.name unmatched + OPENAI_MODEL matched.
— gpt-5.5 via Qwen Code /review
There was a problem hiding this comment.
Fixed in latest push — the old candidate-loop is gone. Model is now resolved via strict precedence (argv > settings > env), and modelProvider is looked up separately based on resolvedModel. No fallthrough to OPENAI_MODEL anymore.
| | undefined; | ||
| const candidates = [ | ||
| argv.model, | ||
| settings.model?.name, |
There was a problem hiding this comment.
[Critical] When settings.model.name is set but does not match any configured provider, this loop falls through to OPENAI_MODEL / QWEN_MODEL and passes that env-matched modelProvider into resolveModelConfig. Core resolution still prioritizes modelProvider over settings, so the env provider overrides the configured settings model entirely. A user with settings.model.name = "custom-model" and a leftover OPENAI_MODEL matching a provider can unexpectedly run the env/provider model instead of the settings-selected model.
Please only use env-based provider lookup when neither argv.model nor settings.model.name is set, or adjust the resolver call so an env-matched provider can supply metadata without overriding the higher-precedence model. Also add an unmocked/integration test for settings.model.name unmatched + OPENAI_MODEL matched.
— gpt-5.5 via Qwen Code /review
There was a problem hiding this comment.
Same as above — the old loop is gone. Model resolved via strict precedence now.
|
Thanks for the review @wenshao. You're absolutely right — the previous implementation had a fundamental flaw. The bug you identified: The candidate loop would fall through to when didn't match any provider, then pass that env-matched into , which overrides the user's model choice. Fix applied in d947114:
All 36 unit tests + 4524 full test suite pass. |
| input?.cli?.model || | ||
| input?.env?.['OPENAI_MODEL'] || | ||
| input?.settings?.model || | ||
| ''; |
There was a problem hiding this comment.
[Critical] This mock returns sources.model as a string, but resolveModelConfig is typed to return ModelConfigResolutionResult, whose sources values must be ConfigSource objects. As written, npm run typecheck fails on this mock implementation, so the PR cannot pass the TypeScript build gate.
| ''; | |
| sources: { | |
| model: | |
| model === 'custom-model' | |
| ? { kind: 'settings', path: 'model.name' } | |
| : { kind: 'env', envKey: 'OPENAI_MODEL' }, | |
| }, |
— gpt-5.5 via Qwen Code /review
There was a problem hiding this comment.
Fixed in 1353d21 - updated the mock to return proper ConfigSource objects with kind, path, and envKey fields matching the reviewer's suggestion. Also added [Regression] tests (5 tests) guarding against the original model precedence bug.
| }); | ||
|
|
||
| it('[Regression] Non-OpenAI auth ignores OPENAI_MODEL', () => { | ||
| const argv = {}; |
There was a problem hiding this comment.
[Suggestion] This regression test does not actually assert that OPENAI_MODEL is ignored. The resolveModelConfig mock returns claude-3 unconditionally, and the test only checks result.model, so it would still pass if resolveCliGenerationConfig incorrectly passed OPENAI_MODEL through to the resolver for Anthropic auth.
| const argv = {}; | |
| // For non-OpenAI auth, OPENAI_MODEL should not be in the model resolution | |
| expect(result.model).toBe('claude-3'); | |
| const callArgs = vi.mocked(resolveModelConfig).mock.calls[0][0]; | |
| expect(callArgs.env?.['OPENAI_MODEL']).toBeUndefined(); |
— gpt-5.5 via Qwen Code /review
wenshao
left a comment
There was a problem hiding this comment.
No issues found. LGTM! ✅ — gpt-5.5 via Qwen Code /review
| } else if (settings.model?.name) { | ||
| resolvedModel = settings.model.name; | ||
| } else if (authType === AuthType.USE_OPENAI) { | ||
| resolvedModel = env['OPENAI_MODEL'] || env['QWEN_MODEL']; |
There was a problem hiding this comment.
[Critical] This only derives resolvedModel from env for OpenAI auth. For USE_ANTHROPIC, USE_GEMINI, or USE_VERTEX_AI, core still supports auth-specific model env vars (ANTHROPIC_MODEL, GEMINI_MODEL, GOOGLE_MODEL), but this CLI-side provider lookup now skips them entirely when settings.model.name is unset. That means a matching modelProviders[authType] entry will not be passed to resolveModelConfig, so provider metadata such as envKey, baseUrl, and generationConfig can be lost for non-OpenAI env-selected models.
Please resolve env fallback by selected auth type instead of hard-coding OpenAI here, reusing the same env model mapping as core resolution while still preserving the new argv/settings precedence.
— gpt-5.5 via Qwen Code /review
There was a problem hiding this comment.
Fixed — AUTH_ENV_MODEL_VARS mapping now covers all auth types (USE_OPENAI, USE_GEMINI, USE_VERTEX_AI, USE_ANTHROPIC). Env resolution iterates only the env vars for the current authType.
|
|
||
| // Filter env to prevent OPENAI_MODEL/QWEN_MODEL from overriding higher-priority sources. | ||
| // Only pass these env vars when we actually resolved the model from them. | ||
| const filteredEnv = { ...env }; |
There was a problem hiding this comment.
[Suggestion] The filtering decision is based on value equality instead of the source that actually selected the model. If settings.model.name happens to equal OPENAI_MODEL, this keeps OPENAI_MODEL in filteredEnv; since core resolves model as modelProvider > cli > env > settings, an unmatched settings-selected model can then be attributed to env rather than settings. That weakens the precedence fix for source-aware downstream behavior.
| const filteredEnv = { ...env }; | |
| const modelResolvedFromEnv = | |
| authType === AuthType.USE_OPENAI && | |
| !argv.model && | |
| !settings.model?.name && | |
| (resolvedModel === env['OPENAI_MODEL'] || | |
| resolvedModel === env['QWEN_MODEL']); | |
| const filteredEnv = { ...env }; | |
| if (!modelResolvedFromEnv) { | |
| delete filteredEnv['OPENAI_MODEL']; | |
| delete filteredEnv['QWEN_MODEL']; | |
| } |
— gpt-5.5 via Qwen Code /review
There was a problem hiding this comment.
Fixed — filtering now uses sourceEnvVar (source-based), not value equality. sourceEnvVar is only set when model was resolved from an env var. If model came from argv or settings, sourceEnvVar is undefined and ALL model env vars are stripped.
…ring - Add AUTH_MODEL_ENV_VARS map so all auth types (Anthropic, Gemini, Vertex AI) get proper env var fallback, not just OpenAI - Track modelResolvedFromEnv flag instead of fragile value equality, fixing the case where settings.model.name == OPENAI_MODEL would incorrectly keep OPENAI_MODEL in filteredEnv - Filter ALL model env vars when model was NOT resolved from env, not just OPENAI_MODEL/QWEN_MODEL - Add tests: Anthropic/Gemini/Vertex AI env fallback, value-equality fix, env filtering when model from argv/settings Addresses review feedback on PR QwenLM#3645. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ection Fixes model precedence regression from QwenLM#3567 (reverted in QwenLM#3633): - argv.model > settings.model.name > OPENAI_MODEL > QWEN_MODEL - settings.model.name wins over OPENAI_MODEL when it matches a provider - OPENAI_MODEL only used as fallback when no explicit model is configured - Added tests proving both paths: settings wins, and OPENAI_MODEL fallback works Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…v override The previous implementation iterated through candidates [argv.model, settings.model.name, OPENAI_MODEL, QWEN_MODEL] and picked the first matching provider. This caused settings.model.name to be overridden by OPENAI_MODEL when the former didn't match any provider. Fix by: 1. First resolve target model using strict precedence 2. Only look up provider for that specific model 3. Filter OPENAI_MODEL/QWEN_MODEL from env when model was resolved from settings or argv, preventing the core resolver from picking up these env vars Also fixes Edge Case 5 test (was testing buggy behavior) and adds integration test verifying settings.model.name wins over OPENAI_MODEL.
The mockImplementation used ModelConfigSourcesInput type which wasn't properly resolved. Remove the type annotation to fix TypeScript and ESLint errors. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The mockImplementation was causing TS2345 in CI because the return type didn't match ModelConfigResolutionResult. Fixed by using optional chaining and removing type annotations that caused CI build failures. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The mock of resolveModelConfig returned sources as { model: string }
instead of { model: ConfigSource }, causing a type error in CI only
(where strictBuild is enabled). Use proper ConfigSource objects.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Use proper ConfigSource objects with path/envKey details as suggested in the review, matching what resolveModelConfig actually returns. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds [Regression] tests that guard against the original bug where OPENAI_MODEL incorrectly overrode settings.model.name. Tests cover: - settings.model.name precedence over OPENAI_MODEL - OPENAI_MODEL used when settings.model.name not set - argv.model overriding both settings and env - QWEN_MODEL as fallback when OPENAI_MODEL not set - Non-OpenAI auth ignoring OPENAI_MODEL Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add missing assertion in Non-OpenAI auth test to verify OPENAI_MODEL is filtered from env passed to resolveModelConfig - Clean up modelProvider lookup comment to clarify the old bug is fixed Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
f43911f to
3b55822
Compare
|
@wenshao Addressed all review feedback in latest commits:
Ready for another look! 🚀 |
wenshao
left a comment
There was a problem hiding this comment.
No issues found. LGTM! ✅ — gpt-5.5 via Qwen Code /review
wenshao
left a comment
There was a problem hiding this comment.
Direction is correct, but blocking issues remain
The target semantics (argv > settings.model.name > OPENAI_MODEL > QWEN_MODEL) and the architectural approach — separate model selection from provider lookup, then strip env so the core resolver can't override settings — are right.
However, this PR brings back the same flaky-test failure mode that #3633 reverted from #3567, plus some leftover cleanup.
🚨 Blocking: should use process.env when env is not provided (line 568) fails when shell has OPENAI_MODEL set
Reproduction:
OPENAI_MODEL=ci-leak-model OPENAI_API_KEY=fake OPENAI_BASE_URL=http://x \
npx vitest run src/utils/modelConfigUtils.test.ts
# Tests 1 failed | 40 passedFull CLI suite shows the same single failure (4690 passed | 1 failed).
Walkthrough:
makeMockSettings()defaults tomodel: { name: 'default-model' }.- New code resolves
resolvedModel = 'default-model'. 'default-model' !== process.env.OPENAI_MODEL⇒ strip path triggers.filteredEnv(a new object missingOPENAI_MODEL/QWEN_MODEL) is passed toresolveModelConfig.- The assertion
env: process.env(line 591) fails becausefilteredEnvis no longer structurally equal toprocess.env.
CI is green only because CI doesn't export OPENAI_MODEL. But the user cohort this PR is targeting is exactly the developers with leftover OPENAI_MODEL in their shell — they will hit this on npm test locally. Same failure signature documented in #3633 (run 24949085749).
Cleanest fix: decouple the whole suite from the developer's shell by sanitizing in beforeEach:
beforeEach(() => {
vi.resetModules();
process.env = { ...originalEnv };
delete process.env['OPENAI_MODEL'];
delete process.env['QWEN_MODEL'];
mockWriteStderrLine.mockClear();
});The model: undefined workaround you applied to the sibling test at line 535 is a symptom of the same underlying fragility — see inline comment.
🧹 Stray // trigger rebuild at line 1201
Leftover CI-poke comment at the end of the test file. Please remove before merge.
⚠️ PR description contradicts the actual behavior
The description says:
OPENAI_MODEL is only used when:
- settings.model.name does not match any provider
But Edge Case 5 and the implementation do the opposite — settings.model.name wins over OPENAI_MODEL even when unmatched. That's the right semantics (an explicit /model choice shouldn't be silently overridden), but please update the description so reviewers and future readers aren't confused.
📌 Two prior review threads still open
- Critical on
modelConfigUtils.ts:107— env fallback hardcoded toUSE_OPENAI.ANTHROPIC_MODEL/GEMINI_MODEL/GOOGLE_MODELare real env model selectors inAUTH_ENV_MAPPINGSbut this branch never feeds them into provider lookup, so non-OpenAI auths loseenvKey/baseUrl/generationConfigmetadata for env-selected models. Either reuseAUTH_ENV_MAPPINGS[authType].modelhere, or explicitly scope this PR to OpenAI in the description and file a follow-up. - Suggestion on
modelConfigUtils.ts:124— strip uses value equality. Ifsettings.model.name === OPENAI_MODELby coincidence, env stays infilteredEnvand source attribution can flip fromsettingstoenv. Doesn't change the resolved model id, but affects logging/UI attribution.
Once the failing test and the stray comment are fixed, the rest can land as follow-up.
| const argv = {}; | ||
| const settings = makeMockSettings(); | ||
| const settings = makeMockSettings({ | ||
| model: undefined as unknown as Settings['model'], |
There was a problem hiding this comment.
[Critical — root cause exposed here] This model: undefined is a per-test workaround for the new strip-path side effect, and it leaves the sibling test should use process.env when env is not provided (line 568) silently exposed to the same problem.
That sibling test still uses the default makeMockSettings() (which sets model: { name: 'default-model' }), so the strip path triggers and a filtered env is passed to resolveModelConfig — breaking the env: process.env assertion. Reproduction:
OPENAI_MODEL=ci-leak-model OPENAI_API_KEY=fake OPENAI_BASE_URL=http://x \
npx vitest run src/utils/modelConfigUtils.test.ts
# Tests 1 failed | 40 passedThis is the same failure signature that #3633 rolled back from #3567 (run 24949085749). CI passes only because CI doesn't export OPENAI_MODEL — but the user cohort this PR is targeting is exactly developers with leftover OPENAI_MODEL in their shell.
Suggested fix: instead of patching individual tests with model: undefined, sanitize once in beforeEach so the whole suite is decoupled from the developer's shell:
beforeEach(() => {
vi.resetModules();
process.env = { ...originalEnv };
delete process.env['OPENAI_MODEL'];
delete process.env['QWEN_MODEL'];
mockWriteStderrLine.mockClear();
});Then the model: undefined here can be reverted, and future tests don't need the workaround either.
| }); | ||
| }); | ||
| }); | ||
| // trigger rebuild |
There was a problem hiding this comment.
[Cleanup] Stray // trigger rebuild comment leftover from CI debugging. Please remove before merge.
superseded by newer changes-requested review
Changes applied to address review feedback:✅ Blocking issue fixed:
|
- Sanitize OPENAI_MODEL/QWEN_MODEL in beforeEach to prevent flaky tests - Remove stray "// trigger rebuild" comment - Add AUTH_ENV_MODEL_VARS mapping for all auth types (not just OpenAI) - Fix filteredEnv logic to strip ALL model env vars when model not from env - Use sourceEnvVar tracking to only keep the env var that was actually used Fixes the blocking test failure when OPENAI_MODEL is set in shell env. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
| if (argv.model) { | ||
| resolvedModel = argv.model; | ||
| } else if (settings.model?.name) { | ||
| resolvedModel = settings.model.name; |
There was a problem hiding this comment.
[Critical] The new precedence logic applies settings.model.name ahead of the auth-specific model environment variable for every auth type, not just OpenAI. When selectedAuthType is inferred from env, for example GEMINI_API_KEY + GEMINI_MODEL, a stale persisted settings.model.name now wins over GEMINI_MODEL.
This can send a settings model from one provider to a newly selected provider when users switch auth via environment variables. Please restrict the new settings.model.name > OPENAI_MODEL > QWEN_MODEL precedence override to AuthType.USE_OPENAI, or otherwise preserve the existing env-over-settings behavior for non-OpenAI auth types.
— gpt-5.5 via Qwen Code /review
There was a problem hiding this comment.
Intentional — PR title and description state precedence applies to all auth types. AUTH_ENV_MODEL_VARS resolves correct env vars per authType. Non-OpenAI auth now correctly ignores OPENAI_MODEL (see Edge Case 4 test).
|
Thank you for your time, energy, and patience in reviewing this PR! I really appreciate you thoroughly examining the model precedence fixes. After you pointed out the issue with how env vars were being handled, I went back and properly implemented the correct precedence:
The test coverage is much better now too - added comprehensive tests covering all the edge cases. Thanks again for the careful review - your feedback made this a much stronger implementation! 🙏 |
|
All issues from the review have been addressed in this commit: Changes made:
The semantics are correct: |
| vi.resetModules(); | ||
| process.env = { ...originalEnv }; | ||
| delete process.env['OPENAI_MODEL']; | ||
| delete process.env['QWEN_MODEL']; |
There was a problem hiding this comment.
[Critical] This cleanup only removes OPENAI_MODEL and QWEN_MODEL, but the production code now filters all auth-specific model env vars (ANTHROPIC_MODEL, GEMINI_MODEL, and GOOGLE_MODEL too). If any of those are present in a developer or CI environment, should use process.env when env is not provided fails because resolveCliGenerationConfig passes a filtered env while the test still expects the full process.env.
| delete process.env['QWEN_MODEL']; | |
| delete process.env['OPENAI_MODEL']; | |
| delete process.env['QWEN_MODEL']; | |
| delete process.env['ANTHROPIC_MODEL']; | |
| delete process.env['GEMINI_MODEL']; | |
| delete process.env['GOOGLE_MODEL']; |
— gpt-5.5 via Qwen Code /review
wenshao
left a comment
There was a problem hiding this comment.
No issues found. LGTM! ✅ — gpt-5.5 via Qwen Code /review
…wenLM#3645) * fix(cli): correct OPENAI_MODEL precedence without breaking /model selection Fixes model precedence regression from QwenLM#3567 (reverted in QwenLM#3633): - argv.model > settings.model.name > OPENAI_MODEL > QWEN_MODEL - settings.model.name wins over OPENAI_MODEL when it matches a provider - OPENAI_MODEL only used as fallback when no explicit model is configured - Added tests proving both paths: settings wins, and OPENAI_MODEL fallback works Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(cli): separate model selection from provider lookup to prevent env override The previous implementation iterated through candidates [argv.model, settings.model.name, OPENAI_MODEL, QWEN_MODEL] and picked the first matching provider. This caused settings.model.name to be overridden by OPENAI_MODEL when the former didn't match any provider. Fix by: 1. First resolve target model using strict precedence 2. Only look up provider for that specific model 3. Filter OPENAI_MODEL/QWEN_MODEL from env when model was resolved from settings or argv, preventing the core resolver from picking up these env vars Also fixes Edge Case 5 test (was testing buggy behavior) and adds integration test verifying settings.model.name wins over OPENAI_MODEL. * fix(types): remove unused type annotation from mock The mockImplementation used ModelConfigSourcesInput type which wasn't properly resolved. Remove the type annotation to fix TypeScript and ESLint errors. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * trigger CI * chore: trigger clean CI build * fix(test): make mock compatible with CI type checking The mockImplementation was causing TS2345 in CI because the return type didn't match ModelConfigResolutionResult. Fixed by using optional chaining and removing type annotations that caused CI build failures. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(test): make mock compatible with CI type checking The mock of resolveModelConfig returned sources as { model: string } instead of { model: ConfigSource }, causing a type error in CI only (where strictBuild is enabled). Use proper ConfigSource objects. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(test): update mock to match reviewer suggestion Use proper ConfigSource objects with path/envKey details as suggested in the review, matching what resolveModelConfig actually returns. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(cli): add regression tests for model precedence Adds [Regression] tests that guard against the original bug where OPENAI_MODEL incorrectly overrode settings.model.name. Tests cover: - settings.model.name precedence over OPENAI_MODEL - OPENAI_MODEL used when settings.model.name not set - argv.model overriding both settings and env - QWEN_MODEL as fallback when OPENAI_MODEL not set - Non-OpenAI auth ignoring OPENAI_MODEL Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(cli): address reviewer feedback on model precedence PR - Add missing assertion in Non-OpenAI auth test to verify OPENAI_MODEL is filtered from env passed to resolveModelConfig - Clean up modelProvider lookup comment to clarify the old bug is fixed Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(cli): address review feedback on model precedence PR - Sanitize OPENAI_MODEL/QWEN_MODEL in beforeEach to prevent flaky tests - Remove stray "// trigger rebuild" comment - Add AUTH_ENV_MODEL_VARS mapping for all auth types (not just OpenAI) - Fix filteredEnv logic to strip ALL model env vars when model not from env - Use sourceEnvVar tracking to only keep the env var that was actually used Fixes the blocking test failure when OPENAI_MODEL is set in shell env. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(cli): fix test assertion for filtered env, add source-based filtering comments --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Mistral Vibe <vibe@mistral.ai>
fix(cli): correct model precedence — argv > settings > auth env vars
Summary
Fixes model resolution precedence in
resolveCliGenerationConfig:Precedence (all auth types)
argv.model>settings.model.name> auth-specific env model varsOPENAI_MODEL,QWEN_MODEL,ANTHROPIC_MODEL, etc.) are ONLY used when neitherargv.modelnorsettings.model.nameis setKey fixes
AUTH_ENV_MODEL_VARSmapping (mirrors core'sAUTH_ENV_MAPPINGS) for all auth typesfilteredEnvproperly strips auth-specific model env vars when model was NOT resolved from envbeforeEachsanitizesOPENAI_MODEL/QWEN_MODELfrom shell env, preventing flaky testsValidation
npx vitest run packages/cli/src/utils/modelConfigUtils.test.ts✅ (41/41)npm run lint✅npm run typecheck✅Related
Related: #1045 (CLOSEs — timeout behavior, env var precedence)