Skip to content

fix(tools): normalize model reference before registry lookup#66165

Closed
yqli2420 wants to merge 2 commits into
openclaw:mainfrom
yqli2420:fix/image-tool-model-normalization
Closed

fix(tools): normalize model reference before registry lookup#66165
yqli2420 wants to merge 2 commits into
openclaw:mainfrom
yqli2420:fix/image-tool-model-normalization

Conversation

@yqli2420

Copy link
Copy Markdown
Contributor

Summary

Describe the problem and fix in 2–5 bullets:

  • Problem: The image and pdf tools failed with "Unknown model" errors for configured Ollama models because they performed registry lookups using raw, non-normalized model references.
  • Why it matters: This prevented users from using valid, configured vision-capable models via the built-in tools, blocking a core workflow for image analysis.
  • What changed: Modified resolveModelFromRegistry to normalize provider/model IDs using normalizeModelRef before lookup. Refactored resolveModelRuntimeApiKey to accept normalized references and simplify API key retrieval.
  • What did NOT change: The underlying model registry implementation, configuration loading, or the actual model inference logic.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

User-visible / Behavior Changes

The image tool now successfully processes requests using configured Ollama models (e.g., ollama/qwen3.5:397b-cloud) instead of returning an "Unknown model" error.

Security Impact (required)

  • New permissions/capabilities? (No)
  • Secrets/tokens handling changed? (No)
  • New/changed network calls? (No)
  • Command/tool execution surface changed? (No)
  • Data access scope changed? (No)
  • If any Yes, explain risk + mitigation:

Repro + Verification

Environment

  • OS: Linux (Ubuntu)
  • Runtime/container: Node v25.9.0
  • Model/provider: ollama/qwen3.5:397b-cloud
  • Integration/channel (if any): CLI
  • Relevant config (redacted): agents.defaults.imageModel.primary set to ollama/qwen3.5:397b-cloud

Steps

  1. Configure agents.defaults.imageModel.primary to an Ollama model with image support (e.g., ollama/qwen3.5:397b-cloud).
  2. Verify model status via openclaw models status.
  3. Execute the image tool with a prompt and image URL.

Expected

The tool analyzes the image using the configured model and returns a description.

Actual

The tool returns a successful response with the image analysis (previously returned {"status": "error", "tool": "image", "error": "Unknown model: ..."}).

Evidence

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Human Verification (required)

  • Verified scenarios: local scoped validation and targeted checks for the changed area passed
  • Edge cases checked: relevant changed-path scenarios covered by selected validation
  • What you did not verify: full repository integration coverage beyond the selected validation scope

Compatibility / Migration

  • Backward compatible? (Yes)
  • Config/env changes? (No)
  • Migration needed? (No)
  • If yes, exact upgrade steps: N/A

Failure Recovery (if this breaks)

  • How to disable/revert this change quickly: Revert the commit modifying src/agents/tools/media-tool-shared.ts.
  • Files/config to restore: src/agents/tools/media-tool-shared.ts.
  • Known bad symptoms reviewers should watch for: If normalizeModelRef behaves unexpectedly, valid models might fail to resolve.

Risks and Mitiggedations

The change relies on the normalizeModelRef utility. If this utility handles edge cases differently than the previous raw lookup, it could theoretically affect resolution. However, since it is a shared utility imported from model-selection.js, it aligns behavior with the rest of the system and reduces inconsistency.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: XS labels Apr 13, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b4686f6f48

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +419 to 423
return requireApiKey({
provider: params.provider,
modelId: params.modelId,
agentDir: params.agentDir,
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Resolve provider auth before calling requireApiKey

resolveModelRuntimeApiKey now calls requireApiKey with { provider, modelId, agentDir }, but requireApiKey expects a resolved auth object containing apiKey (from getApiKeyForModel) and a provider string. In the PDF tool path (runPdfPrompt), this means API key resolution is skipped and the call will always throw (No API key resolved...) even when credentials are configured, so PDF analysis fails for all providers.

Useful? React with 👍 / 👎.

@greptile-apps

greptile-apps Bot commented Apr 13, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR correctly adds normalizeModelRef before the registry lookup in resolveModelFromRegistry, which fixes the "Unknown model" error for Ollama models with non-normalized references. However, the accompanying refactor of resolveModelRuntimeApiKey introduces two P0 bugs that will break the PDF tool.

  • resolveModelRuntimeApiKey now calls requireApiKey with a plain { provider, modelId, agentDir } object instead of the required ResolvedProviderAuth shape, omitting the mandatory source/mode fields and the second positional argument — this will always throw at runtime.
  • The caller in pdf-tool.ts (lines 147–152) was not updated and still passes the old { model, cfg, agentDir, authStorage } shape, causing a TypeScript compile error.
  • getApiKeyForModel and setRuntimeApiKey — which previously resolved and registered credentials — were removed with no replacement.

Confidence Score: 1/5

Not safe to merge — the refactored resolveModelRuntimeApiKey will throw on every call, and the unchanged pdf-tool.ts caller will fail TypeScript compilation.

Two P0 defects: (1) requireApiKey is called with the wrong type and a missing argument, so every resolveModelRuntimeApiKey call throws at runtime; (2) pdf-tool.ts still uses the old function signature, breaking compilation. The resolveModelFromRegistry normalization fix is correct and valuable, but the bundled refactor makes this PR a net regression for the PDF tool.

src/agents/tools/media-tool-shared.ts (lines 412–424) and the unchanged caller in src/agents/tools/pdf-tool.ts (lines 147–152)

Comments Outside Diff (1)

  1. src/agents/tools/pdf-tool.ts, line 147-152 (link)

    P0 resolveModelRuntimeApiKey called with old, removed signature

    The PR changed resolveModelRuntimeApiKey's parameter shape from { model, cfg, agentDir, authStorage } to { modelRegistry, provider, modelId, agentDir? }, but this call site in pdf-tool.ts still passes { model, cfg, agentDir, authStorage }. This is a TypeScript compile error (pnpm build / pnpm tsgo will fail) and will break the PDF tool at runtime for all callers.

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: src/agents/tools/pdf-tool.ts
    Line: 147-152
    
    Comment:
    **`resolveModelRuntimeApiKey` called with old, removed signature**
    
    The PR changed `resolveModelRuntimeApiKey`'s parameter shape from `{ model, cfg, agentDir, authStorage }` to `{ modelRegistry, provider, modelId, agentDir? }`, but this call site in `pdf-tool.ts` still passes `{ model, cfg, agentDir, authStorage }`. This is a TypeScript compile error (`pnpm build` / `pnpm tsgo` will fail) and will break the PDF tool at runtime for all callers.
    
    How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/agents/tools/pdf-tool.ts
Line: 147-152

Comment:
**`resolveModelRuntimeApiKey` called with old, removed signature**

The PR changed `resolveModelRuntimeApiKey`'s parameter shape from `{ model, cfg, agentDir, authStorage }` to `{ modelRegistry, provider, modelId, agentDir? }`, but this call site in `pdf-tool.ts` still passes `{ model, cfg, agentDir, authStorage }`. This is a TypeScript compile error (`pnpm build` / `pnpm tsgo` will fail) and will break the PDF tool at runtime for all callers.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/agents/tools/media-tool-shared.ts
Line: 418-423

Comment:
**Incorrect call to `requireApiKey`**

The `requireApiKey` helper expects `(resolvedProviderAuth, providerName)`. Here it is invoked with a single object `{ provider, modelId, agentDir }` that does not match `ResolvedProviderAuth` — the `source` and `mode` fields are missing. This will throw at runtime because the resolved value cannot be extracted. The second positional argument is also omitted.

On top of this, `getApiKeyForModel` (which resolves credentials through profiles/env/config) and `setRuntimeApiKey` from the old implementation have been dropped with no replacement, so no real credential is ever resolved by this path.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "fix: address syntax/type errors" | Re-trigger Greptile

Comment on lines +418 to 423
resolveModelFromRegistry(params);
return requireApiKey({
provider: params.provider,
modelId: params.modelId,
agentDir: params.agentDir,
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P0 Incorrect call to requireApiKey

The requireApiKey helper expects (resolvedProviderAuth, providerName). Here it is invoked with a single object { provider, modelId, agentDir } that does not match ResolvedProviderAuth — the source and mode fields are missing. This will throw at runtime because the resolved value cannot be extracted. The second positional argument is also omitted.

On top of this, getApiKeyForModel (which resolves credentials through profiles/env/config) and setRuntimeApiKey from the old implementation have been dropped with no replacement, so no real credential is ever resolved by this path.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/tools/media-tool-shared.ts
Line: 418-423

Comment:
**Incorrect call to `requireApiKey`**

The `requireApiKey` helper expects `(resolvedProviderAuth, providerName)`. Here it is invoked with a single object `{ provider, modelId, agentDir }` that does not match `ResolvedProviderAuth` — the `source` and `mode` fields are missing. This will throw at runtime because the resolved value cannot be extracted. The second positional argument is also omitted.

On top of this, `getApiKeyForModel` (which resolves credentials through profiles/env/config) and `setRuntimeApiKey` from the old implementation have been dropped with no replacement, so no real credential is ever resolved by this path.

How can I resolve this? If you propose a fix, please make it concise.

@vincentkoc

Copy link
Copy Markdown
Member

Thanks for chasing this. The core normalization fix is valid, but the branch also refactored the media-tool auth path in a way that breaks the PDF tool. I carried the safe version forward as #66422 on top of current main, with the lookup fix isolated, coverage added, and changelog wired, so I’m closing this PR in favor of the maintainer carry.

@vincentkoc vincentkoc closed this Apr 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: XS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

image tool returns 'Unknown model' for configured ollama models with image input

2 participants