fix(provider/google): support multimodal tool-result parts in function responses by GidianB · Pull Request #12777 · vercel/ai

GidianB · 2026-02-23T20:59:29Z

Background

Google provider tool results with output.type = 'content' were not fully mapped into Gemini functionResponse.parts, which blocked reliable multimodal tool-result flows for images/files.

Summary

This PR adds multimodal tool-result support for @ai-sdk/google function responses and documents provider limitations.

Provider changes

Updated packages/google/src/convert-to-google-generative-ai-messages.ts to:
- Map image-data and file-data tool-result parts into functionResponse.parts as inlineData.
- Support URL-style parts (image-url, file-url) only when they are base64 data: URLs.
- Throw UnsupportedFunctionalityError for remote HTTP(S) URL-style tool-result parts.
Updated packages/google/src/google-generative-ai-prompt.ts types to allow:
- functionResponse.parts with inline media payloads.

Tests

Added/updated tests in packages/google/src/convert-to-google-generative-ai-messages.test.ts for:
- image-data mapping
- file-data mapping
- image-url base64 data: URL mapping
- Error on non-data image-url
- Error on non-data file-url

Docs

Updated content/docs/03-ai-sdk-core/15-tools-and-tool-calling.mdx:
- Added Google support note (Gemini 3 models).
- Documented that Google tool-result URL-style media must use base64 data: URLs (remote HTTP(S) URLs are not supported).

Examples

Replaced combined toggle example with focused scripts:
- examples/ai-functions/src/generate-text/google-image-tool-result-base64.ts
- examples/ai-functions/src/generate-text/google-image-tool-result-url.ts
- examples/ai-functions/src/generate-text/google-pdf-tool-results-base64.ts
- examples/ai-functions/src/generate-text/google-pdf-tool-results-url.ts
Removed:
- examples/ai-functions/src/generate-text/google-image-tool-results.ts

Changeset

Added .changeset/google-multimodal-tool-results.md (patch for @ai-sdk/google).

Manual Verification

Ran targeted provider tests:
- pnpm --filter @ai-sdk/google test:node -- convert-to-google-generative-ai-messages.test.ts
Ran type checks:
- pnpm --filter @ai-sdk/google type-check
- pnpm --filter @example/ai-functions type-check
Ran lint/format checks on changed files:
- pnpm exec eslint ...
- pnpm exec prettier --check ...

Checklist

Tests have been added / updated (for bug fixes / features)
Documentation has been added / updated (for bug fixes / features)
A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root)
I have reviewed this pull request (self-review)

Future Work

Related Issues

N/A

…n responses

IdoZ14 · 2026-03-03T11:29:01Z

Hey @GidianB, I've looked over this issue myself, and I have a question regarding the image-url and file-url cases.
We have a use case where we want the model (specifically on vertex) to load the file from gcs using a gcs url (gs://...), and Ive tested locally and it seems to work.
Any particular reason why you only added support for data: urls?

AlonSh · 2026-03-10T16:06:07Z

@R-Taneja I'm facing something similar, currently solving locally with bun patch but would love to have it fix officially. How can we get such a thing assigned? Is there a matching issue?

felixarntz

@GidianB Thank you for this! Overall the logic looks great, but we need to continue supporting existing non-Gemini 3 models.

I'm pushing a few changes, mostly to refresh against recent example restructuring, plus reinstating the deleted example, plus also having those examples for streamText.

With those examples, you should be able to test the logic in support of the older Gemini models too. Could you please update the PR?

packages/google/src/convert-to-google-generative-ai-messages.ts

examples/ai-functions/src/generate-text/google-image-tool-results.ts

examples/ai-functions/src/generate-text/google/image-tool-result-base64.ts

felixarntz · 2026-03-13T17:05:50Z

packages/google/src/convert-to-google-generative-ai-messages.ts

+  const parsedDataUrl = parseBase64DataUrl(url);
+  if (parsedDataUrl == null) {
+    throw new UnsupportedFunctionalityError({
+      functionality: `URL-based ${type} tool result parts are not supported. Use image-data/file-data or base64 data URLs.`,
+    });
+  }


Technically, this would not have caused an error before, so let's think about whether this is the best path forward.

Before, the URL would have just been passed to the model, as a URL. It probably wouldn't have been able to do anything with it, so that's also not a great situation either way. But maybe it's better to still do pass the URL in this case, rather than throw an error.

I think we should include a warning in this case (via warnings), rather than throw. Curious what you think @lgrammel

I reverted this to the pre-error behavior for now: no throw and no warning, just fallback to the legacy text/JSON representation until we get more guidance.

felixarntz · 2026-03-13T17:32:42Z

@IdoZ14

Hey @GidianB, I've looked over this issue myself, and I have a question regarding the image-url and file-url cases. We have a use case where we want the model (specifically on vertex) to load the file from gcs using a gcs url (gs://...), and Ive tested locally and it seems to work. Any particular reason why you only added support for data: urls?

According to https://ai.google.dev/api/caching#FunctionResponsePart, this is not supported; it says that only inline base64 files can be used. Now of course it might be that the documentation is incomplete, but we need a way to verify. Could you share more context on how exactly you solved it locally? code snippets or a full example would be helpful.

IdoZ14 · 2026-03-15T12:40:27Z

Hey @felixarntz First off thanks for taking this on! it will help us a lot!
I see what you mean, Ive found it says its supported here:

maybe its only on vertex models.

Locally I called the vertex api directly, with some fake message history that links to a gcs file uri.
Something like this

// Before run:
// export ACCESS_TOKEN="your-access-token"

const PROJECT_ID = "your-project-id";
const LOCATION = "global";
const MODEL = "gemini-3-flash-preview";
const ACCESS_TOKEN = process.env.ACCESS_TOKEN;

const gcsFileUri = "gs://your-bucket/path/to/document.pdf";
const fileName = "document.pdf";

const url = `https://aiplatform.googleapis.com/v1beta1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL}:generateContent`;

const payload = {
  tools: [
    {
      functionDeclarations: [
        {
          name: "get_document_summary",
          description: "Fetches an insurance document and provides its summary.",
          parameters: {
            type: "object",
            properties: {
              file_name: { type: "string" }
            },
            required: ["file_name"]
          }
        }
      ]
    }
  ],
  contents: [
    {
      role: "user",
      parts: [
        { text: "Can you load the insurance policy document named Policy 1.PDF from storage?" }
      ]
    },
    {
      role: "model",
      parts: [
        {
          functionCall: {
            name: "get_document_summary",
            args: { file_name: fileName }
          }
        }
      ]
    },
    {
      role: "user",
      parts: [
        {
          functionResponse: {
            name: "get_document_summary",
            response: {
              status: "ok",
              file_name: fileName,
              gcs_uri: gcsFileUri
            },
            parts: [
              {
                fileData: {
                  fileUri: gcsFileUri,
                  mimeType: "application/pdf"
                }
              }
            ]
          }
        }
      ]
    },
    {
      role: "user",
      parts: [{ text: "Please summarize what is in that file." }]
    }
  ],
  generationConfig: {
    temperature: 0
  }
};

const res = await fetch(url, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${ACCESS_TOKEN}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify(payload)
});

const bodyText = await res.text();
console.log("status:", res.status, res.statusText);
console.log(bodyText);

This is was a file uri in tool response, but I also tried out in a model message (less useful IMO but still worked)

{
  role: "model",
  parts: [
    {
      fileData: {
        fileUri: gcsFileUri,
        mimeType: gcsMimeType,
      },
    },
  ],
},

…Gemini 3 models

GidianB · 2026-03-17T12:08:06Z

@felixarntz I updated the PR to preserve the legacy tool-result conversion path for pre-Gemini 3 models, while keeping multimodal functionResponse.parts for Gemini 3 / 3.1 models.

I also reverted the non-data: URL handling to the previous behavior for now: no throw and no warning, just fallback to the legacy text/JSON representation until we get more guidance there.

Verified with:

pnpm --filter @ai-sdk/google type-check
pnpm --filter @ai-sdk/google test:node
pnpm --filter @ai-sdk/google test:edge

felixarntz

@GidianB Looks great, thanks for restoring the old functionality so that now all Gemini models should work!

I broke the logic into two separate functions to reduce complexity a bit, and tested thoroughly, including a temporary local change for testing the "file data" part documented on Vertex. But it didn't work, neither for Gemini API nor with Vertex API:

Gemini API doesn't error, but clearly doesn't process the file
Vertex API errors that it cannot download the image - probably only works with certain Google-owned URLs

I added examples covering these use-cases already, which currently fail. But I think it's a good idea to keep them in here, so we can hopefully make them pass in the future - potentially with Vertex only. Anyway, that's better as a separate follow up PR. Please feel free to work on it if you're interested!

vercel-ai-sdk · 2026-03-20T18:31:43Z

⚠️ Backport to release-v6.0 created but has conflicts: #13684

@GidianB

…in function responses (#13684) This is an automated backport of #12777 to the release-v6.0 branch. FYI @GidianB ~~This backport has conflicts that need to be resolved manually.~~ Conflicts resolved. ### `git cherry-pick` output ``` Auto-merging packages/google/src/convert-to-google-generative-ai-messages.test.ts Auto-merging packages/google/src/convert-to-google-generative-ai-messages.ts CONFLICT (content): Merge conflict in packages/google/src/convert-to-google-generative-ai-messages.ts Auto-merging packages/google/src/google-generative-ai-language-model.ts error: could not apply 18c1970... fix(provider/google): support multimodal tool-result parts in function responses (#12777) hint: After resolving the conflicts, mark them with hint: "git add/rm <pathspec>", then run hint: "git cherry-pick --continue". hint: You can instead skip this commit with "git cherry-pick --skip". hint: To abort and get back to the state before "git cherry-pick", hint: run "git cherry-pick --abort". hint: Disable this message with "git config set advice.mergeConflict false" ``` --------- Co-authored-by: Gidian <gidianbateman@gmail.com> Co-authored-by: Felix Arntz <felix.arntz@vercel.com>

vercel-ai-sdk · 2026-03-20T20:40:20Z

🚀 Published in:

Package	Version
`ai`	`7.0.0-beta.32`
`@ai-sdk/angular`	`3.0.0-beta.32`
`@ai-sdk/gateway`	`4.0.0-beta.18`
`@ai-sdk/google`	`4.0.0-beta.15`
`@ai-sdk/google-vertex`	`5.0.0-beta.20`
`@ai-sdk/langchain`	`3.0.0-beta.32`
`@ai-sdk/llamaindex`	`3.0.0-beta.32`
`@ai-sdk/mistral`	`4.0.0-beta.8`
`@ai-sdk/react`	`4.0.0-beta.32`
`@ai-sdk/rsc`	`3.0.0-beta.33`
`@ai-sdk/svelte`	`5.0.0-beta.32`
`@ai-sdk/vue`	`4.0.0-beta.32`

fix(provider/google): support multimodal tool-result parts in functio…

4edcdbc

…n responses

felixarntz added 5 commits March 13, 2026 11:51

Merge branch 'main' into fix/google-multimodal-tool-results

efbfb47

reinstate removed example covering gemini-2.5

7944016

update changeset

0058f1d

fix relative path after moving files

02d28f4

add matching examples for streamText

1ce83b6

felixarntz requested changes Mar 13, 2026

View reviewed changes

fix(provider/google): preserve legacy tool-result conversion for pre-…

62cdd8b

…Gemini 3 models

felixarntz added 3 commits March 20, 2026 12:08

Merge branch 'main' into fix/google-multimodal-tool-results

160daaf

break logic into its own function to reduce complexity

b4df707

adjust examples, test file support (not working yet)

699d3e5

felixarntz approved these changes Mar 20, 2026

View reviewed changes

felixarntz added the backport Admins only: add this label to a pull request in order to backport it to the prior version label Mar 20, 2026

felixarntz merged commit 18c1970 into vercel:main Mar 20, 2026
19 of 20 checks passed

vercel-ai-sdk bot pushed a commit that referenced this pull request Mar 20, 2026

Backport conflicts for PR #12777 to release-v6.0

3e7f4c8

vercel-ai-sdk bot mentioned this pull request Mar 20, 2026

Backport: fix(provider/google): support multimodal tool-result parts in function responses #13684

Merged

vercel-ai-sdk bot removed the backport Admins only: add this label to a pull request in order to backport it to the prior version label Mar 20, 2026

Conversation

GidianB commented Feb 23, 2026

Background

Summary

Provider changes

Tests

Docs

Examples

Changeset

Manual Verification

Checklist

Future Work

Related Issues

Uh oh!

IdoZ14 commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlonSh commented Mar 10, 2026

Uh oh!

felixarntz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

felixarntz Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

GidianB Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

felixarntz commented Mar 13, 2026

Uh oh!

IdoZ14 commented Mar 15, 2026

Uh oh!

GidianB commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixarntz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vercel-ai-sdk bot commented Mar 20, 2026

Uh oh!

vercel-ai-sdk bot commented Mar 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

IdoZ14 commented Mar 3, 2026 •

edited

Loading

GidianB commented Mar 17, 2026 •

edited

Loading