Skip to content

feat(google): add support for image search, replace obsolete google_search_retrieval implementation#12926

Merged
felixarntz merged 9 commits intomainfrom
fa-fix-google-search
Mar 4, 2026
Merged

feat(google): add support for image search, replace obsolete google_search_retrieval implementation#12926
felixarntz merged 9 commits intomainfrom
fa-fix-google-search

Conversation

@felixarntz
Copy link
Copy Markdown
Collaborator

Background

Google introduced image search to their API, which can be used for grounding when generating images (e.g. with gemini-3.1-flash-image-preview): https://ai.google.dev/gemini-api/docs/image-generation#image-search

In the current AI SDK implementation, the googleSearch tool had mode and dynamicThreshold parameters, which were actually from the google_search_retrieval tool that Google defines, which only worked with Gemini 1.5 models. Since those models are retired and can no longer be accessed, these parameters were effectively dead code — silently ignored for all Gemini 2.0+ models. Meanwhile, the current Google Search API supports new capabilities (searchTypes for image search, timeRangeFilter) that the SDK didn't support.

Summary

Replaces the obsolete google_search_retrieval code paths with support for the current googleSearch API parameters:

  • searchTypes: Enables web and/or image search grounding (e.g. { imageSearch: {} })
  • timeRangeFilter: Restricts search results to a time range via startTime/endTime

The googleSearchRetrieval logic and supportsDynamicRetrieval variable are removed entirely. Since they were impossible to use in practice, this is effectively not a breaking change (because such code would already be broken).

The grounding metadata schema and source extraction are extended to handle image grounding chunks (with sourceUri, imageUri, title, domain), enabling proper source attribution for image search results.

Manual Verification

Ran the new examples against the Google API:

  • examples/ai-functions/src/generate-text/google/search-types.ts — verified searchTypes and timeRangeFilter pass through correctly
  • examples/ai-functions/src/generate-text/google/image-search.ts — verified image search grounding returns image results with proper source attribution
  • examples/ai-functions/src/stream-text/google/search-types.ts — verified streaming with search types
  • examples/ai-functions/src/stream-text/google/image-search.ts — verified streaming image search

Checklist

  • Tests have been added / updated (for bug fixes / features)
  • Documentation has been added / updated (for bug fixes / features)
  • A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root)
  • I have reviewed this pull request (self-review)

Future Work

Consider adding a dedicated source type in v7 to include both imageUri and sourceUri separately for image grounding chunks. In the PR implementation, we cannot surface both because our spec does not support it.

@tigent tigent bot added ai/provider related to a provider package. Must be assigned together with at least one `provider/*` label documentation Improvements or additions to documentation feature New feature or request provider/google Issues related to the @ai-sdk/google provider labels Feb 27, 2026
@felixarntz felixarntz added the backport Admins only: add this label to a pull request in order to backport it to the prior version label Feb 27, 2026
@felixarntz felixarntz merged commit 2565e70 into main Mar 4, 2026
18 checks passed
@felixarntz felixarntz deleted the fa-fix-google-search branch March 4, 2026 22:46
vercel-ai-sdk bot pushed a commit that referenced this pull request Mar 4, 2026
@vercel-ai-sdk vercel-ai-sdk bot removed the backport Admins only: add this label to a pull request in order to backport it to the prior version label Mar 4, 2026
@vercel-ai-sdk
Copy link
Copy Markdown
Contributor

vercel-ai-sdk bot commented Mar 4, 2026

⚠️ Backport to release-v5.0 created but has conflicts: #13070

@vercel-ai-sdk
Copy link
Copy Markdown
Contributor

vercel-ai-sdk bot commented Mar 4, 2026

🚀 Published in:

Package Version
@ai-sdk/google 3.0.39
@ai-sdk/google-vertex 4.0.76

felixarntz added a commit that referenced this pull request Mar 5, 2026
…e google_search_retrieval implementation (#13070)

This is an automated backport of #12926 to the release-v5.0 branch. FYI
@felixarntz
~~This backport has conflicts that need to be resolved manually.~~
Conflicts resolved.

### `git cherry-pick` output

```
Auto-merging content/providers/01-ai-sdk-providers/15-google-generative-ai.mdx
CONFLICT (content): Merge conflict in content/providers/01-ai-sdk-providers/15-google-generative-ai.mdx
CONFLICT (file location): examples/ai-functions/src/generate-text/google/image-search.ts added in 2565e70 (feat(google): add support for image search, replace obsolete google_search_retrieval implementation (#12926)) inside a directory that was renamed in HEAD, suggesting it should perhaps be moved to examples/ai-core/src/generate-text/image-search.ts.
CONFLICT (file location): examples/ai-functions/src/generate-text/google/search-types.ts added in 2565e70 (feat(google): add support for image search, replace obsolete google_search_retrieval implementation (#12926)) inside a directory that was renamed in HEAD, suggesting it should perhaps be moved to examples/ai-core/src/generate-text/search-types.ts.
CONFLICT (file location): examples/ai-functions/src/stream-text/google/image-search.ts added in 2565e70 (feat(google): add support for image search, replace obsolete google_search_retrieval implementation (#12926)) inside a directory that was renamed in HEAD, suggesting it should perhaps be moved to examples/ai-core/src/stream-text/image-search.ts.
CONFLICT (file location): examples/ai-functions/src/stream-text/google/search-types.ts added in 2565e70 (feat(google): add support for image search, replace obsolete google_search_retrieval implementation (#12926)) inside a directory that was renamed in HEAD, suggesting it should perhaps be moved to examples/ai-core/src/stream-text/search-types.ts.
Auto-merging packages/google/src/google-generative-ai-language-model.test.ts
CONFLICT (content): Merge conflict in packages/google/src/google-generative-ai-language-model.test.ts
Auto-merging packages/google/src/google-generative-ai-language-model.ts
Auto-merging packages/google/src/google-prepare-tools.test.ts
Auto-merging packages/google/src/google-prepare-tools.ts
CONFLICT (content): Merge conflict in packages/google/src/google-prepare-tools.ts
Auto-merging packages/google/src/tool/google-search.ts
CONFLICT (content): Merge conflict in packages/google/src/tool/google-search.ts
error: could not apply 2565e70... feat(google): add support for image search, replace obsolete google_search_retrieval implementation (#12926)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
```

---------

Co-authored-by: Felix Arntz <felix.arntz@vercel.com>
felixarntz added a commit that referenced this pull request Mar 6, 2026
…arch with multimodal output (#13171)

## Summary

This helps test and verify that thought signatures now work correctly
end-to-end, for both images and web search and image search sources,
including in multi-turn content.

## Manual Verification

Did some extensive testing with these two new examples - everything
works as expected.

## Checklist

- [ ] Tests have been added / updated (for bug fixes / features)
- [ ] Documentation has been added / updated (for bug fixes / features)
- [ ] A _patch_ changeset for relevant packages has been added (for bug
fixes / features - run `pnpm changeset` in the project root)
- [x] I have reviewed this pull request (self-review)

## Related Issues

Follow up to #12926
vercel-ai-sdk bot pushed a commit that referenced this pull request Mar 6, 2026
…arch with multimodal output (#13171)

## Summary

This helps test and verify that thought signatures now work correctly
end-to-end, for both images and web search and image search sources,
including in multi-turn content.

## Manual Verification

Did some extensive testing with these two new examples - everything
works as expected.

## Checklist

- [ ] Tests have been added / updated (for bug fixes / features)
- [ ] Documentation has been added / updated (for bug fixes / features)
- [ ] A _patch_ changeset for relevant packages has been added (for bug
fixes / features - run `pnpm changeset` in the project root)
- [x] I have reviewed this pull request (self-review)

## Related Issues

Follow up to #12926
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai/provider related to a provider package. Must be assigned together with at least one `provider/*` label documentation Improvements or additions to documentation feature New feature or request provider/google Issues related to the @ai-sdk/google provider

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants