Skip to content

fix(provider/google): correctly mark reasoning files as such and fix related multi-turn errors#13262

Merged
felixarntz merged 4 commits intomainfrom
fa-fix-google-missing-file-reasoning
Mar 9, 2026
Merged

fix(provider/google): correctly mark reasoning files as such and fix related multi-turn errors#13262
felixarntz merged 4 commits intomainfrom
fa-fix-google-missing-file-reasoning

Conversation

@felixarntz
Copy link
Copy Markdown
Collaborator

Background

When using Gemini thinking models with image output, the Google API returns thought: true on non-text parts (e.g. inlineData images). The SDK correctly maps thought: true on text parts to type: 'reasoning', but silently strips the flag from file parts. This causes problems in multi-turn exchanges: the thought-image gets sent back as a regular image, causing API errors because they're interpreted as regular images but lack thought signatures (which per Google API are only present on the non-reasoning images).

Summary

Since we can't introduce a new part type without a spec change, this preserves the thought flag on file parts via providerMetadata/providerOptions.

  • Add thought: z.boolean().nullish() to the inlineData member in the provider-specific response schema
  • In both doGenerate and doStream, propagate thought: true into providerMetadata on file parts (alongside thoughtSignature)
  • In convertToGoogleGenerativeAIMessages, read thought from providerMetadata and send it back to the API on file parts

Function examples for generate-text and stream-text demonstrating multi-step image generation with thinking are added, as well as an E2E example that allows interactive multi-turn with reasoning output that includes the images.

Manual Verification

Test with the new examples to verify; for the E2E example making two turns would result in an error without the packages fixes from this PR.

Checklist

  • Tests have been added / updated (for bug fixes / features)
  • Documentation has been added / updated (for bug fixes / features)
  • A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root)
  • I have reviewed this pull request (self-review)

Future Work

A provider-agnostic mechanism to distinguish thought content from output content regardless of part type (as discussed in #12516) would be the ideal long-term solution, removing the need for provider-specific providerMetadata checks. However, this will require a spec change and therefore is only an option for v7 - we should explore that further.

Related Issues

Fixes #11461

@tigent tigent bot added ai/provider related to a provider package. Must be assigned together with at least one `provider/*` label bug Something isn't working as documented provider/google Issues related to the @ai-sdk/google provider reproduction provided labels Mar 9, 2026
@felixarntz felixarntz added the backport Admins only: add this label to a pull request in order to backport it to the prior version label Mar 9, 2026
@gr2m
Copy link
Copy Markdown
Collaborator

gr2m commented Mar 9, 2026

Since we can't introduce a new part type without a spec change

we can now. Okay to merge as is and backport to v6, but then we can implement a spec change in main for v7 beta

@felixarntz felixarntz merged commit 2edd14e into main Mar 9, 2026
31 checks passed
@felixarntz felixarntz deleted the fa-fix-google-missing-file-reasoning branch March 9, 2026 21:04
vercel-ai-sdk bot pushed a commit that referenced this pull request Mar 9, 2026
…related multi-turn errors (#13262)

## Background

When using Gemini thinking models with image output, the Google API
returns `thought: true` on non-text parts (e.g. `inlineData` images).
The SDK correctly maps `thought: true` on text parts to `type:
'reasoning'`, but silently strips the flag from file parts. This causes
problems in multi-turn exchanges: the thought-image gets sent back as a
regular image, causing API errors because they're interpreted as regular
images but lack thought signatures (which per Google API are only
present on the non-reasoning images).

## Summary

Since we can't introduce a new part type without a spec change, this
preserves the `thought` flag on file parts via
`providerMetadata`/`providerOptions`.

- Add `thought: z.boolean().nullish()` to the `inlineData` member in the
provider-specific response schema
- In both `doGenerate` and `doStream`, propagate `thought: true` into
`providerMetadata` on file parts (alongside `thoughtSignature`)
- In `convertToGoogleGenerativeAIMessages`, read `thought` from
`providerMetadata` and send it back to the API on file parts

Function examples for `generate-text` and `stream-text` demonstrating
multi-step image generation with thinking are added, as well as an E2E
example that allows interactive multi-turn with reasoning output that
includes the images.

## Manual Verification

Test with the new examples to verify; for the E2E example making two
turns would result in an error without the `packages` fixes from this
PR.

## Checklist

- [x] Tests have been added / updated (for bug fixes / features)
- [ ] Documentation has been added / updated (for bug fixes / features)
- [x] A _patch_ changeset for relevant packages has been added (for bug
fixes / features - run `pnpm changeset` in the project root)
- [x] I have reviewed this pull request (self-review)

## Future Work

A provider-agnostic mechanism to distinguish thought content from output
content regardless of part type (as discussed in #12516) would be the
ideal long-term solution, removing the need for provider-specific
`providerMetadata` checks. However, this will require a spec change and
therefore is only an option for v7 - we should explore that further.

## Related Issues

Fixes #11461
@vercel-ai-sdk vercel-ai-sdk bot removed the backport Admins only: add this label to a pull request in order to backport it to the prior version label Mar 9, 2026
@vercel-ai-sdk
Copy link
Copy Markdown
Contributor

vercel-ai-sdk bot commented Mar 9, 2026

✅ Backport PR created: #13281

vercel-ai-sdk bot added a commit that referenced this pull request Mar 9, 2026
…h and fix related multi-turn errors (#13281)

This is an automated backport of #13262 to the release-v6.0 branch. FYI
@felixarntz

Co-authored-by: Felix Arntz <felix.arntz@vercel.com>
@vercel-ai-sdk
Copy link
Copy Markdown
Contributor

vercel-ai-sdk bot commented Mar 9, 2026

🚀 Published in:

Package Version
@ai-sdk/google 4.0.0-beta.4
@ai-sdk/google-vertex 5.0.0-beta.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai/provider related to a provider package. Must be assigned together with at least one `provider/*` label bug Something isn't working as documented provider/google Issues related to the @ai-sdk/google provider reproduction provided

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No way to distinguish thought file from regular file in Gemini 3 image responses

2 participants