Skip to content

Backport: fix(provider/google): correctly mark reasoning files as such and fix related multi-turn errors#13281

Merged
vercel-ai-sdk[bot] merged 1 commit intorelease-v6.0from
backport-pr-13262-to-release-v6.0
Mar 9, 2026
Merged

Backport: fix(provider/google): correctly mark reasoning files as such and fix related multi-turn errors#13281
vercel-ai-sdk[bot] merged 1 commit intorelease-v6.0from
backport-pr-13262-to-release-v6.0

Conversation

@vercel-ai-sdk
Copy link
Copy Markdown
Contributor

@vercel-ai-sdk vercel-ai-sdk bot commented Mar 9, 2026

This is an automated backport of #13262 to the release-v6.0 branch. FYI @felixarntz

…related multi-turn errors (#13262)

## Background

When using Gemini thinking models with image output, the Google API
returns `thought: true` on non-text parts (e.g. `inlineData` images).
The SDK correctly maps `thought: true` on text parts to `type:
'reasoning'`, but silently strips the flag from file parts. This causes
problems in multi-turn exchanges: the thought-image gets sent back as a
regular image, causing API errors because they're interpreted as regular
images but lack thought signatures (which per Google API are only
present on the non-reasoning images).

## Summary

Since we can't introduce a new part type without a spec change, this
preserves the `thought` flag on file parts via
`providerMetadata`/`providerOptions`.

- Add `thought: z.boolean().nullish()` to the `inlineData` member in the
provider-specific response schema
- In both `doGenerate` and `doStream`, propagate `thought: true` into
`providerMetadata` on file parts (alongside `thoughtSignature`)
- In `convertToGoogleGenerativeAIMessages`, read `thought` from
`providerMetadata` and send it back to the API on file parts

Function examples for `generate-text` and `stream-text` demonstrating
multi-step image generation with thinking are added, as well as an E2E
example that allows interactive multi-turn with reasoning output that
includes the images.

## Manual Verification

Test with the new examples to verify; for the E2E example making two
turns would result in an error without the `packages` fixes from this
PR.

## Checklist

- [x] Tests have been added / updated (for bug fixes / features)
- [ ] Documentation has been added / updated (for bug fixes / features)
- [x] A _patch_ changeset for relevant packages has been added (for bug
fixes / features - run `pnpm changeset` in the project root)
- [x] I have reviewed this pull request (self-review)

## Future Work

A provider-agnostic mechanism to distinguish thought content from output
content regardless of part type (as discussed in #12516) would be the
ideal long-term solution, removing the need for provider-specific
`providerMetadata` checks. However, this will require a spec change and
therefore is only an option for v7 - we should explore that further.

## Related Issues

Fixes #11461
@tigent tigent bot added ai/core core functions like generateText, streamText, etc. Provider utils, and provider spec. ai/provider related to a provider package. Must be assigned together with at least one `provider/*` label ai/ui anything UI related bug Something isn't working as documented provider/google Issues related to the @ai-sdk/google provider labels Mar 9, 2026
@vercel-ai-sdk vercel-ai-sdk bot merged commit 9d46b93 into release-v6.0 Mar 9, 2026
28 checks passed
@vercel-ai-sdk vercel-ai-sdk bot deleted the backport-pr-13262-to-release-v6.0 branch March 9, 2026 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai/core core functions like generateText, streamText, etc. Provider utils, and provider spec. ai/provider related to a provider package. Must be assigned together with at least one `provider/*` label ai/ui anything UI related bug Something isn't working as documented provider/google Issues related to the @ai-sdk/google provider

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant