feat(provider): add support for provider references and uploading files as supported per provider#13816
Conversation
…id with newer equivalents
…ider reference for a file as part of model messages
lgrammel
left a comment
There was a problem hiding this comment.
might be good to update the migration guide as part of this PR so we do not forget about the breaking changes / required migrations
| data, | ||
| signatures: [...imageMediaTypeSignatures, ...documentMediaTypeSignatures], | ||
| }) ?? | ||
| (isLikelyText(data) ? 'text/plain' : undefined); |
There was a problem hiding this comment.
mimeType and mediaType are the same afaik (mime type is the original name, media type is the current standard name), and web standards have moved to mediaType, which is why i standardized everything on mediaType. Re-introducing mimeType would make it inconsistent (and confusing imo if we have both) - if we have mimeType anywhere we should rename it to mediaType (unless it is provider specific).
| } | ||
| | { | ||
| type: 'file-id'; | ||
| type: 'file-reference'; |
There was a problem hiding this comment.
above we used data: LanguageModelV4DataContent | SharedV4ProviderReference;, here we have separate file-reference/file-data types.
this is somewhat inconsistent. the separation here makes sense to me (bc file reference do not need filename etc) - wondering if it should be changed above
| const providerMap = { | ||
| anthropic: (modelId: string) => anthropic(modelId), | ||
| google: (modelId: string) => google(modelId), | ||
| openai: (modelId: string) => openai.responses(modelId), | ||
| xai: (modelId: string) => xai(modelId), | ||
| }; |
There was a problem hiding this comment.
use provider registry? (or todo for future work)
There was a problem hiding this comment.
will do this in a separate PR next week
| prompt: standardizedPrompt, | ||
| supportedUrls: await model.supportedUrls, | ||
| download, | ||
| provider: model.provider.split('.')[0], |
There was a problem hiding this comment.
maybe consider extracting a function so we can search for it when we removed this in the future
There was a problem hiding this comment.
since it's temporary, I think it's okay not to introduce an external helper for this.
looking for provider.split('.')[0] throughout the codebase will show all relevant usage
|
🚀 Published in:
|
Background
The AI SDK supports passing media files inline or via URL, but has no way to upload files directly to a provider or reference previously uploaded files across providers. Some providers return internal file IDs (not URLs) from their upload APIs, and switching providers mid-conversation requires a way to map the same logical file to different provider-specific identifiers.
Summary
Introduces
uploadFileas a top-level function andProviderReference(Record<string, string>) as the provider-independent way to reference uploaded files.SharedV4ProviderReference(provider package),FilesV4interface withuploadFilemethod,UploadFileResult;mediaTypeandfilename** are top-level parameters as they're widely supported and useduploadFile({ files, data, mediaType?, filename?, providerOptions? })in theaipackage, with auto-detection of media type from file bytes when not providedfiles()interface on Anthropic, Google, OpenAI, and xAI providers, each implementingFilesV4.uploadFile*.jsonl), which we don't support anyway.LanguageModelV4FilePart.datanow acceptsSharedV4ProviderReferencein addition toDataContent; providers that support file references (Anthropic, Google, OpenAI, xAI) resolve them viaresolveProviderReference; all other providers throwUnsupportedFunctionalityErrorfile-idandimage-file-idtool result output types replaced withfile-referenceandimage-file-referenceusingSharedV4ProviderReferenceinstead ofstring | Record<string, string>reasoning-filewas not touched — it is model-generated as part of reasoning output, so provider references are not applicableuploadFileandProviderReference, and a new architecture guide are includedDesign decisions
ProviderReferenceis a plainRecord<string, string>rather than a wrapper class, keeping it simple to create and mergeisLikelyTextheuristic for media type detection and thedocumentMediaTypeSignaturesare kept internal (not exported) — they work well enough foruploadFilebut are not general-purpose utilitiesresolveProviderReference(provider-utils) does the lookup by provider name and throws with a clear error listing available providers when the reference doesn't contain an entry for the current providerOpen questions
typeproperty inProviderReferenceto distinguish different kinds of provider references (e.g. file vs skill, see Add support for creating and managing remote skills (e.g. Anthropic, OpenAI) via abstraction with new top-level API #12855)?mediaTypeandfilenamebe top-level fields in theuploadFileresult object?providerMetadata.mergeProviderReferencesis currently inlined in an example (multi-provider.ts) — should we offer this as a utility, or leave it for later?file-idandimage-file-idwere removed intoModelOutputreturn value — should we deprecate them instead and/or offer auto-migration via codemod?Manual Verification
Upload file examples were added for all 4 supported providers (Anthropic, Google, OpenAI, xAI), each with image, PDF, and text variants.
Checklist
pnpm changesetin the project root)Future Work
Reuse the new
ProviderReferenceapproach for #12855.Related Issues
Fixes #12995