-
Notifications
You must be signed in to change notification settings - Fork 4.2k
Consider aligning FilePart and how files are handled in ToolResultOutput #14010
Copy link
Copy link
Open
Labels
ai/corecore functions like generateText, streamText, etc. Provider utils, and provider spec.core functions like generateText, streamText, etc. Provider utils, and provider spec.ai/providerrelated to a provider package. Must be assigned together with at least one `provider/*` labelrelated to a provider package. Must be assigned together with at least one `provider/*` labeldeprecationpull requests that introduced a deprecationpull requests that introduced a deprecationfeatureNew feature or requestNew feature or request
Milestone
Metadata
Metadata
Assignees
Labels
ai/corecore functions like generateText, streamText, etc. Provider utils, and provider spec.core functions like generateText, streamText, etc. Provider utils, and provider spec.ai/providerrelated to a provider package. Must be assigned together with at least one `provider/*` labelrelated to a provider package. Must be assigned together with at least one `provider/*` labeldeprecationpull requests that introduced a deprecationpull requests that introduced a deprecationfeatureNew feature or requestNew feature or request
Type
Fields
Give feedbackNo fields configured for issues without a type.
FilePart(as part ofContentPart) uses a singletype: "file"with an overloadeddatafield (DataContent | URL, further expanded in #13816). In contrast, file content withinToolResultOutputalready uses distinguishable types:file-data,file-url,file-id, etc.This inconsistency causes practical problems:
stringvalue indatais always assumed to be base64. Using it for anything else (e.g. a URL string) is brittle or requires hacky workarounds. See feat(provider/openai): full implementation for Files API file-ids and urls in Images, PDFs, across Chat and Responses #7804 for prior workaround, plus this was a real constraint Add support for uploading media files (e.g. images, audio, documents) via abstraction with new top-level API #12995 where we wouldn't be able to support a string-based provider reference because of this.data(string vs URL vs Uint8Array) to determine intent, rather than matching on a discriminatedtypefield.Proposal
FilePartinto multiple discriminated types, mirroring whatToolResultOutputalready does."file") as deprecated parts of the union, which can be auto-mapped to the new discriminated types.Note: This affects both
packages/provider-utils/src/types/content-part.tsandpackages/ai/src/prompt/content-part.ts.Aside: we should also consider getting rid of the
filevsimagedistinction, because an image is simply a file with a specific media type. That's a separate change though, potentially to implement before this one.