Skip to content

Commit 4024a3a

Browse files
gr2mclaudevercel[bot]
authored
fix/unbounded download dos (#12445)
- Replace unbounded `arrayBuffer()`/`blob()` calls in `download()` and `downloadBlob()` with streaming reads that enforce a **2 GiB default size limit** - Add `abortSignal` passthrough from callers (`transcribe`, `generateVideo`) to `fetch()` - Check `Content-Length` header for early rejection before reading body - Track bytes incrementally via `ReadableStream.getReader()`, abort with `DownloadError` when limit exceeded - Expose configurable `download` parameter on `transcribe()` and `experimental_generateVideo()` (instead of adding a new `maxDownloadSize` argument) — keeps download config separate from API function signatures - Export `createDownload({ maxBytes })` factory from `ai` for custom size limits closes #9481 / addresses #9481 (comment) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
1 parent ee508b4 commit 4024a3a

File tree

16 files changed

+615
-29
lines changed

16 files changed

+615
-29
lines changed

‎.changeset/fix-unbounded-download-dos.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
'ai': patch
3+
'@ai-sdk/provider-utils': patch
4+
---
5+
6+
security: prevent unbounded memory growth in download functions
7+
8+
The `download()` and `downloadBlob()` functions now enforce a default 2 GiB size limit when downloading from user-provided URLs. Downloads that exceed this limit are aborted with a `DownloadError` instead of consuming unbounded memory and crashing the process. The `abortSignal` parameter is now passed through to `fetch()` in all download call sites.
9+
10+
Added `download` option to `transcribe()` and `experimental_generateVideo()` for providing a custom download function. Use the new `createDownload({ maxBytes })` factory to configure download size limits.

‎content/docs/03-ai-sdk-core/36-transcription.mdx‎

Lines changed: 57 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -54,21 +54,75 @@ const transcript = await transcribe({
5454
});
5555
```
5656

57+
### Download Size Limits
58+
59+
When `audio` is a URL, the SDK downloads the file with a default **2 GiB** size limit.
60+
You can customize this using `createDownload`:
61+
62+
```ts highlight="1,8"
63+
import { experimental_transcribe as transcribe, createDownload } from 'ai';
64+
import { openai } from '@ai-sdk/openai';
65+
66+
const transcript = await transcribe({
67+
model: openai.transcription('whisper-1'),
68+
audio: new URL('https://example.com/audio.mp3'),
69+
download: createDownload({ maxBytes: 50 * 1024 * 1024 }), // 50 MB limit
70+
});
71+
```
72+
73+
You can also provide a fully custom download function:
74+
75+
```ts highlight="6-12"
76+
import { experimental_transcribe as transcribe } from 'ai';
77+
import { openai } from '@ai-sdk/openai';
78+
79+
const transcript = await transcribe({
80+
model: openai.transcription('whisper-1'),
81+
audio: new URL('https://example.com/audio.mp3'),
82+
download: async ({ url }) => {
83+
const res = await myAuthenticatedFetch(url);
84+
return {
85+
data: new Uint8Array(await res.arrayBuffer()),
86+
mediaType: res.headers.get('content-type') ?? undefined,
87+
};
88+
},
89+
});
90+
```
91+
92+
If a download exceeds the size limit, a `DownloadError` is thrown:
93+
94+
```ts
95+
import { experimental_transcribe as transcribe, DownloadError } from 'ai';
96+
import { openai } from '@ai-sdk/openai';
97+
98+
try {
99+
await transcribe({
100+
model: openai.transcription('whisper-1'),
101+
audio: new URL('https://example.com/audio.mp3'),
102+
});
103+
} catch (error) {
104+
if (DownloadError.isInstance(error)) {
105+
console.log('Download failed:', error.message);
106+
}
107+
}
108+
```
109+
57110
### Abort Signals and Timeouts
58111

59112
`transcribe` accepts an optional `abortSignal` parameter of
60113
type [`AbortSignal`](https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal)
61114
that you can use to abort the transcription process or set a timeout.
62115

116+
This is particularly useful when combined with URL downloads to prevent long-running requests:
117+
63118
```ts highlight="8"
64119
import { openai } from '@ai-sdk/openai';
65120
import { experimental_transcribe as transcribe } from 'ai';
66-
import { readFile } from 'fs/promises';
67121

68122
const transcript = await transcribe({
69123
model: openai.transcription('whisper-1'),
70-
audio: await readFile('audio.mp3'),
71-
abortSignal: AbortSignal.timeout(1000), // Abort after 1 second
124+
audio: new URL('https://example.com/audio.mp3'),
125+
abortSignal: AbortSignal.timeout(5000), // Abort after 5 seconds
72126
});
73127
```
74128

‎content/docs/07-reference/01-ai-sdk-core/11-transcribe.mdx‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,13 @@ console.log(transcript);
6969
isOptional: true,
7070
description: 'Additional HTTP headers for the request.',
7171
},
72+
{
73+
name: 'download',
74+
type: '(options: { url: URL; abortSignal?: AbortSignal }) => Promise<{ data: Uint8Array; mediaType: string | undefined }>',
75+
isOptional: true,
76+
description:
77+
'Custom download function for fetching audio from URLs. Use `createDownload()` from `ai` to create a download function with custom size limits, e.g. `createDownload({ maxBytes: 50 * 1024 * 1024 })`. Default: built-in download with 2 GiB limit.',
78+
},
7279
]}
7380
/>
7481

‎content/docs/07-reference/01-ai-sdk-core/13-generate-video.mdx‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,13 @@ console.log(videos);
140140
isOptional: true,
141141
description: 'Additional HTTP headers for the request.',
142142
},
143+
{
144+
name: 'download',
145+
type: '(options: { url: URL; abortSignal?: AbortSignal }) => Promise<{ data: Uint8Array; mediaType: string | undefined }>',
146+
isOptional: true,
147+
description:
148+
'Custom download function for fetching videos from URLs. Use `createDownload()` from `ai` to create a download function with custom size limits, e.g. `createDownload({ maxBytes: 50 * 1024 * 1024 })`. Default: built-in download with 2 GiB limit.',
149+
},
143150
]}
144151
/>
145152

‎packages/ai/scripts/check-bundle-size.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ import { writeFileSync, statSync } from 'fs';
33
import { join } from 'path';
44

55
// Bundle size limits in bytes
6-
const LIMIT = 555 * 1024;
6+
const LIMIT = 560 * 1024;
77

88
interface BundleResult {
99
size: number;

‎packages/ai/src/generate-video/generate-video.ts‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ import {
2525
imageMediaTypeSignatures,
2626
videoMediaTypeSignatures,
2727
} from '../util/detect-media-type';
28-
import { download } from '../util/download/download';
28+
import { createDownload } from '../util/download/create-download';
2929
import { prepareRetries } from '../util/prepare-retries';
3030
import { VERSION } from '../version';
3131
import type { GenerateVideoResult } from './generate-video-result';
@@ -57,6 +57,8 @@ export type GenerateVideoPrompt =
5757
*
5858
* @returns A result object that contains the generated videos.
5959
*/
60+
const defaultDownload = createDownload();
61+
6062
export async function experimental_generateVideo({
6163
model: modelArg,
6264
prompt: promptArg,
@@ -71,6 +73,7 @@ export async function experimental_generateVideo({
7173
maxRetries: maxRetriesArg,
7274
abortSignal,
7375
headers,
76+
download: downloadFn = defaultDownload,
7477
}: {
7578
/**
7679
* The video model to use.
@@ -140,6 +143,17 @@ export async function experimental_generateVideo({
140143
* Only applicable for HTTP-based providers.
141144
*/
142145
headers?: Record<string, string>;
146+
147+
/**
148+
* Custom download function for fetching videos from URLs.
149+
* Use `createDownload()` from `ai` to create a download function with custom size limits.
150+
*
151+
* @default createDownload() (2 GiB limit)
152+
*/
153+
download?: (options: {
154+
url: URL;
155+
abortSignal?: AbortSignal;
156+
}) => Promise<{ data: Uint8Array; mediaType: string | undefined }>;
143157
}): Promise<GenerateVideoResult> {
144158
const model = resolveVideoModel(modelArg);
145159

@@ -195,8 +209,9 @@ export async function experimental_generateVideo({
195209
for (const videoData of result.videos) {
196210
switch (videoData.type) {
197211
case 'url': {
198-
const { data, mediaType: downloadedMediaType } = await download({
212+
const { data, mediaType: downloadedMediaType } = await downloadFn({
199213
url: new URL(videoData.url),
214+
abortSignal,
200215
});
201216

202217
// Filter out generic/unknown media types that should fall through to detection

‎packages/ai/src/transcribe/transcribe.ts‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ import {
1010
audioMediaTypeSignatures,
1111
detectMediaType,
1212
} from '../util/detect-media-type';
13-
import { download } from '../util/download/download';
13+
import { createDownload } from '../util/download/create-download';
1414
import { prepareRetries } from '../util/prepare-retries';
1515
import { TranscriptionResult } from './transcribe-result';
1616
import { VERSION } from '../version';
@@ -29,13 +29,16 @@ import { Warning } from '../types';
2929
*
3030
* @returns A result object that contains the generated transcript.
3131
*/
32+
const defaultDownload = createDownload();
33+
3234
export async function transcribe({
3335
model,
3436
audio,
3537
providerOptions = {},
3638
maxRetries: maxRetriesArg,
3739
abortSignal,
3840
headers,
41+
download: downloadFn = defaultDownload,
3942
}: {
4043
/**
4144
* The transcription model to use.
@@ -80,6 +83,17 @@ export async function transcribe({
8083
* Only applicable for HTTP-based providers.
8184
*/
8285
headers?: Record<string, string>;
86+
87+
/**
88+
* Custom download function for fetching audio from URLs.
89+
* Use `createDownload()` from `ai` to create a download function with custom size limits.
90+
*
91+
* @default createDownload() (2 GiB limit)
92+
*/
93+
download?: (options: {
94+
url: URL;
95+
abortSignal?: AbortSignal;
96+
}) => Promise<{ data: Uint8Array; mediaType: string | undefined }>;
8397
}): Promise<TranscriptionResult> {
8498
const resolvedModel = resolveTranscriptionModel(model);
8599
if (!resolvedModel) {
@@ -98,7 +112,7 @@ export async function transcribe({
98112

99113
const audioData =
100114
audio instanceof URL
101-
? (await download({ url: audio })).data
115+
? (await downloadFn({ url: audio, abortSignal })).data
102116
: convertDataContentToUint8Array(audio);
103117

104118
const result = await retry(() =>

‎packages/ai/src/util/download/create-download.ts‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
import { download as internalDownload } from './download';
2+
3+
/**
4+
* Creates a download function with configurable options.
5+
*
6+
* @param options - Configuration options for the download function.
7+
* @param options.maxBytes - Maximum allowed download size in bytes. Default: 2 GiB.
8+
* @returns A download function that can be passed to `transcribe()` or `experimental_generateVideo()`.
9+
*/
10+
export function createDownload(options?: { maxBytes?: number }) {
11+
return ({ url, abortSignal }: { url: URL; abortSignal?: AbortSignal }) =>
12+
internalDownload({ url, maxBytes: options?.maxBytes, abortSignal });
13+
}

‎packages/ai/src/util/download/download.test.ts‎

Lines changed: 50 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
import { createTestServer } from '@ai-sdk/test-server/with-vitest';
22
import { DownloadError } from '@ai-sdk/provider-utils';
33
import { download } from './download';
4-
import { describe, it, expect } from 'vitest';
4+
import { describe, it, expect, vi } from 'vitest';
55

66
const server = createTestServer({
77
'http://example.com/file': {},
8+
'http://example.com/large': {},
89
});
910

1011
describe('download', () => {
@@ -66,4 +67,52 @@ describe('download', () => {
6667
expect(error).toBeInstanceOf(DownloadError);
6768
}
6869
});
70+
71+
it('should abort when response exceeds default size limit', async () => {
72+
// Create a response that claims to be larger than 2 GiB
73+
server.urls['http://example.com/large'].response = {
74+
type: 'binary',
75+
headers: {
76+
'content-type': 'application/octet-stream',
77+
'content-length': `${3 * 1024 * 1024 * 1024}`,
78+
},
79+
body: Buffer.from(new Uint8Array(10)),
80+
};
81+
82+
try {
83+
await download({
84+
url: new URL('http://example.com/large'),
85+
});
86+
expect.fail('Expected download to throw');
87+
} catch (error: unknown) {
88+
expect(DownloadError.isInstance(error)).toBe(true);
89+
expect((error as DownloadError).message).toContain(
90+
'exceeded maximum size',
91+
);
92+
}
93+
});
94+
95+
it('should pass abortSignal to fetch', async () => {
96+
const controller = new AbortController();
97+
controller.abort();
98+
99+
server.urls['http://example.com/file'].response = {
100+
type: 'binary',
101+
headers: {
102+
'content-type': 'application/octet-stream',
103+
},
104+
body: Buffer.from(new Uint8Array([1, 2, 3])),
105+
};
106+
107+
try {
108+
await download({
109+
url: new URL('http://example.com/file'),
110+
abortSignal: controller.signal,
111+
});
112+
expect.fail('Expected download to throw');
113+
} catch (error: unknown) {
114+
// The fetch should be aborted, resulting in a DownloadError wrapping an AbortError
115+
expect(DownloadError.isInstance(error)).toBe(true);
116+
}
117+
});
69118
});

‎packages/ai/src/util/download/download.ts‎

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
import { DownloadError } from '@ai-sdk/provider-utils';
1+
import {
2+
DownloadError,
3+
readResponseWithSizeLimit,
4+
DEFAULT_MAX_DOWNLOAD_SIZE,
5+
} from '@ai-sdk/provider-utils';
26
import {
37
withUserAgentSuffix,
48
getRuntimeEnvironmentUserAgent,
@@ -9,11 +13,21 @@ import { VERSION } from '../../version';
913
* Download a file from a URL.
1014
*
1115
* @param url - The URL to download from.
16+
* @param maxBytes - Maximum allowed download size in bytes. Defaults to 100 MiB.
17+
* @param abortSignal - An optional abort signal to cancel the download.
1218
* @returns The downloaded data and media type.
1319
*
14-
* @throws DownloadError if the download fails.
20+
* @throws DownloadError if the download fails or exceeds maxBytes.
1521
*/
16-
export const download = async ({ url }: { url: URL }) => {
22+
export const download = async ({
23+
url,
24+
maxBytes,
25+
abortSignal,
26+
}: {
27+
url: URL;
28+
maxBytes?: number;
29+
abortSignal?: AbortSignal;
30+
}) => {
1731
const urlText = url.toString();
1832
try {
1933
const response = await fetch(urlText, {
@@ -22,6 +36,7 @@ export const download = async ({ url }: { url: URL }) => {
2236
`ai-sdk/${VERSION}`,
2337
getRuntimeEnvironmentUserAgent(),
2438
),
39+
signal: abortSignal,
2540
});
2641

2742
if (!response.ok) {
@@ -32,8 +47,14 @@ export const download = async ({ url }: { url: URL }) => {
3247
});
3348
}
3449

50+
const data = await readResponseWithSizeLimit({
51+
response,
52+
url: urlText,
53+
maxBytes: maxBytes ?? DEFAULT_MAX_DOWNLOAD_SIZE,
54+
});
55+
3556
return {
36-
data: new Uint8Array(await response.arrayBuffer()),
57+
data,
3758
mediaType: response.headers.get('content-type') ?? undefined,
3859
};
3960
} catch (error) {

0 commit comments

Comments
 (0)