feat: Streaming file uploads#14775
Conversation
🦋 Changeset detectedLatest commit: c464cfb The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
| { | ||
| getPrototypeOf() { | ||
| // Trick validators into thinking this is a normal File | ||
| return File.prototype; |
There was a problem hiding this comment.
@Rich-Harris not sure the best way to do this part - we can't use actual File objects because that would require the data to already be buffered, and setting __proto__ = File.prototype causes private class member issues.
| if (this.#stream) throw new TypeError('_setup_internal called twice'); | ||
| let cursor = 0; | ||
| let chunk_index = 0; | ||
| this.#stream = new ReadableStream({ |
There was a problem hiding this comment.
This is very complicated and kinda repeats code from get_buffer above, but I tested the performance and the overhead of making a ReadableStream if we just need a little buffer is pretty huge (especially for the common case).
|
@Rich-Harris you said
on your issue - but is that true? Couldn't people do v.pipe(
v.file(),
v.check(async file => {
const text = await file.text();
return text.startsWith("MAGIC");
}, 'File header invalid')
) |
|
just want to say that i'm not ignoring this PR, i'm very excited about it, just absolutely crunched at the moment so haven't had a chance to take a look yet. very soon hopefully! thank you for opening it |
| { | ||
| getPrototypeOf() { | ||
| // Trick validators into thinking this is a normal File | ||
| return File.prototype; | ||
| } | ||
| } |
Rich-Harris
left a comment
There was a problem hiding this comment.
incredible work, thank you! very excited to have this in the framework
* start * pass in form_dat * serialization * start deserializer * finished? deserializer * upload progress via XHR * simplify file offsets, sort small files first * don't cache stream * fix scoped ids * tests * re-add comment * move location & pathname back to headers * skip test on node 18 * changeset * polyfill file for node 18 test * fix refreshes * optimize file offset table * typo * add lazyfile tests * avoid double-sending form keys * remove xhr for next PR * fix requests stalling if files aren't read * Update new-rivers-run.md * encode text before determining length --------- Co-authored-by: Rich Harris <rich.harris@vercel.com>
Closes #14773
Implements a custom binary format used for uploading forms. Also:
sveltekit:fooform data keys, because now ametaobject is transferred along with the form data.sveltekit:fookeys are only set when the form is enhanced anyway, so we already don't have them in the no-js caseBinary format specs:
Example:
Files are sorted smallest to largest, so large files don't have to be streamed in their entirety before we can start reading a small file.
Still needs review:
Need to skip a test on node 18 because it doesn't haveFilesupport, but I don't know if we can mock that with undici or somethingapplication/x-sveltekit-formdataa good content-typePlease don't delete this checklist! Before submitting the PR, please make sure you do the following:
Tests
pnpm testand lint the project withpnpm lintandpnpm checkChangesets
pnpm changesetand following the prompts. Changesets that add features should beminorand those that fix bugs should bepatch. Please prefix changeset messages withfeat:,fix:, orchore:.Edits