feat(ext/web): transferable {Readable,Writable,Transform}Stream by devsnek · Pull Request #31126 · denoland/deno

devsnek · 2025-10-29T07:05:41Z

https://streams.spec.whatwg.org/#rs-transfer
https://streams.spec.whatwg.org/#ws-transfer
https://streams.spec.whatwg.org/#ts-transfer

remaining test failures are due to our DOMException not correctly being serializable and can be solved in a followup.

// example

const INDEX_HTML = Deno.readTextFileSync("./index.html");
const worker = new Worker("./the_algorithm.js", { type: "module" });
Deno.serve(async (req) => {
  if (req.method === "POST" && req.path === "/the-algorithm") {
    const { port1, port2 } = new MessageChannel();
    worker.postMessage({ stream: req.body, port: port1 }, { transfer: [req.body, port1] });
    const res = await new Promise((resolve) => {
      port1.onmessage = (e) => resolve(e.data);
    });
    return new Response(res);
  }
  if (req.path === "/") {
    return new Response(INDEX_HTML, { "content-type": "text/html" });
  }
  return new Response(null, { status: 404 });
});

bartlomieju · 2025-12-03T11:53:20Z

tests/wpt/runner/expectation.json

+        "cancel should be propagated to the original",
+        "cancel should abort a pending read()",


I'm a bit worried that these tests are currently failing - have you investigated why they fail?

They fails because of DOMException not being serialized properly.

bartlomieju · 2025-12-03T11:53:26Z

tests/wpt/runner/expectation.json

+      ],
+      "worker.html": true,
+      "writable-stream.html": [
+        "writing a unclonable object should error the stream"


coderabbitai · 2025-12-08T10:51:03Z

Walkthrough

The pull request implements cross-realm stream transfer support. Changes span the streams module (06_streams.js) to add MessagePort-based transferable support for ReadableStream, WritableStream, and TransformStream with appropriate host object branding and transfer lifecycle management. The message port layer (13_message_port.js and message_port.rs) extends serialization/deserialization to handle multiple resources simultaneously. Supporting changes include Pyenv entries in .gitignore and test infrastructure updates for dynamic shim generation and expanded test expectations.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 57.89% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly matches the PR's primary objective: implementing transferability for ReadableStream, WritableStream, and TransformStream.
Description check	✅ Passed	The description is directly related to the changeset, providing spec references and a practical example of the new transferable streams functionality.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch transfer-readable-stream

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

bartlomieju

LGTM, let's go

coderabbitai

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

ext/web/message_port.rs (1)
261-267: Missing check for MultiResource in self-transfer validation.

The op_message_port_post_message function checks if a Resource is being transferred to itself, but it doesn't handle MultiResource. If a MultiResource contains the message port's RID, the self-transfer check will be bypassed.

Consider extending the check to handle MultiResource:
   for js_transferable in &data.transferables {
     if let JsTransferable::Resource(_name, id) = js_transferable
       && *id == rid
     {
       return Err(MessagePortError::TransferSelf);
     }
+    if let JsTransferable::MultiResource(_name, ids) = js_transferable {
+      for id in ids {
+        if *id == rid {
+          return Err(MessagePortError::TransferSelf);
+        }
+      }
+    }
   }

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 642f2a4 and ae586eb.

📒 Files selected for processing (6)

.gitignore (1 hunks)
ext/web/06_streams.js (6 hunks)
ext/web/13_message_port.js (2 hunks)
ext/web/message_port.rs (4 hunks)
tests/wpt/runner/expectation.json (1 hunks)
tests/wpt/runner/runner.ts (2 hunks)

🧰 Additional context used

📓 Path-based instructions (2)

**/*.rs

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.rs: For debugging Rust code, set breakpoints in IDE debuggers (VS Code with rust-analyzer, IntelliJ IDEA) or use lldb directly
Use eprintln!() or dbg!() macros for debug prints in Rust code

Files:

ext/web/message_port.rs

⚙️ CodeRabbit configuration file

Don't worry about coverage of Rust docstrings. Don't be nitpicky about it. Leave it to the author's judgement if such a documentation is necessary.

Files:

ext/web/message_port.rs

**/*.{ts,tsx,js}

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.{ts,tsx,js}: For JavaScript runtime debugging, enable V8 inspector with --inspect-brk flag and connect Chrome DevTools to chrome://inspect
Use console.log() for debug prints in JavaScript runtime code

Files:

tests/wpt/runner/runner.ts
ext/web/13_message_port.js
ext/web/06_streams.js

🧠 Learnings (1)

📚 Learning: 2025-11-24T16:19:37.808Z

Learnt from: CR
Repo: denoland/deno PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-11-24T16:19:37.808Z
Learning: Applies to tests/specs/**/{__test__.jsonc,*.out} : Output assertions in spec tests should use `__test__.jsonc` inline fields or `.out` files with special matching syntax: `[WILDCARD]`, `[WILDLINE]`, `[WILDCHAR]`, `[WILDCHARS(n)]`, `[UNORDERED_START]`/`[UNORDERED_END]`, and `[# comment]`

Applied to files:

tests/wpt/runner/expectation.json

🧬 Code graph analysis (1)

ext/web/message_port.rs (1)

ext/web/13_message_port.js (2)

rid (469-469)

transferables (371-371)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)

GitHub Check: test debug windows-x86_64
GitHub Check: test debug linux-x86_64
GitHub Check: test debug macos-x86_64
GitHub Check: test debug macos-aarch64
GitHub Check: lint debug linux-x86_64
GitHub Check: lint debug windows-x86_64
GitHub Check: build libs
GitHub Check: lint debug macos-x86_64

🔇 Additional comments (20)

.gitignore (1)

48-50: LGTM!

Standard pyenv configuration for ignoring local Python version files.

ext/web/13_message_port.js (2)

388-394: LGTM!

The multiResource deserialization correctly mirrors the resource case and passes multiple rids to the receive method.

470-480: LGTM!

The conditional logic correctly handles both single and multi-resource transfers by checking the type of the returned rid. The implementation branches appropriately and maintains consistent data structures.

tests/wpt/runner/runner.ts (1)

192-219: Clarify whether per-script MessageChannel mock isolation is intentional.

The shim code is re-executed before each script (lines 242–265), creating a new MessageChannel and overriding globalThis.postMessage for each script execution. While port1 and port2 are block-scoped and discarded after each script runs, the postMessage override is global and persists, meaning only the last script's override remains active. If test scripts depend on shared mock channel state or expect postMessage to route to a consistent channel instance, this per-script recreation should be refactored to run the shim setup only once per test.

tests/wpt/runner/expectation.json (1)

5374-5394: Test expectations align with PR scope and known issues.

The mixed success/failure states make sense—tests are fully passing where stream transfer works, and partially/failing where they involve serialization of DOMException or non-serializable objects (the known issue deferred to a follow-up). The specific failing test cases all relate directly to error handling or non-serializable chunks, which confirms the expectations reflect the current implementation state accurately.

To ensure the test case names and expectations are accurate, please verify that:

The test names match the actual test definitions in the WPT files (e.g., "transferring a non-serializable chunk should error both sides" exists in readable-stream.html)

No additional tests should be passing given that DOMException serialization is not yet fixed

ext/web/message_port.rs (4)

47-51: LGTM - MultiResource variant addition.

The new MultiResource variant follows the same pattern as the existing Resource variant, appropriately extending the enum to support multiple transferred resources.

179-185: LGTM - JsTransferable MultiResource variant.

The serialization format mirrors the existing Resource variant structure with a Vec<ResourceId> for multiple resources.

202-213: LGTM - Deserialization logic for MultiResource.

The deserialization correctly iterates over resource IDs, takes each resource from the table, transfers them, and aggregates into a Vec. Error handling is consistent with the single-resource path.

234-240: LGTM - Serialization logic for MultiResource.

The serialization correctly maps over transferred resources, receives each one, and adds them to the resource table. The iterator chain is clean and idiomatic.

ext/web/06_streams.js (11)

5161-5162: LGTM - Host object branding for ReadableStream.

Adding [core.hostObjectBrand] enables the core transfer mechanism to identify and handle ReadableStream instances.

6170-6171: LGTM - Host object branding for TransformStream.

Consistent with ReadableStream branding pattern.

6182-6185: LGTM - Brand placeholder path for TransformStream.

This enables internal construction of TransformStream instances during transfer receiving steps, matching the pattern used by ReadableStream and WritableStream.

6386-6387: LGTM - Host object branding for WritableStream.

Consistent with the branding pattern used for other stream types.

6754-6769: LGTM - Message packing utilities.

Clean helper functions for cross-realm message protocol. The error handling wrapper properly re-throws after notifying the other side.

6775-6820: LGTM - Cross-realm readable stream setup.

The implementation correctly:

Sets up message handlers for chunk/close/error messages

Handles messageerror events with proper cleanup

Uses appropriate pull/cancel algorithms that communicate via the port

Calls port.start() to begin receiving messages

6901-6912: LGTM - ReadableStream transfer steps.

Properly checks for locked state, sets up cross-realm writable, and pipes the stream. The promise is correctly marked as handled.

6928-6939: LGTM - WritableStream transfer steps.

Mirrors the readable pattern - checks lock, sets up cross-realm readable, and pipes.

6956-6971: LGTM - TransformStream transfer steps.

Correctly checks both readable and writable sides for locks before transferring. Delegates to the individual stream transfer steps.

6987-7029: LGTM - Transferable resource registrations.

All three stream types are properly registered with send/receive handlers that:

Create MessageChannel for communication

Call appropriate transfer step functions

Use the MessagePort transferable resource for the underlying port transfer

TransformStream correctly uses two channels (one per direction)

6826-6895: The backpressure gating logic is intentional and working as designed.

Lines 6855-6858 are unreachable on the first write invocation, but this is by design: the initial backpressurePromise acts as a gate that blocks the first write until the remote signals readiness via a "pull" message (line 6833). On subsequent writes, if backpressurePromise has been set to undefined, these lines execute to allow writes to proceed immediately. The code correctly implements a cross-realm backpressure mechanism and does not contain the issue suggested.

ext/web/06_streams.js

ericf · 2025-12-10T23:43:18Z

It seems that the Transferable type declaration wasn't updated to allow this without TypeScript compile errors:

deno/cli/tsc/dts/lib.deno_web.d.ts

Lines 1217 to 1257 in ed55de8

    
           type Transferable = MessagePort | ArrayBuffer; 
        
           /** 
        
            * Options that control structured serialization operations such as 
        
            * `structuredClone(value, options)` and `MessagePort.postMessage(message, options)`. 
        
            * 
        
            * The optional `transfer` array lists {@link Transferable} objects whose 
        
            * underlying resources should be moved (transferred) to the receiving side 
        
            * instead of being cloned. After a successful transfer: 
        
            * 
        
            * - For an `ArrayBuffer`, the original buffer becomes neutered (its 
        
            *   `byteLength` is set to `0`). 
        
            * - For a `MessagePort`, the port becomes unusable on the sending side and 
        
            *   future events will arrive only on the transferred port at the receiver. 
        
            * 
        
            * Validation rules: 
        
            * - Each transferable may appear only once in the `transfer` list. 
        
            * - A `MessagePort` cannot be listed together with its counterpart port from 
        
            *   the same `MessageChannel` in the same transfer operation. 
        
            * - Duplicate or otherwise invalid entries will cause a `DataCloneError` 
        
            *   `DOMException` to be thrown. 
        
            * 
        
            * Transferring improves performance for large binary data and allows moving 
        
            * communication endpoints without copying. 
        
            * 
        
            * @example 
        
            * ```ts 
        
            * // Transferring an ArrayBuffer (zero-copy for large data) 
        
            * const buffer = new ArrayBuffer(16); 
        
            * const cloned = structuredClone(buffer, { transfer: [buffer] }); 
        
            * 
        
            * // After transfer, the original buffer is neutered 
        
            * console.log(buffer.byteLength); // 0 
        
            * console.log(cloned.byteLength); // 16 
        
            * 
        
            * @category Platform 
        
            */ 
        
           interface StructuredSerializeOptions { 
        
             /** List of transferable objects whose ownership is moved instead of cloned. */ 
        
             transfer?: Transferable[]; 
        
           }

type Transferable = MessagePort | ArrayBuffer;

bartlomieju · 2025-12-11T20:11:22Z

@ericf can you give an example program that fails to type-check properly?

ericf · 2025-12-11T22:06:51Z

@bartlomieju the example used in the description of this PR:

// example

const INDEX_HTML = Deno.readTextFileSync("./index.html");
const worker = new Worker("./the_algorithm.js", { type: "module" });
Deno.serve(async (req) => {
  if (req.method === "POST" && req.path === "/the-algorithm") {
    const { port1, port2 } = new MessageChannel();
    worker.postMessage({ stream: req.body, port: port1 }, { transfer: [req.body, port1] });
    const res = await new Promise((resolve) => {
      port1.onmessage = (e) => resolve(e.data);
    });
    return new Response(res);
  }
  if (req.path === "/") {
    return new Response(INDEX_HTML, { "content-type": "text/html" });
  }
  return new Response(null, { status: 404 });
});

TS2769 [ERROR]: No overload matches this call.
  Overload 1 of 2, '(message: any, transfer: Transferable[]): void', gave the following error.
    Object literal may only specify known properties, and 'transfer' does not exist in type 'Transferable[]'.
  Overload 2 of 2, '(message: any, options?: StructuredSerializeOptions | undefined): void', gave the following error.
    Type 'ReadableStream<Uint8Array<ArrayBuffer>> | null' is not assignable to type 'Transferable'.
      Type 'null' is not assignable to type 'Transferable'.
    worker.postMessage({ stream: req.body, port: port1 }, { transfer: [req.body, port1] });
           ~~~~~~~~~~~
    at file:///Users/eferraiuolo/Scratch/transferable-streams/test1.ts:8:12

And a very simple example:

➜  transferable-streams cat test2.ts 
const stream = ReadableStream.from(["foo", "bar"]);
const worker = new Worker("./worker.js", { type: "module" });
worker.postMessage({ stream }, [stream]);
➜  transferable-streams deno --version
deno 2.6.0 (stable, release, aarch64-apple-darwin)
v8 14.2.231.17-rusty
typescript 5.9.2
➜  transferable-streams deno check test2.ts 
Check test2.ts
TS2769 [ERROR]: No overload matches this call.
  Overload 1 of 2, '(message: any, transfer: Transferable[]): void', gave the following error.
    Type 'ReadableStream<string>' is not assignable to type 'Transferable'.
  Overload 2 of 2, '(message: any, options?: StructuredSerializeOptions | undefined): void', gave the following error.
    Type 'ReadableStream<string>[]' has no properties in common with type 'StructuredSerializeOptions'.
worker.postMessage({ stream }, [stream]);
       ~~~~~~~~~~~
    at file:///Users/eferraiuolo/Scratch/transferable-streams/test2.ts:3:8

error: Type checking failed.

#31126 (comment)

## Summary - Enables `structuredClone()` and `postMessage()` for `DOMException` objects using the cloneable resource registry from #32672 - Serializes `message`, `name`, and `stack`; `code` is derived from `name` on deserialization - Supersedes #31156 Ref #31126 (comment) Closes #31983 --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

devsnek force-pushed the transfer-readable-stream branch from 04fdba5 to 5e24cd8 Compare October 31, 2025 15:36

devsnek force-pushed the transfer-readable-stream branch from 5e24cd8 to 6b59191 Compare November 13, 2025 02:31

devsnek added 2 commits November 14, 2025 14:12

feat(ext/web): transferable {Readable,Writable,Transform}Stream

0b24450

update

c6021d8

devsnek force-pushed the transfer-readable-stream branch 2 times, most recently from fd3bafd to a648831 Compare November 14, 2025 07:38

fix?

ca149e3

devsnek force-pushed the transfer-readable-stream branch from a648831 to ca149e3 Compare November 15, 2025 03:16

bartlomieju reviewed Dec 3, 2025

View reviewed changes

Merge branch 'main' into transfer-readable-stream

ae586eb

bartlomieju approved these changes Dec 8, 2025

View reviewed changes

coderabbitai bot reviewed Dec 8, 2025

View reviewed changes

ext/web/06_streams.js Show resolved Hide resolved

bartlomieju enabled auto-merge (squash) December 8, 2025 11:02

bartlomieju merged commit a15cafe into main Dec 8, 2025
36 of 37 checks passed

bartlomieju deleted the transfer-readable-stream branch December 8, 2025 12:31

devsnek mentioned this pull request Dec 12, 2025

fix(cli): add streams as transferable types #31586

Merged

bartlomieju pushed a commit that referenced this pull request Dec 12, 2025

fix(cli): add streams as transferable types (#31586)

b27665f

#31126 (comment)

bartlomieju mentioned this pull request Mar 12, 2026

fix(web): support structuredClone for DOMException #32675

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ext/web): transferable {Readable,Writable,Transform}Stream#31126

feat(ext/web): transferable {Readable,Writable,Transform}Stream#31126
bartlomieju merged 4 commits intomainfrom
transfer-readable-stream

devsnek commented Oct 29, 2025 •

edited

Loading

Uh oh!

bartlomieju Dec 3, 2025

Uh oh!

bartlomieju Dec 8, 2025

Uh oh!

bartlomieju Dec 3, 2025

Uh oh!

coderabbitai bot commented Dec 8, 2025 •

edited

Loading

Uh oh!

bartlomieju left a comment

Uh oh!

coderabbitai bot left a comment

Uh oh!

Uh oh!

Uh oh!

ericf commented Dec 10, 2025

Uh oh!

bartlomieju commented Dec 11, 2025

Uh oh!

ericf commented Dec 11, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		"cancel should be propagated to the original",
		"cancel should abort a pending read()",

Conversation

devsnek commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bartlomieju Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

bartlomieju Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

bartlomieju Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Pre-merge checks and finishing touches

Uh oh!

bartlomieju left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ericf commented Dec 10, 2025

Uh oh!

bartlomieju commented Dec 11, 2025

Uh oh!

ericf commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

devsnek commented Oct 29, 2025 •

edited

Loading

coderabbitai bot commented Dec 8, 2025 •

edited

Loading

ericf commented Dec 11, 2025 •

edited

Loading