Add @wordpress/worker-threads package and migrate @wordpress/vips from @shopify/web-worker#74629
Add @wordpress/worker-threads package and migrate @wordpress/vips from @shopify/web-worker#74629adamsilverstein wants to merge 33 commits into
Conversation
* Add wasmInlinePlugin to inline vips * Add wasm inlining tests * doc blocks
* Add shopify/webworker to babel config * Add "@shopify/web-worker" dependency * Add vips-worker build files
* Add wasmInlinePlugin to inline vips * Add wasm inlining tests * prettier * Tests, try 2 * Improve doc blocks * plugin: match other plugin pattern * Add shopify/webworker to babel config * Add "@shopify/web-worker" dependency * Add vips-worker build files * try: add worker setup test * prettier * correct package order * Fix the build * fix @shopify/web-worker version * stub shopify/web-worker * remove console log
This new package provides utilities for type-safe Web Worker communication using an RPC (Remote Procedure Call) pattern. It allows calling methods on a worker as if they were local async functions. Key features: - wrap(): Creates a proxy for a Worker that exposes its methods as async functions - terminate(): Terminates a wrapped worker and cleans up resources - expose(): Exposes an object's methods to be called from the main thread - Automatic transferable detection for efficient ArrayBuffer transfer - Full TypeScript support with the Remote<T> type This package replaces the need for @shopify/web-worker without requiring webpack or babel, making it compatible with esbuild. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds support for a new wpWorkers field in package.json that declares
worker entry points to be bundled as self-contained JavaScript files.
Workers are bundled with all dependencies included (no externals)
since they need to be fully self-contained when loaded in a
Worker context. The WASM inlining plugin is included to support
packages like @wordpress/vips that embed WASM modules.
Example usage in package.json:
"wpWorkers": {
"./worker": "./src/worker.ts"
}
This will produce worker.mjs (ESM) and optionally worker.cjs (CJS)
in the respective build output directories.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace the @shopify/web-worker dependency with the new @wordpress/worker-threads package for Web Worker RPC communication. Changes: - Update worker.ts to use expose() from @wordpress/worker-threads - Update vips-worker.ts to use wrap() and terminate() - Add wpWorkers field to package.json for worker bundling - Make package ESM-only (remove CJS) due to wasm-vips using top-level await - Update tsconfig.json to reference the new worker-threads package The new implementation maintains the same public API while removing the dependency on webpack/babel-specific tooling. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Remove the @shopify/web-worker package from devDependencies since it's no longer needed after migrating to @wordpress/worker-threads. This removes the dependency on webpack and babel-specific tooling for Web Worker RPC communication. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
Size Change: 0 B Total Size: 3.09 MB ℹ️ View Unchanged
|
Adds comprehensive test coverage for the worker-threads package: - rpc.test.ts: Tests for RPC protocol utilities (message creation, validation, posting) - transferables.test.ts: Tests for transferable object detection (ArrayBuffer, TypedArrays, nested structures, circular refs) - main-thread.test.ts: Tests for wrap() and terminate() functions - worker-thread.test.ts: Tests for expose() function Also fixes transferables.ts to check for MessagePort availability before using instanceof (not available in all environments). Total: 89 unit tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The vips-worker.ts test file was mocking @shopify/web-worker which has been removed. This test is no longer applicable since vips now uses @wordpress/worker-threads. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Unlinked AccountsThe following contributors have not linked their GitHub and WordPress.org accounts: @Copilot, @kleisauke. Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases. If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
There was a problem hiding this comment.
Pull request overview
This PR introduces a new @wordpress/worker-threads package to provide type-safe Web Worker RPC communication and migrates @wordpress/vips away from the archived @shopify/web-worker library. The motivation is to replace an unmaintained dependency that requires webpack/Babel with a lightweight, esbuild-compatible GPL-2.0+ licensed solution.
Changes:
- Created new
@wordpress/worker-threadspackage withwrap(),terminate(), andexpose()APIs for Worker RPC communication - Enhanced build system with
wpWorkersfield support and WASM inlining in@wordpress/build - Migrated
@wordpress/vipsto ESM-only using the new worker-threads package, removing@shopify/web-workerdependency
Reviewed changes
Copilot reviewed 24 out of 26 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/worker-threads/* | New package providing type-safe Worker RPC with automatic transferable detection |
| packages/wp-build/lib/build.mjs | Added wpWorkers field support and WASM inlining plugin for bundling worker scripts |
| packages/vips/src/worker.ts | New worker entry point using expose() from worker-threads |
| packages/vips/src/vips-worker.ts | New wrapper using wrap() and terminate() from worker-threads |
| packages/vips/src/index.ts | Updated WASM loading to use inline base64 data URLs |
| packages/vips/package.json | Made ESM-only, added worker-threads dependency and wpWorkers field |
| test/e2e/specs/editor/various/wasm-inlining.spec.js | Added build verification test for WASM inlining |
| test/unit/jest.config.js | Added stub mapping for shopify/web-worker transformed imports |
| test/unit/config/shopify-web-worker-stub.js | Jest stub for shopify/web-worker transformed paths |
| tsconfig.json | Added worker-threads package reference |
| pr-description.md | PR description documentation file |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@adamsilverstein I've opened a new pull request, #74635, to work on those changes. Once the pull request is ready, I'll request review from you. |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
@adamsilverstein I've opened a new pull request, #74636, to work on those changes. Once the pull request is ready, I'll request review from you. |
|
Yes, the generated code looks quite nice 🙂 It can be simplified at some places:
But otherwise it looks good. However! One alternative we should explore is using the |
@jsnajdr - Excellent, thanks for the feedback. I can work on these changes specifically while also considering the alternative you suggested...
Which exact library are you referring to here? I see https://github.com/developit/remote-ui from from Jason but not recently updated and the Shopify library it is forked from https://github.com/Shopify/remote-dom (renamed to remote dom at some point). |
Since we completely control the postMessage channel between the worker and its creator, there's no need for detailed validation of message structure. Now only checks if the type field matches a valid MessageType. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Instead of recursively traversing all nested objects looking for transferables, now only checks direct elements in the args array (for CALL messages) or the result value (for RESULT messages). This simplification establishes an API contract that transferables (like ArrayBuffer with image data) should be passed as standalone parameters rather than nested within objects. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
I asked Claude to analyze this option and got this response: Analysis:
|
| Feature | @remote-ui/rpc |
@wordpress/worker-threads |
|---|---|---|
| Primary use case | UI extensibility with function callbacks | Binary data processing |
| Function passing across boundary | ✅ Full support with retain/release | ❌ Not supported |
| ArrayBuffer transfer | ❌ Not supported | ✅ Zero-copy transfer |
| TypedArray support | ❌ Not supported | ✅ Auto-detected |
| ImageBitmap transfer | ❌ Not supported | ✅ Auto-detected |
| Stream transfer | ❌ Not supported | ✅ Auto-detected |
| Memory management | Automatic for functions | Automatic for transferables |
| License | MIT | GPL (WordPress) |
| Code size | ~2000+ lines | ~717 lines |
Different Design Goals
@remote-ui/rpc
Designed for Shopify's UI extensibility model where:
- Extension code runs in a sandboxed worker
- UI components need event handlers/callbacks passed across the boundary
- Functions are proxied with automatic retain/release memory management
- Focus is on developer ergonomics for UI interactions
@wordpress/worker-threads
Designed for efficient binary data processing where:
- Large binary buffers (images, WASM memory) transfer between threads
- Zero-copy ArrayBuffer transfer is essential for performance
- Focus is on efficient data transfer, not function proxying
Current Implementation Details
How @wordpress/vips Uses Worker Threads
Main Thread: vipsResizeImage(id, 50MB buffer, ...)
→ wrap() creates CALL message with buffer
→ findTransferables() detects ArrayBuffer
→ postMessage with [buffer] in transfer list (zero-copy)
→ Worker receives buffer ownership
→ Worker processes with WASM (wasm-vips)
→ Worker returns new processed buffer
→ Main thread receives result with zero-copy
Transferables Auto-Detected by @wordpress/worker-threads
From packages/worker-threads/src/transferables.ts:
- ArrayBuffer
- MessagePort
- ImageBitmap
- OffscreenCanvas
- ReadableStream
- WritableStream
- TransformStream
- TypedArrays (Uint8Array, etc.)
What Would Be Required to Use @remote-ui/rpc
To use @remote-ui/rpc for this use case, you would need one of:
-
Add binary data support to
@remote-ui/rpc- Major upstream change, unlikely to be accepted as it doesn't align with the library's design goals -
Wrap
@remote-ui/rpcwith transferable handling - Would require writing the same transferable detection code that already exists in@wordpress/worker-threads, negating any simplicity benefit -
Serialize binary data differently - e.g., Base64 encoding, which would:
- Increase payload size by ~33%
- Require encoding/decoding overhead
- Eliminate zero-copy benefits
- Destroy performance for large images
Code Size Comparison
@wordpress/worker-threads (717 lines total)
57 lines - index.ts (exports)
206 lines - main-thread.ts (wrap, terminate)
152 lines - rpc.ts (message protocol)
140 lines - transferables.ts (auto-detection)
79 lines - types.ts (TypeScript types)
83 lines - worker-thread.ts (expose)
This is already minimal and purpose-built for the use case.
Recommendation
Keep @wordpress/worker-threads for the following reasons:
-
Fundamental incompatibility:
@remote-ui/rpccannot transfer ArrayBuffer/TypedArray, which is the core requirement for WASM image processing -
Already minimal: The current implementation is only 717 lines and well-designed for its purpose
-
Different problems: The libraries solve different problems - function proxying vs. binary data transfer
-
No benefit: Switching would require adding the same transferable handling code, providing no simplification
Alternative Approaches to Explore
If the goal is to reduce custom code or use established libraries, consider:
-
Comlink (Google) - Supports transferables, similar API to worker-threads, MIT license
-
workerize - Simpler but may not have full transferable support
-
threads.js - Full-featured worker library with transferable support
However, the current @wordpress/worker-threads implementation is already simple, well-tested, and fit for purpose.
Sources
Simplified in b98f484
Simplied in 55a5b22 |
|
@swissspidy I went ahead and removed HEIF support in be86016 which I am hopeful won't break anything :) |
@jsnajdr I took another pass at simplifying our library, leveraging comctx https://github.com/molvqingtai/comctx in #74702 This rcp library is MIT licensed and actively maintained (although not incredibly popular). |
This is an interesting question 🙂 There is the But where are the sources? Not easy to find, but turns out they are in the Shopify/remote-dom repo. Just not in the |
|
FWIW, this would break AVIF support. Note that wasm-vips doesn't support HEIC as it includes patent-encumbered HEVC-related logic. |
|
@kleisauke super - thanks for jumping in to clarify that! I will revert that commit - we certainly want AVIF support. I was under the impression from previous discussions that we needed to unbundle the vips heif support because of the licensing issues. |
This would be an additional reason why the original |
|
I don't know how you got to this conclusion. It definitely works in that plugin. IIRC they added ArrayBuffer support at some point, but it was not documented in the readme. |
This was from Claude so possible out of date or just wrong. Thanks for clarifying. |
That's the conclusion of the Claude analysis in #74629 (comment). Which probably relied solely on the (outdated) information in the README. Now when we settled on |
|
Closing in favor of #74785 which leverages comctx. |
What?
Follow on from #74478
Fixes #74352
See #69254
This PR introduces a new
@wordpress/worker-threadspackage that provides type-safe Web Worker RPC communication, and migrates@wordpress/vipsto use it instead of@shopify/web-worker.Why?
The
@shopify/web-workerpackage has several issues for Gutenberg:Alternative libraries were evaluated:
The solution is to create our own lightweight package that:
How?
New
@wordpress/worker-threadspackageProvides three main exports:
wrap(worker)- Creates a proxy for a Worker that exposes its methods as async functionsterminate(remote)- Terminates a wrapped worker and cleans up resourcesexpose(api)- Exposes an object's methods to be called from the main thread (used in worker script)Features:
Remote<T>typeBuild system enhancement
Added
wpWorkersfield support to@wordpress/build. Packages can now declare worker entry points in package.json:{ "wpWorkers": { "./worker": "./src/worker.ts" } }Workers are bundled as self-contained files with all dependencies included.
Migration of
@wordpress/vipsworker.tsto useexpose()from the new packagevips-worker.tsto usewrap()andterminate()@shopify/web-workerdependencyTesting Instructions
npm installto update dependenciesnpm run buildto build all packagespackages/worker-threads/build-module/contains the built filespackages/vips/build-module/contains:index.mjs- Main entry pointvips-worker.mjs- Worker API wrapperworker.mjs- Self-contained worker bundle (~17MB with WASM)Testing the worker functionality
The vips package functionality can be tested after merging to to the
feature/client-side-media-devbranch through the media upload flow:npm run wp-env start