Load vips library with new wordpress/worker-threads by adamsilverstein · Pull Request #74785 · WordPress/gutenberg

adamsilverstein · 2026-01-20T18:08:54Z

What?

Follow on from #74478

This PR introduces a new @wordpress/worker-threads package that provides type-safe Web Worker RPC communication, and migrates @wordpress/vips to use it instead of @shopify/web-worker.

Why?

The @shopify/web-worker package has several issues for Gutenberg:

Unmaintained - The repository is archived
Requires webpack and Babel - Gutenberg has moved to esbuild
Build-time magic - Uses webpack chunk naming and Babel transforms that don't work with our build system

Alternative libraries were evaluated:

Comlink - Uses Apache 2.0 license, incompatible with GPL-2.0+ for combined work in WordPress core
threads.js - Unmaintained
workerize - Unmaintained, requires webpack

The solution is to create our own lightweight package that:

Works with esbuild (no webpack/babel required)
Is GPL-2.0+ licensed
Provides similar ergonomics to Comlink/Shopify web-worker

How?

New `@wordpress/worker-threads` package

Provides three main exports:

wrap(worker) - Creates a proxy for a Worker that exposes its methods as async functions
terminate(remote) - Terminates a wrapped worker and cleans up resources
expose(api) - Exposes an object's methods to be called from the main thread (used in worker script)

Features:

Automatic transferable detection for efficient ArrayBuffer transfer (zero-copy)
Full TypeScript support with Remote<T> type
~1.5KB bundle size
Promise-based RPC with proper error propagation

The `comctx` dependency

The @wordpress/worker-threads package uses comctx as its underlying RPC mechanism. Comctx is a lightweight cross-context RPC library that provides:

defineProxy API - Creates provider/injector pairs for RPC communication. On the worker side, provide(adapter) exposes methods; on the main thread, inject(adapter) creates a proxy to call those methods.
Adapter-based architecture - Separates communication logic from transport. The adapter tells comctx how to send/receive messages without restricting the specific method. This makes it flexible for any messaging context (Web Workers, iframes, extensions, etc.).
Native transferable support - No internal serialization; uses structured cloning with native support for transferable objects like ArrayBuffer for zero-copy transfers.
MIT License - Compatible with GPL-2.0+ for WordPress core, unlike Comlink (Apache 2.0)

Why comctx over other options:

vs Comlink: Comlink uses Apache 2.0 license which is incompatible with GPL-2.0+ for combined work. Comctx uses MIT license.
vs custom implementation: Comctx provides battle-tested RPC primitives (message correlation, error propagation, transferable detection) that would be complex to reimplement correctly.

The @wordpress/worker-threads package wraps comctx with WordPress-specific adapters (WorkerInjectAdapter for main thread, WorkerProvideAdapter for worker) and adds termination handling.

Build system enhancement

Added wpWorkers field support to @wordpress/build. Packages can now declare worker entry points in package.json:

{
  "wpWorkers": {
    "./worker": "./src/worker.ts"
  }
}

Workers are bundled as self-contained files with all dependencies included.

Migration of `@wordpress/vips`

Updated worker.ts to use expose() from the new package
Updated vips-worker.ts to use wrap() and terminate()
Made package ESM-only (wasm-vips uses top-level await which is incompatible with CJS)
Removed @shopify/web-worker dependency

Testing Instructions

Run npm install to update dependencies
Run npm run build to build all packages
Verify the build succeeds without errors
Check that packages/worker-threads/build-module/ contains the built files
Check that packages/vips/build-module/ contains:
- index.mjs - Main entry point
- vips-worker.mjs - Worker API wrapper
- worker.mjs - Self-contained worker bundle (~17MB with WASM)

Testing the worker functionality

The vips package functionality can be tested in feature/client-side-media-dev (in #74568) through the media upload flow:

Start wp-env: npm run wp-env start
Upload an image to the media library
Image processing (resize, format conversion) should work via the worker

* Add wasmInlinePlugin to inline vips * Add wasm inlining tests * doc blocks

* Add shopify/webworker to babel config * Add "@shopify/web-worker" dependency * Add vips-worker build files

* Add wasmInlinePlugin to inline vips * Add wasm inlining tests * prettier * Tests, try 2 * Improve doc blocks * plugin: match other plugin pattern * Add shopify/webworker to babel config * Add "@shopify/web-worker" dependency * Add vips-worker build files * try: add worker setup test * prettier * correct package order * Fix the build * fix @shopify/web-worker version * stub shopify/web-worker * remove console log

This new package provides utilities for type-safe Web Worker communication using an RPC (Remote Procedure Call) pattern. It allows calling methods on a worker as if they were local async functions. Key features: - wrap(): Creates a proxy for a Worker that exposes its methods as async functions - terminate(): Terminates a wrapped worker and cleans up resources - expose(): Exposes an object's methods to be called from the main thread - Automatic transferable detection for efficient ArrayBuffer transfer - Full TypeScript support with the Remote<T> type This package replaces the need for @shopify/web-worker without requiring webpack or babel, making it compatible with esbuild. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Adds support for a new wpWorkers field in package.json that declares worker entry points to be bundled as self-contained JavaScript files. Workers are bundled with all dependencies included (no externals) since they need to be fully self-contained when loaded in a Worker context. The WASM inlining plugin is included to support packages like @wordpress/vips that embed WASM modules. Example usage in package.json: "wpWorkers": { "./worker": "./src/worker.ts" } This will produce worker.mjs (ESM) and optionally worker.cjs (CJS) in the respective build output directories. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Replace the @shopify/web-worker dependency with the new @wordpress/worker-threads package for Web Worker RPC communication. Changes: - Update worker.ts to use expose() from @wordpress/worker-threads - Update vips-worker.ts to use wrap() and terminate() - Add wpWorkers field to package.json for worker bundling - Make package ESM-only (remove CJS) due to wasm-vips using top-level await - Update tsconfig.json to reference the new worker-threads package The new implementation maintains the same public API while removing the dependency on webpack/babel-specific tooling. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Adds comprehensive test coverage for the worker-threads package: - rpc.test.ts: Tests for RPC protocol utilities (message creation, validation, posting) - transferables.test.ts: Tests for transferable object detection (ArrayBuffer, TypedArrays, nested structures, circular refs) - main-thread.test.ts: Tests for wrap() and terminate() functions - worker-thread.test.ts: Tests for expose() function Also fixes transferables.ts to check for MessagePort availability before using instanceof (not available in all environments). Total: 89 unit tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

The vips-worker.ts test file was mocking @shopify/web-worker which has been removed. This test is no longer applicable since vips now uses @wordpress/worker-threads. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…74636) * Initial plan * Remove unnecessary async keywords from test callbacks Co-authored-by: adamsilverstein <2676022+adamsilverstein@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: adamsilverstein <2676022+adamsilverstein@users.noreply.github.com>

…74635) * Initial plan * Optimize transferables duplicate detection using Set Co-authored-by: adamsilverstein <2676022+adamsilverstein@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: adamsilverstein <2676022+adamsilverstein@users.noreply.github.com>

…compilation The vips package's vips-worker.ts imports from worker-code.ts, which is a generated file that's gitignored. During CI builds, after clean:packages runs, the worker-code.ts file is deleted. TypeScript compilation happens before wp-build generates the actual worker code, causing the build to fail with: "Cannot find module './worker-code'" This adds a step before TypeScript compilation that generates placeholder worker-code.ts files for packages that define wpWorkers in their package.json. The actual worker code is still generated later by wp-build, overwriting the placeholder. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

improve docs to explain build process Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Since we completely control the postMessage channel between the worker and its creator, there's no need for detailed validation of message structure. Now only checks if the type field matches a valid MessageType. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Instead of recursively traversing all nested objects looking for transferables, now only checks direct elements in the args array (for CALL messages) or the result value (for RESULT messages). This simplification establishes an API contract that transferables (like ArrayBuffer with image data) should be passed as standalone parameters rather than nested within objects. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Replace custom RPC implementation with the worker-rpc library. This significantly reduces code complexity by leveraging an established library for message-passing between main and worker threads. Changes: - Add worker-rpc dependency to package.json - Simplify main-thread.ts to use RpcProvider - Simplify worker-thread.ts to use RpcProvider - Remove custom RPC message types from types.ts - Delete rpc.ts (replaced by worker-rpc) - Simplify findTransferables to accept arrays directly - Update all tests for new implementation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Switch from worker-rpc to comctx for RPC communication - Use comctx's defineProxy with adapters for main/worker threads - Remove transferables.ts since comctx handles transfers automatically with the transfer: true option - Update tests to use addEventListener pattern that comctx expects - Add comctx to jest transformIgnorePatterns Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

jsnajdr · 2026-01-28T10:25:38Z

+ * Generate placeholder files for worker code in packages that define wpWorkers.
+ * This must run before TypeScript compilation because vips-worker.ts imports worker-code.ts.
+ */
+async function generateWorkerPlaceholders() {


This code is a clear candidate to be moved into its own script, somewhere in bin/packages/generate-worker-placeholders.mjs.

The bin/build.mjs script is a simple process runner that executes a series of other processes with exec, it doesn't do anything just by itself.

Ok, makes sense - I will do that.

addressed in f31636f

adamsilverstein · 2026-01-28T14:21:34Z

This comment by @manzoorwanijk is very relevant also for this PR: #74651 (comment) The PR adds another 250 lines to the build.mjs file. Can we split the worker build code into another source file?

Sure, good suggestion - will do.

adamsilverstein · 2026-01-28T14:31:34Z

Some time ago we've been fixing this Gutenberg issue...

Thanks for the additional context @jsnajdr - that helps me understand the issues raised here.

…js module

adamsilverstein · 2026-01-29T15:17:48Z

And... I broke the build with the recent changes :( working on fixing that now.

adamsilverstein · 2026-01-29T15:32:16Z

This comment by @manzoorwanijk is very relevant also for this PR: #74651 (comment) The PR adds another 250 lines to the build.mjs file. Can we split the worker build code into another source file?

Done in 17ff565

adamsilverstein · 2026-01-29T15:40:14Z

And... I broke the build with the recent changes :( working on fixing that now.

Oh, I accidentally committed the build files in d985c3b. I reverted that and added them to gitignore in c5d847a to avoid that happening again.

The build should be fixed now and PR ready for another review. I also merged all recent changes to the feature branch for testing (since this PR doesn't do anything testable other than build) - #74568

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

adamsilverstein · 2026-01-29T15:54:35Z

@jsnajdr @manzoorwanijk - this is ready for review again!

adamsilverstein · 2026-01-29T18:59:48Z

cc: @adamziel if you have any feedback here

simison · 2026-01-30T11:11:12Z

Would be great if you can update changelog files as well (ref), thanks in advance!

jsnajdr

I think this is good to go, I don't see any blockers. Thanks for this pioneering work 🙂

…ew worker-threads

adamsilverstein · 2026-01-30T15:46:31Z

Would be great if you can update changelog files as well (ref), thanks in advance!

Added in f40162d

adamsilverstein · 2026-01-30T15:48:59Z

I think this is good to go, I don't see any blockers. Thanks for this pioneering work 🙂

Excellent 🎉 🎉 🎉 thank you for your support here and sharing your tips for the best approach. I'm going to merge this once tests pass which will unblock the remaining client side media PRs. As we get further into testing, we can always refine or improve what we have here now.

stokesman · 2026-01-30T19:50:53Z

+		// Step 2.5: Generate worker placeholders
+		// This must happen before TypeScript compilation because some packages
+		// (like vips) have source files that import from generated worker-code.ts
+		await exec( 'node', [
+			'./bin/packages/generate-worker-placeholders.mjs',
+		] );


I could be wrong, but I think this may need to happen in dev.mjs too. I’m here because after a fresh pull from trunk, the dev script failed due to the missing file.

Oh, let me review - I assume you mean running npm run dev?

I wasn't able to reproduce in trunk, maybe something in my local. Going to try a fresh repo checkout.

I’m here because after a fresh pull from trunk, the dev script failed due to the missing file.

@stokesman thanks for reporting this... does this go away if you run npm run build first?

I was able to reproduce on a fresh checkout, opening a PR to fix!

@stokesman I created a follow up PR to fix this - #75104

nickchomey · 2026-02-11T11:48:05Z

Great work on this! I just wanted to flag coincident as an alternative worth considering, since it wasn't listed in the evaluated options.

Why it might be interesting for WordPress:

MIT licensed - same GPL-2.0+ compatibility as comctx
uses sharedarraybuffer and atomics rather than postmessage, for much higher performance
since wasm-vips already requires cross-origin isolation headers, coincident would automatically unlock synchronous worker -> main calls, including window object/Dom access in that context, without code changes. This could be useful if future worker-based features ever need synchronous access to main-thread APIs
also works without SharedArrayBuffer - coincident falls back to async-only postMessage-based RPC when COOP/COEP headers aren't present, so it doesn't impose infrastructure requirements on hosting environments
Maintained by Andrea Giammarchi - well-known in the JS ecosystem, active maintainer
Built-in FFI batching - ffi.assign, ffi.gather, ffi.query, ffi.evaluate reduce roundtrips for chatty interactions

For the current vips use case, comctx is probably the better choice since the communication pattern is simple fire-and-forget-async. But if @wordpress/worker-threads grows to cover more complex scenarios, the sync upgrade path and FFI batching could become relevant. Just wanted to make sure coincident was on the radar as an option before it's too late.

adamsilverstein · 2026-02-11T12:31:49Z

Thanks for the supportive note and reference @nickchomey - I hadn't come across coincident Sounds lioke it would offer some distinct advantages especially if we widen how we are using worker threads. Will keep it in mind as we iterate. cc: @adamziel

nickchomey · 2026-02-11T12:37:32Z

I also just came across this library. It's surely far more comprehensive than desired here, but there might be some useful ideas.

https://github.com/kunkunsh/kkrpc

adamsilverstein and others added 30 commits January 20, 2026 09:58

74421 configure build pipeline (#74426)

58821c5

* Add wasmInlinePlugin to inline vips * Add wasm inlining tests * doc blocks

74422 implement blob loader (#74468)

d13031b

* Add shopify/webworker to babel config * Add "@shopify/web-worker" dependency * Add vips-worker build files

Try: revert scripts package changes

1cb4814

Try: remove babel config change

8301259

Remove obsolete vips-worker test file

0308d35

The vips-worker.ts test file was mocking @shopify/web-worker which has been removed. This test is no longer applicable since vips now uses @wordpress/worker-threads. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fine tuning

117b4c2

prettier

edd8e47

remove pr description

4651fc1

Update packages/vips/src/vips-worker.ts

35334ec

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

prettier

658a5ea

remove shopify traces

a204117

build vips-worker

a9ca712

Fix build for tests

60b08d6

prettier

2170cee

lint fix

9ff1398

Update packages/wp-build/lib/build.mjs

2fca3d8

improve docs to explain build process Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

remove heif support

8a7281e

Remove vips-heif.wasm inlining test

bdbef08

jsnajdr reviewed Jan 28, 2026

View reviewed changes

adamsilverstein added 3 commits January 28, 2026 09:49

Break out placeholder generation

17ff565

remove stray diff

d985c3b

Extract worker build code from build.mjs into separate worker-build.m…

f31636f

…js module

Remove inadvertantly committed build files and add to gitignore

c5d847a

linter fixes

e58dca6

Fix prettier formatting in build.mjs

d1b015b

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

adamsilverstein requested a review from jsnajdr January 29, 2026 15:54

jsnajdr approved these changes Jan 30, 2026

View reviewed changes

Update changelogs for vips, wp-build packages and add changelog for n…

f40162d

…ew worker-threads

adamsilverstein merged commit aec311f into trunk Jan 30, 2026
39 checks passed

adamsilverstein deleted the feature/74352-rebased branch January 30, 2026 16:54

github-project-automation Bot moved this from 🔎 Needs Review to ✅ Done in WordPress 7.0 Editor Tasks Jan 30, 2026

stokesman reviewed Jan 30, 2026

View reviewed changes

adamsilverstein mentioned this pull request Jan 30, 2026

Run generate-worker-placeholders script in dev #75104

Merged

jsnajdr mentioned this pull request Feb 3, 2026

Build: Add vendorScripts config to build packages from node_modules #74343

Merged

adamsilverstein mentioned this pull request Feb 3, 2026

Client Side Media iteration for WordPress 7.0 #74333

Closed

Conversation

adamsilverstein commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

How?

New @wordpress/worker-threads package

The comctx dependency

Build system enhancement

Migration of @wordpress/vips

Testing Instructions

Testing the worker functionality

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamsilverstein commented Jan 28, 2026

Uh oh!

adamsilverstein commented Jan 28, 2026

Uh oh!

adamsilverstein commented Jan 29, 2026

Uh oh!

adamsilverstein commented Jan 29, 2026

Uh oh!

adamsilverstein commented Jan 29, 2026

Uh oh!

adamsilverstein commented Jan 29, 2026

Uh oh!

adamsilverstein commented Jan 29, 2026

Uh oh!

simison commented Jan 30, 2026

Uh oh!

jsnajdr left a comment

Choose a reason for hiding this comment

Uh oh!

adamsilverstein commented Jan 30, 2026

Uh oh!

adamsilverstein commented Jan 30, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nickchomey commented Feb 11, 2026

Uh oh!

adamsilverstein commented Feb 11, 2026

Uh oh!

nickchomey commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

adamsilverstein commented Jan 20, 2026 •

edited

Loading

New `@wordpress/worker-threads` package

The `comctx` dependency

Migration of `@wordpress/vips`