Skip to content

Docs: Add client-side media processing documentation#75895

Open
adamsilverstein wants to merge 22 commits into
trunkfrom
75111-documentation
Open

Docs: Add client-side media processing documentation#75895
adamsilverstein wants to merge 22 commits into
trunkfrom
75111-documentation

Conversation

@adamsilverstein

@adamsilverstein adamsilverstein commented Feb 25, 2026

Copy link
Copy Markdown
Member

Summary

Adds comprehensive documentation for the Client-Side Media Processing feature shipping in WordPress 7.1. The underlying infrastructure landed during the 7.0 cycle when the feature graduated from a Gutenberg experiment; the 7.1 release adds HEIC support, AVIF end-to-end uploads, performance improvements, and finishes hardening before the public release.

  • Architecture doc (docs/explanations/architecture/client-side-media-architecture.md): How the system works — the three-package split (@wordpress/upload-media, @wordpress/vips, @wordpress/media-utils), upload queue and operation chaining, WebAssembly/libvips pipeline, Cross-Origin Isolation and SharedArrayBuffer requirements, browser feature detection, and graceful fallback to server-side processing.
  • How-to guide (docs/how-to-guides/client-side-media.md): Practical guide for plugin/theme developers — disabling the feature, customizing image processing via existing WordPress filters, handling compatibility with metadata hooks, and debugging.
  • Editor filters reference (docs/reference-guides/filters/editor-filters.md): New "Client-side media processing" section documenting the wp_client_side_media_processing_enabled PHP filter, the generate_sub_sizes and convert_format REST API parameters, and the existing core filters that the client-side pipeline respects (big_image_size_threshold, image_editor_output_format, image_save_progressive, wp_image_maybe_exif_rotate).
  • Make/Core draft post (docs/contributors/client-side-media-make-core-post.md): Announcement post draft for make.wordpress.org/core, including a FAQ covering the bandwidth-vs-CPU framing, the security model, browser fallback behavior, hook compatibility, and the Firefox/Safari exclusion rationale.
  • Manifest and TOC updates: docs/manifest.json and docs/toc.json regenerated to surface the new pages in the handbook.

Closes #75111

Test plan

  • Verify documentation renders correctly in the handbook
  • Review technical accuracy of architecture explanations
  • Confirm code examples are correct and follow WordPress coding standards
  • Check all internal cross-references and links resolve

adamsilverstein and others added 3 commits February 24, 2026 21:07
Document the upcoming capability detection areas including
credentialless iframe support, device memory, network
conditions, and Web Worker checks. Notes that browsers
without credentialless support will have client-side media
disabled by default, with a server-side filter opt-in.
Architecture explanation, developer how-to guide, editor
filters reference, and make/core announcement draft for the
client-side media processing feature shipping in WP 7.0.
- Fix wasm-vips links to point to canonical kleisauke/wasm-vips repo
- Add wp_ prefix to filter name in feature-detection docblock
- Use wp.apiFetch instead of hardcoded /wp-json/ path in sideload example

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@adamsilverstein adamsilverstein self-assigned this Feb 26, 2026
@adamsilverstein adamsilverstein added [Type] Task Issues or PRs that have been broken down into an individual action to take [Feature] Client Side Media Media processing in the browser with WASM labels Feb 26, 2026
@github-project-automation github-project-automation Bot moved this to 🔎 Needs Review in WordPress 7.0 Editor Tasks Feb 26, 2026
@adamsilverstein adamsilverstein moved this from 🔎 Needs Review to 🏗️ In Progress in WordPress 7.0 Editor Tasks Feb 26, 2026
@adamsilverstein adamsilverstein added the [Status] In Progress Tracking issues with work in progress label Feb 26, 2026
Document the finalize REST endpoint, editor.media.imageQuality
JS filter, and server-side hook compatibility added in PR #74913
across architecture, how-to, and Make Core post docs. Also fixes
a typo ("eprocessing") in the Make Core post.
@adamsilverstein adamsilverstein marked this pull request as ready for review March 1, 2026 05:33
@github-actions

github-actions Bot commented Mar 1, 2026

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: adamsilverstein <adamsilverstein@git.wordpress.org>
Co-authored-by: juanmaguitar <juanmaguitar@git.wordpress.org>
Co-authored-by: aaronjorbin <jorbin@git.wordpress.org>
Co-authored-by: westonruter <westonruter@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

- Replace COEP/COOP cross-origin isolation with Document-Isolation-Policy
  (Chromium 137+ only; Firefox/Safari not supported)
- Remove outdated "not yet wired through" note on imageQuality hook
- Add client_side_supported_mime_types PHP filter documentation
- Add replace_file parameter to sideload endpoint docs
- Update browser compatibility table and troubleshooting section
- Update feature detection table to match current implementation
Adopt trunk's feature-detection docblock listing — it accurately
reflects implemented checks (hardware concurrency). Drop the
credentialless-iframe bullet since no such check exists in the
code.
@github-actions

github-actions Bot commented Apr 23, 2026

Copy link
Copy Markdown

Flaky tests detected in 47c6a81.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/26815437455
📝 Reported issues:

Comment thread docs/contributors/client-side-media-make-core-post.md Outdated
adamsilverstein and others added 6 commits April 28, 2026 14:07
The feature shipping target moved from 7.0 to 7.1 (per #76756). Bring
the architecture explanation and how-to guide up to date with the
post-7.0-cycle state of the code:

- Reposition introductions around 7.1 with 7.0 as the groundwork cycle.
- HEIC/HEIF: document the canvas-based fallback path (createImageBitmap,
  HTMLImageElement+OffscreenCanvas, HEIC container parsing + WebCodecs
  VideoDecoder), JPEG companion file, and isHeicCanvasSupported() gate.
- AVIF: note the wp_prevent_unsupported_mime_type_uploads bypass when
  generate_sub_sizes=false (#76371).
- Batch thumbnail generation via image.copyMemory() / thumbnailImage()
  (#76979).
- Sub-size deduplication via image_size: string|string[] on the sideload
  route (#77036).
- Single VIPS instance via promise caching (#76780).
- Build-output trimming: remove vips-jxl.wasm (#76639), skip non-min
  worker (#76615), skip WASM-inlined sourcemaps (#75993).
- COI: <img> excluded from cross-origin attribute injection (#76618).
- Loosened feature-detection thresholds: 2 cores, 3g (#76616).
- Post-saving lock fix: Save Draft now respects the lock (#76973).
- convert_format declared as boolean on sideload route (#77565).
- Fix incorrect REST index settings list (only image_sizes and
  image_size_threshold are exposed).
- Remove references to client_side_supported_mime_types filter
  (PR #76549 was closed unmerged); state the supported MIME set is
  fixed at CLIENT_SIDE_SUPPORTED_MIME_TYPES.

Refs #75111.
The feature ships in 7.1 instead of 7.0 (per #76756). Reframe the post
as a 7.1 announcement with a dedicated 'What's new in 7.1' section
listing HEIC support, end-to-end AVIF uploads, batch thumbnail
generation, sub-size deduplication, single VIPS instance, save lock
fix, loosened device requirements, <img> COI exclusion, smaller build
output, and the convert_format boolean coercion fix. Each item is
linked to its merged PR.

Move the original 7.0 cycle deliverables under 'What changed during
7.0' so the historical context stays available for the announcement
audience.

Update technical overview to use the actual function name
(gutenberg_is_client_side_media_processing_enabled), reflect that
DIP replaces COEP/COOP, and remove the inaccurate note that the
imageQuality hook isn't wired up.

Drop the make-believe client_side_supported_mime_types filter; PR
#76549 was closed unmerged. State that the supported MIME set is
fixed at CLIENT_SIDE_SUPPORTED_MIME_TYPES.

Update browser compatibility table to reflect the DIP-based gating
(Chromium 137+, no Firefox/Safari) and add a feature detection
thresholds section.
The handbook generator derives slugs from filename basenames, so
docs/how-to-guides/client-side-media.md and
docs/explanations/architecture/client-side-media.md collided on the
slug "client-side-media" and broke docs:gen in CI.

Rename the architecture file to client-side-media-architecture.md and
update all internal cross-references plus the public-URL references in
the Make/Core draft post.
Run docs:build to add manifest entries for the new how-to guide and
architecture explanation pages, fixing the check-local-changes CI step.
Adds a Frequently Asked Questions section addressing common public
misconceptions raised on issue #76756 — bandwidth-vs-CPU/RAM framing,
the 'never trust the client' security objection, fallback behavior,
hook compatibility, format conversion, and browser support.
@westonruter

Copy link
Copy Markdown
Member

@adamsilverstein this can be removed from the 7.0 project board, right?

@adamsilverstein

Copy link
Copy Markdown
Member Author

@adamsilverstein this can be removed from the 7.0 project board, right?

yes, moved this and all other client-side media tickets that weren't already done to the 7.1 board.

Merge the 7.0 and 7.1 sections of the Make Core post into a single
"What's included" summary so first-time readers (who only see the
feature in 7.1) aren't asked to track which version landed what.

Replace inaccurate "Faster uploads" framing with "Faster downloads"
benefit — the client uploads the original plus every sub-size, so
total bytes go up; the actual win is libvips' better compression
producing smaller files for site visitors.

Add a Mermaid flow diagram in the architecture doc showing both
client- and server-side paths converging at wp_generate_attachment_metadata.

Add a HEIC browser-support note on first mention (Chromium on macOS
and Windows-with-HEVC, Safari on macOS; Firefox unsupported), and
correct the WebCodecs HEVC fallback to say Chromium and cover
Windows via the Microsoft HEVC Video Extension.

Strip out "how we got here" PR-link references throughout the
architecture doc — sub-size dedup, <img> CORS exclusion, convert_format
boolean, image_size array, post-save lock — these were process
artifacts, not what the feature does today. Drop the WASM bundle
size figures and the CPU threshold tuning history for the same
reason. Tighten the WASM module loading bullets.
…client-side media docs

The filter actually fires twice during a client-side upload — once
with context 'create' during the initial upload (WP core's
wp_generate_attachment_metadata() runs even when
intermediate_image_sizes_advanced is filtered to empty) and again
with 'update' at finalize. Earlier wording suggested the filter
was bypassed initially and only ran at finalize, which is wrong.

The double-fire pattern matches how WordPress already handles
big-image uploads on the server, where sub-size generation is
deferred and triggers a second 'update' pass. Frame the contract
that way across all three docs and tell plugin authors to be
idempotent — there's nothing new for them to handle that they
haven't already accommodated for big-image uploads.

Other corrections in the same pass:

- Soften the "security model is unchanged" FAQ in the make/core
  post: AVIF uploads from the client-side path do bypass
  wp_prevent_unsupported_mime_type_uploads when generate_sub_sizes
  is false, which is a deliberate exception worth surfacing.
- Fix HEVC fallback wording in the how-to guide (Chrome 107+ on
  macOS → Chromium 107+ via the platform codec, covering Windows
  HEVC Video Extension as well).
- Note Safari's HEIC canvas fallback in browser-support tables
  across all three docs (architecture and how-to guide both said
  flat "Not supported").
- Add packages/vips/src/worker.ts to the source-files list — it's
  the actual entry point loader.ts dynamically imports.
- Align image quality figure to "0–1, default 0.82" everywhere
  (architecture doc had "82%").
- Drop residual historical PR link in the how-to guide
  troubleshooting table (#76973 Save Draft fix).
- Drop the "graduated from a Gutenberg experiment" version-history
  framing from the architecture doc and make/core post.
- Drop the historical 7.0 iteration link from the make/core post
  related-tracking-issues list.
…ype set

Add a "See also" footer to the architecture doc pointing to the
how-to guide and the editor-filters reference, so plugin authors
who land on the architecture page have an obvious next step toward
actionable guidance.

Add a callout in the editor-filters reference noting that
CLIENT_SIDE_SUPPORTED_MIME_TYPES is fixed and not exposed via a
filter. Plugin authors looking for a "supported types" filter will
land in the filter reference first; documenting the absence
prevents a hunt that the make/core post and how-to guide already
mention but the reference itself didn't.
- Split the long lede sentence in the make/core post explaining
  the finalize 'update' pass into two sentences for readability.
- Cross-reference the security-FAQ MIME-bypass detail from the
  AVIF Key benefits bullet so readers connecting "accepts AVIF"
  to "how" can find the explanation in one click.
- Update the architecture upload-flow Mermaid label to
  "wp_generate_attachment_metadata context: 'update'" so attentive
  readers don't infer the filter only fires at finalize.
- Move the dense parenthetical in the how-to guide finalize
  section into a trailing sentence, keeping the cross-link to
  the server-side compatibility section.
…ision

Reflect four client-side media PRs merged after these docs were last revised,
keeping the architecture and how-to guides in sync with current behavior:

- #76707: document the disabled libvips operation cache (Cache.max(0)) and the
  50-operation worker recycling that bound WASM linear-memory growth, plus the
  globalThis.__vipsDebug hook for capturing the suppressed vips output.
- #74903: document the sideload endpoint's image-dimension validation and its
  400 error responses (rest_upload_dimension_mismatch and friends).
- #78359: note that the finalize endpoint returns the refreshed attachment,
  which the editor uses to keep the block URL (and front-end srcset) in sync.
- #78038: clarify that sub-size filenames derive from the original basename and
  only the scaled full-size copy carries the -scaled suffix.
Add a Requirements section near the top of the how-to guide covering
browser support, required browser capabilities, and device/network
thresholds (>2GB RAM, >=2 cores, faster-than-2g). The system thresholds
were previously only discoverable by scanning the troubleshooting table.

Remove the now-redundant trailing Browser compatibility section, which
the new section subsumes.
Comment thread docs/contributors/client-side-media-make-core-post.md Outdated
Comment thread docs/contributors/client-side-media-make-core-post.md Outdated
Comment thread docs/contributors/client-side-media-make-core-post.md Outdated
Comment thread docs/contributors/client-side-media-make-core-post.md
Comment thread docs/contributors/client-side-media-make-core-post.md Outdated
Comment thread docs/contributors/client-side-media-make-core-post.md Outdated
Comment thread docs/contributors/client-side-media-make-core-post.md Outdated
Comment thread docs/contributors/client-side-media-make-core-post.md
Co-authored-by: Aaron Jorbin <aaronjorbin@users.noreply.github.com>
Comment thread docs/contributors/client-side-media-make-core-post.md Outdated
Comment thread docs/contributors/client-side-media-make-core-post.md Outdated
adamsilverstein and others added 2 commits June 2, 2026 12:55
- Note automatic retry with exponential backoff for resilient uploads (#76765)
- Reframe cross-origin isolation as a capability (SharedArrayBuffer) extenders can use, not an implementation detail
- Clarify that Firefox/Safari are unsupported for the full pipeline, with a pointer to the compatibility section
- Add publication-date browser-version context and a Chrome Platform Status link for tracking Document-Isolation-Policy support
- Fix the Safari compatibility row wording and stray parenthesis
- Merge the duplicated 'Feature detection thresholds' and 'Known limitations' sections into one
- Explain why the AVIF MIME-check exception carries minimal security risk
@adamsilverstein

adamsilverstein commented Jun 2, 2026

Copy link
Copy Markdown
Member Author

Thanks for the thorough review @aaronjorbin and @westonruter. I pushed 47c6a81 addressing the feedback on the make/core draft:

  • Firefox/Safari framing (HEIC bullet) — clarified that Firefox doesn't support in-browser HEIC decode specifically, and added a pointer noting the full WASM pipeline is Chromium-only with a link to the Browser compatibility section.
  • Network retry — the "More resilient uploads" benefit now mentions automatic retry with exponential backoff (from Upload Media: Add retry with exponential backoff and network resilience #76765).
  • Cross-origin isolation reading as an implementation detail — reframed the "What's included" entry around what it enables: SharedArrayBuffer (and high-res timers) in the editor, which plugins can leverage for their own multithreaded/WASM features, with a note that DIP avoids COOP/COEP's page-wide constraints.
  • Browser version context — added a note that Chrome/Edge have supported Document-Isolation-Policy since v137 (mid-2025) and current stable is well past that, so most users already qualify.
  • Where to check support after publishing — noted DIP isn't tracked on caniuse yet and linked the Chrome Platform Status entry as the reliable source.
  • Safari row — fixed the wording and the stray parenthesis.
  • Duplicated sections — merged "Feature detection thresholds" and "Known limitations" into a single "Feature detection and limitations" table (added a Browser row so it's self-contained).
  • Risk of the AVIF MIME-check exception — added a paragraph explaining why the risk is minimal: the relaxed check is a processing-capability gate (not a security control), it's only bypassed when the browser already produced every sub-size, and capability/MIME-allowlist/standard upload validation all still apply.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

[Feature] Client Side Media Media processing in the browser with WASM [Status] In Progress Tracking issues with work in progress [Type] Task Issues or PRs that have been broken down into an individual action to take

Projects

Status: 🔎 Needs Review

Development

Successfully merging this pull request may close these issues.

Add thorough documentation for Client Side Media

4 participants