Skip to content

chore(bench): add browser benchmark harness via Vitest + Playwright#18261

Open
MathiasWP wants to merge 2 commits into
sveltejs:mainfrom
MathiasWP:bench-browser-harness
Open

chore(bench): add browser benchmark harness via Vitest + Playwright#18261
MathiasWP wants to merge 2 commits into
sveltejs:mainfrom
MathiasWP:bench-browser-harness

Conversation

@MathiasWP

@MathiasWP MathiasWP commented May 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a real-browser performance benchmark harness for code paths where JSDOM disagrees with production engines. Uses Vitest's bench() API in browser mode (headless Chromium via Playwright); tinybench handles warmup, adaptive iteration, and sample collection. The existing JSDOM benchmark suite is unchanged — it stays the right home for reactivity-graph perf (kairo, sbench, derived/effects) where JSDOM is faithful.

Why

JSDOM is a pure-JS DOM implementation. For DOM-heavy operations (creation, removal, attribute writes, layout-triggering work, transitions) it can disagree with real engines by an order of magnitude or more — sometimes inverting the verdict on a perf change. One concrete example that motivated this:

createElement(tag) vs createElementNS(NAMESPACE_HTML, tag). JSDOM showed ≈0% difference (both delegate to the same internal path). Real Chromium shows createElement is ~89% faster — there's a documented fast path in Blink that skips namespace lookup. Measured here, 3 runs, median:

hz per-call
createElementNS('http://www.w3.org/1999/xhtml', tag) (current) ~634k ~1.58 µs
createElement(tag) (proposed fast path) ~1,200k ~0.83 µs

JSDOM hid this win entirely. Without the harness we'd miss real wins like this.

Authoring API

Intentionally minimal — describe what to measure, with optional setup/teardown. No manual warmup, trial collection, or statistics:

import { bench, describe } from 'vitest';
import { mount, unmount } from 'svelte';
import * as $ from 'svelte/internal/client';

describe('html-swap', () => {
	let target, instance, value;
	bench(
		'swap 100-node {@html} fragment',
		() => { $.flush(() => $.set(value, /* alternate */)); },
		{
			setup() {
				target = document.createElement('div');
				document.body.appendChild(target);
				value = $.state(html_a);
				instance = mount(HtmlSwap, { target, props: { value } });
			},
			teardown() { unmount(instance); target.remove(); }
		}
	);
});

Output:

name                                hz       min   max    mean    p99    rme       samples
swap 100-node {@html} fragment   17,770    0.000   2.6    0.056   0.2    ±2.81%      8,885
create_element(htmlTag) ×10     634,678    0.000   2.5    0.0016  0.1    ±3.20%    317,339
  • hz ops/sec
  • rme relative margin of error — anything under ~5% is statistically trustworthy
  • samples tinybench picks adaptively to reach convergence

Running

pnpm bench:browser              # all browser benches
pnpm bench:browser swap         # filter by name fragment

First run downloads headless Chromium (~250 MB, cached in ~/Library/Caches/ms-playwright/). If Playwright complains the executable is missing:

cd packages/svelte && pnpm exec playwright install chromium

What's in scope vs. what isn't

In: the harness itself + two seed benchmarks that demonstrate the workload shape (html_swap, create_element). Both serve as living examples for contributors. The README documents when to add a browser bench vs. when to stick with JSDOM.

Out: any actual perf change to packages/svelte/src/. The harness is generic infrastructure; perf changes are separate PRs.

What's next

  • A bench:browser:compare script (run twice across branches, diff hz). For now: run on main, switch branch, run again, compare.
  • Multi-browser (Firefox, Webkit) once the repo is on Vitest 4.x (separate effort) — Vitest 4's instances array makes this trivial.
  • More seed benchmarks as audit findings are validated.
  • CI integration — out of scope for this PR; local-only for now.

Test plan

  • pnpm bench:browser runs both seed benchmarks and produces stable output (rme < 5%)
  • Existing runtime test suite still passes (5882 tests in runtime-runes / runtime-legacy)
  • No regression in JSDOM benchmarks (this PR doesn't touch them)
  • CI integration (deferred)

🤖 Generated with Claude Code

Adds a real-browser performance benchmark harness for code paths where JSDOM
diverges from production engines (DOM operations, attribute writes, layout-
triggering work, transitions). Uses Vitest's bench() API in browser mode with
Playwright-driven headless Chromium; tinybench handles warmup, adaptive
iteration, and sample collection.

Bench-author API is intentionally minimal — describe what to measure, with
optional setup/teardown:

  bench('label', () => { ...hot loop... }, { setup, teardown });

Each bench reports hz, mean, p99, rme, and sample count. The existing JSDOM
benchmark suite in benchmarking/benchmarks/ is unchanged and remains the right
home for reactivity-graph perf (kairo, sbench, derived/effects), where JSDOM
gives correct relative ordering.

New files:
  - benchmarking/browser/vitest.config.js
  - benchmarking/browser/benches/{html_swap,create_element}.bench.js
  - benchmarking/browser/README.md (when to use, how to write benches)

New dev dep: @vitest/browser ~2.1.9. playwright was already installed.

Run:
  pnpm bench:browser              # all
  pnpm bench:browser swap         # filter

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented May 21, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 2c6a132

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

MathiasWP added a commit to MathiasWP/svelte that referenced this pull request May 21, 2026
Mirrors what the HTML branch does. The trailing `undefined` argument to
`createElementNS` hits a measurably slower path in V8/Blink than the bare
2-arg form (~10% slower for SVG creation in the benches in sveltejs#18261).
SVG/MathML creation is much colder than HTML, but the fix is one line and
makes the wrapper symmetric across both namespaces.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
MathiasWP added a commit to MathiasWP/svelte that referenced this pull request May 21, 2026
The prior pass diverged on first mismatch and then ran LIS over the entire
suffix `[i_start, length)` — fine for big reorders (reverse, shuffle), but
expensive on small ones where most of the tail was actually stable. In a
real-browser benchmark a single far swap regressed ~27% vs the legacy
heuristic because the LIS bookkeeping (Map allocations, the LIS sequence
itself, chain rebuild) dominated for an O(2) move workload.

Walk forward AND backward to identify the stable prefix and suffix in
lockstep, then run LIS only over the unstable middle. Falls back to the
legacy heuristic when the walks expose an add/remove (length mismatch in
the middle), an INERT/offscreen entry, or a non-branch sibling effect —
same eligibility rules as before.

Measured in real Chromium (Chromium 145) via the browser bench harness in
sveltejs#18261:

|             | baseline | prior LIS | new LIS  |     vs heuristic     |
| ----------- | -------: | --------: | -------: | -------------------: |
| swap (far)  |   ~115k  |     ~84k  |  ~129k   |   **+12%** (was -27%) |
| reverse     | ~4,800   |   ~30,900 |  ~33,400 |   **+595%** (was +544%) |
| shuffle     |  ~20,000 |   ~25,000 |  ~28,400 |   **+42%** (was +25%) |

A small refactor pulls the flag eligibility check into `is_eligible_branch`
since it now appears in three places (prefix walk, suffix walk, and the
old-chain walk in the middle pass).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
MathiasWP added a commit to MathiasWP/svelte that referenced this pull request May 21, 2026
These benchmarks belong with the dedicated browser benchmark infrastructure
in sveltejs#18261, not bundled into this PR. The real-browser numbers in the PR
description come from that harness.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rich-Harris pushed a commit that referenced this pull request May 28, 2026
…18262)

## Summary

The current wrapper always calls `document.createElementNS(namespace ??
NAMESPACE_HTML, tag, options)` — even for HTML elements (the >99% case),
and even when `options` would be `undefined`. Two effects compound:

1. **Route HTML elements through `createElement`** — Blink has a fast
path that skips the namespace lookup `createElementNS` always performs.
2. **Omit the trailing `undefined` argument** — V8/Blink take a slower
path for `createElementNS(ns, tag, undefined)` (and `createElement(tag,
undefined)`) than for the bare 2-arg form. This applies symmetrically to
the SVG/MathML branch, where the wrapper now also avoids the `undefined`
3rd arg.

The wrapper dispatches to the fastest call shape for every input —
`{HTML, non-HTML}` × `{with is, without is}` × no `undefined` ever.

## Affects

Every place Svelte constructs a DOM element internally:
`<svelte:element>`, `run_scripts`, the per-component `<style>` injector,
the `{@html}` wrapper, and the `<template>` element used to clone string
templates.

## Numbers

Measured in headless Chromium (Chromium 145) using the browser bench
harness from #18261, 2–3 runs, median. Lower per-call is better; higher
hz is better.

### Raw call shapes (what each shape costs in the browser)

|                                             |       hz | per-call |
| ------------------------------------------- | -------: | -------: |
| `createElement(tag)`                        |  ~1,210k | ~0.83 µs |
| `createElement(tag, undefined)`             |    ~901k | ~1.11 µs |
| `createElementNS(NS_HTML, tag)`             |    ~913k | ~1.10 µs |
| `createElementNS(NS_HTML, tag, undefined)`  |    ~630k | ~1.59 µs |
| `createElement(tag, { is })`                |    ~484k | ~2.07 µs |
| `createElementNS(SVG_NS, tag)`              |    ~333k | ~3.01 µs |
| `createElementNS(SVG_NS, tag, undefined)`   |    ~303k | ~3.30 µs |
| `createElementNS(SVG_NS, tag, { is })`      |    ~245k | ~4.08 µs |

Two stable effects fall out:

- **Trailing `undefined` is consistently slower** than the bare form —
~26% on `createElement`, ~31% on `createElementNS` (HTML), ~10% on
`createElementNS` (SVG).
- **`createElement` skips a namespace lookup** that `createElementNS`
always performs — ~32% delta for equal-shape calls (`createElement(tag)`
vs `createElementNS(NS_HTML, tag)`).

### Per-case impact of this PR

| Case (namespace, `is`) | Old wrapper call | Old hz | New wrapper call
| New hz | Speedup |
| ----------------------------- |
--------------------------------------------- | --------: |
------------------------------ | --------: | ------: |
| HTML, no `is` (dominant path) | `createElementNS(NS_HTML, tag,
undefined)` | ~630k | `createElement(tag)` | ~1,210k | **~92%** (1.92×)
|
| HTML, with `is` | `createElementNS(NS_HTML, tag, { is })` | (slow) |
`createElement(tag, { is })` | ~484k | similar to above (just the
NS-skip) |
| SVG/MathML, no `is` | `createElementNS(ns, tag, undefined)` | ~303k |
`createElementNS(ns, tag)` | ~333k | **~10%** |
| SVG/MathML, with `is` | `createElementNS(ns, tag, { is })` | ~245k |
same | ~245k | no change |

No measurable change in JSDOM (both shapes route through the same JS
implementation there).

The headline gain — **~92% on the dominant HTML-no-`is` path** —
combines both effects roughly equally: dropping the `undefined` (~46%)
and switching to `createElement` (~32%).

Wrapper-vs-raw was also verified: the new wrapper measured to within ±2%
of raw `document.createElement(tag)` for the HTML-no-`is` case, so the
function-call indirection adds no measurable overhead.

See #18261 for the benchmark and methodology.

## Test plan

- [x] All 5881 runtime tests still pass (runtime-runes + runtime-legacy)
- [x] `<svelte:element xmlns={null}>` correctly falls back to HTML
(covered by existing `dynamic-element-dynamic-namespace` test)
- [x] SVG/MathML namespaces still go through `createElementNS`
- [x] Custom-element `is` option still honoured on both branches

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vitest 4 (now on main) changed the browser-mode config API:
- `provider: 'playwright'` string → `playwright()` factory from the new
  `@vitest/browser-playwright` package
- single `browser.name` field → `browser.instances` array (multi-browser ready)

Also:
- bump `@vitest/browser` 2.1.9 → 4.1.8 and align the whole vitest family on
  4.1.8 (browser packages must match vitest core exactly)
- drop the `optimizeDeps.include` workaround — it failed to resolve esm-env/clsx
  as bare specifiers and is unnecessary under Vitest 4's browser dep optimizer
  (cold runs no longer trigger a reload)
- document running benches across multiple engines via `instances`

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant