perf: re-encode the HTML AST as struct-of-arrays behind a path-based visitor API#21331
Merged
Conversation
One AST node is now an integer id into parallel typed-array columns (type, flags, offsets, parent/firstChild/lastChild/nextSibling links) plus two side arrays for the string payload and attribute lists — no per-node object and no per-parent children array. All consumer reads go through a new accessor `A` (the same seam as the CSS parser's `A`), and the columns are module-level and reused across parses, with the side arrays released after each processor walk so a parse never pins its source. Tree construction, node structure, offsets and skip behavior are unchanged: the html5lib corpus (8389 tree-construction + tokenizer inputs) serializes byte-identically to the previous object AST, including all skip combinations and fragment modes. Retained AST heap on a 2.4MB attribute-heavy document drops ~56% (45.2MB → 19.7MB), and ~50% (30.9MB → 15.4MB) under HtmlParser's skip config, with parse time neutral to slightly faster.
One attribute is now an integer id into parallel columns (name/value offsets plus one flags byte); an element holds a contiguous run (start + count). The value string is derived from the source by offset on read — only valueless attributes and offset-less adoption-agency clones store an override — and the html5lib serializer name is derived from the adjusted name plus a namespace flag, so per attribute only the interned name pointer is retained. Repeated `<html>`/`<body>` tags merge by re-allocating the element's run; foreign- content adjusts rewrite the run in place. `A` gains scalar attr accessors (attrCount/attrAt/findAttr/attrName/attrValue/attr*Start/attr*End) used by `HtmlParser`; `A.attributes` stays as a materializing test convenience. Byte-identical: the 8389-input html5lib differential corpus (trees, offsets, attribute spans, all skip combos, fragment modes) matches the object AST exactly. Cumulative retained AST heap on the 2.4MB benchmark document vs the object AST, now counting typed-array backing stores (the previous commit's figures missed them — heapUsed excludes ArrayBuffers): 45.2MB → 19.4MB (−57%) full, 30.9MB → 9.9MB (−68%) under HtmlParser's skip config; median parse time ~15% faster in both configs.
Visitors now receive the language's AST accessor as their first argument — `(api, node, parent, ctx)` — for both the CSS and HTML `SourceProcessor`s, so a consumer needs nothing beyond what the walk hands it (no module-level `A` import), and the accessor can later become per-parse state without touching any visitor. The generic visitor typedefs gain a `TApi` parameter; each grammar passes its own accessor (`CssAst` / `HtmlAst` typedefs). Rename the abbreviated HTML accessor methods to full names — `attributeCount` / `attributeAt` / `findAttribute` / `attributeName` / `attributeValue` / `attributeNameStart`-`attributeValueEnd` — plus the `HtmlAttributeRef` / `AttributeRun` typedefs and the tokenizer's local attribute state (the CSS accessor already used full words). No behavior change: the html5lib differential corpus (8389 inputs) still serializes byte-identically, and parse time is unchanged in both parsers.
Drop `HtmlParser`'s import of the accessor object entirely: the element and comment visitors read through the `api` they receive, and the two helpers that inspect attributes outside a visitor body (`attrSourceSpan`, `reconcileScriptTypeAttr`) take the accessor as a parameter. `HtmlParser` now consumes the parser exactly as an external package user would — nothing but `SourceProcessor`, `NodeType`, and what the walk hands each visitor. The accessor export stays: consumers that call `buildHtmlAst` directly (without a walk) — the html5lib serializer and unit-test materializer — can only read refs through it.
Visitors now receive a single `path` argument (Babel's `path` shape): the
language's accessor object with the walk's current position on it —
`path.node`, `path.parent` (null at a root) and `path.skipChildren()` —
and every field-read method defaulting to the current node
(`path.tagName()`, `path.findAttribute("type")`, explicit refs still
accepted: `path.tagName(other)`). The path is one reused module-level object
whose position is rebound before each callback, so unlike Babel there is no
per-node path allocation; it is only valid during the callback, and future
per-node functionality lands on it without touching visitor signatures.
The current position lives in module variables exposed through getters
(avoids self-referential `this` typing in the object literal), and the
per-process() visitor-context closure is gone.
The HTML walk is now iterative over the SoA link columns (firstChild /
nextSibling descend, the parent column ascends; template content fragments
gained a parent link for the ascent), with no recursion at all. This removes
the recursion depth limit: ~100k-deep nesting previously threw
`RangeError: Maximum call stack size exceeded` in the walk, and now
completes. The tree-link accessor `parent(n)` was renamed `parentOf(n)`
to free `path.parent` for the position field.
Measured on interleaved A/B runs against the previous commit: parse+walk
time and retained memory are unchanged in both parsers (deltas < 1.5%,
within noise); the html5lib differential corpus (8389 inputs) still
serializes byte-identically.
🦋 Changeset detectedLatest commit: 2e4fa23 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
This PR is packaged and the instant preview is available (e6fb547). Install it locally:
npm i -D webpack@https://pkg.pr.new/webpack@e6fb547
yarn add -D webpack@https://pkg.pr.new/webpack@e6fb547
pnpm add -D webpack@https://pkg.pr.new/webpack@e6fb547 |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #21331 +/- ##
==========================================
+ Coverage 92.50% 92.63% +0.13%
==========================================
Files 594 594
Lines 65387 65758 +371
Branches 18228 18269 +41
==========================================
+ Hits 60483 60917 +434
+ Misses 4904 4841 -63
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR optimizes webpack’s experimental HTML (and related CSS) parsing pipeline by switching the HTML AST to a struct-of-arrays (SoA) representation and migrating both HTML/CSS visitors to a single, reusable Babel-style path argument, reducing allocations and enabling an iterative walk to avoid deep-recursion stack overflows.
Changes:
- Replaced the HTML AST’s object-per-node layout with an SoA backend accessed via a new exported accessor
A. - Updated HTML and CSS visitor APIs to receive a single
pathaccessor object (includingskipChildren()), aligning both grammars. - Updated/added tests and a changeset to validate SoA access and visitor behavior (including html5lib serialization).
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/walkCssTokensParser.unittest.js | Updates tests to use the new CSS path visitor argument instead of node helpers. |
| test/html5lib.spectest.js | Serializes the HTML tree via accessor A and SoA sibling/child links for corpus verification. |
| test/buildHtmlAst.unittest.js | Materializes the SoA tree into plain objects for assertions; exercises the full accessor surface. |
| lib/util/SourceProcessor.js | Updates the generic visitor types/docs to the new single-argument path visitor shape. |
| lib/html/syntax.js | Implements the SoA HTML AST, accessor A, and an iterative tree walk behind SourceProcessor. |
| lib/html/HtmlParser.js | Migrates HTML processing to the path/accessor-based visitor API and attribute ref reads. |
| lib/css/syntax.js | Migrates the CSS walker to the single path visitor API and adds node/parent bindings. |
| lib/css/CssParser.js | Updates CSS parser visitors to consume the new path argument and skipChildren() behavior. |
| .changeset/html-parser-soa-ast.md | Adds a patch changeset describing the memory reduction improvement. |
Comments suppressed due to low confidence (1)
lib/html/HtmlParser.js:1353
attrSourceSpanassumes the attribute has valid source offsets (attributeNameStart(...) >= 0). For cloned/synthesized attributes (offsets -1), this would slice from a negative index and produce incorrect copied attribute text. Guard against offset-less attributes before copying them intocopyableAttrsText.
for (const copyableName of COPYABLE_SIBLING_ATTRS) {
const copyableAttr = path.findAttribute(copyableName);
if (copyableAttr !== 0) {
copyableAttrsText += attrSourceSpan(
path,
source,
copyableAttr
);
if (copyableName === "crossorigin") {
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+83
to
+90
| default: | ||
| // Text / Comment | ||
| return { | ||
| type: /** @type {typeof NodeType.Text} */ (type), | ||
| data: A.data(ref), | ||
| start: A.start(ref), | ||
| end: A.end(ref) | ||
| }; |
Comment on lines
+11
to
+14
| * A visitor receives a single `path` argument (the Babel `path` shape): the | ||
| * language's AST accessor with the current position on it — `path.node`, | ||
| * `path.parent` (0 = none) — plus `path.skipChildren()` (enter only) to stop | ||
| * the walk descending, and every field-read method (which defaults to the |
Comment on lines
+8306
to
+8308
| doctypePublicId() { | ||
| return _hDocPub; | ||
| }, |
Comment on lines
+8312
to
+8314
| doctypeSystemId() { | ||
| return _hDocSys; | ||
| }, |
…eview The merge with main brought #21329's `<base href>` support, written against the old node-object visitor API; adapt it to the path accessor (`path.namespace()` / `path.findAttribute("href")` / `path.attributeValue(ref)`). Also address Copilot review comments: repair the unittest's `HtmlNodeRef` typedef (a rename artifact referenced an undefined `MatNodeRef`), widen the materializer's Text/Comment type cast to the union it actually handles, let `doctypePublicId`/`doctypeSystemId` accept an optional node for call-shape uniformity, and correct the `path.parent` doc (null at a root, not 0).
Merging this PR will improve performance by 84.51%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Memory | benchmark "lodash", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' |
850.9 KB | 130.9 KB | ×6.5 |
| ⚡ | Memory | benchmark "future-defaults", scenario '{"name":"mode-production","mode":"production"}' |
11 MB | 7.5 MB | +45.62% |
| ⚡ | Memory | benchmark "wasm-modules-sync", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' |
190.2 KB | 131.7 KB | +44.44% |
| ⚡ | Memory | benchmark "asset-modules-source", scenario '{"name":"mode-development-rebuild","mode":"development","watch":true}' |
245.9 KB | 192 KB | +28.06% |
| ⚡ | Memory | benchmark "wasm-modules-sync", scenario '{"name":"mode-production","mode":"production"}' |
7.9 MB | 6.5 MB | +22.15% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing perf/html-parser-soa-ast (2e4fa23) with main (5ce1c22)
40342bc to
e603447
Compare
…ion edge cases Fix the TS2345 in the foster-parenting test (cast inside the map callback instead of annotating the parameter against the widened element type). Add unit coverage for the new path accessor surface (HTML and CSS) and for tree-construction paths previously only reached by the html5lib corpus: column growth, skipped-comment text merging, template foster parenting, colgroup recovery, select/hr/selectedcontent handling, Noah's Ark, adoption agency attribute cloning, foreign-content integration points and attribute adjustment, repeated-body attribute merging, frameset replacement, and fragment-context parsing.
| @@ -341,12 +340,10 @@ describe("walkCssTokens — SourceProcessor", () => { | |||
| new SourceProcessor() | |||
Contributor
Types CoverageCoverage after merging perf/html-parser-soa-ast into main will be
Coverage Report |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The experimental HTML parser held one object per AST node plus a children array per parent, dominating parse-time heap. Nodes and attributes now live in parallel typed-array columns (an integer id per node/attribute) behind an accessor, cutting retained AST memory 57–68% (45.2→19.4 MB full tree, 30.9→9.9 MB under
HtmlParser's skip config on a 2.4 MB document) and parse+walk CPU ~17%, measured interleaved against main. Visitors (HTML and CSS) now receive a single Babel-stylepathargument (one reused object, no per-node allocation), and the HTML walk is iterative over the link columns, fixing aRangeErrorstack overflow on ~100k-deep nesting; trees serialize byte-identically to the previous implementation across the full html5lib corpus (8389 inputs, all skip combinations and fragment modes). Refs #21323.What kind of change does this PR introduce?
perf
Did you add tests for your changes?
Yes —
test/buildHtmlAst.unittest.jsandtest/html5lib.spectest.jsnow exercise the new accessor/path API (including a materialization helper that reads every accessor), and the existing conformance, skip-parity, andconfigCases/{html,css}suites cover the representation change.Does this PR introduce a breaking change?
No — the visitor/accessor API of the experimental HTML/CSS parsers is webpack-internal, and build output is unchanged.
If relevant, what needs to be documented once your changes are merged or what have you already documented?
n/a
Use of AI
This PR was implemented with Claude Code under my direction: it performed the mechanical SoA conversion, API migration, and benchmarking, gated by a differential harness asserting byte-identical serialization against the previous implementation over the html5lib corpus; I reviewed the design and the resulting diff.
Generated by Claude Code