test: add full WHATWG lexer integration tests#20974
Conversation
|
There was a problem hiding this comment.
Pull request overview
Adds a new configCases integration test under test/configCases/html/full-lexer-integration/ to exercise the HTML module pipeline with a “kitchen sink” HTML fixture and verify the emitted HTML string via Jest snapshots.
Changes:
- Added a new HTML config case enabling
experiments.html. - Added a large
page.htmlfixture intended to hit many tokenizer/parser states (comments, entities, RCDATA/RAWTEXT/SCRIPT_DATA, CDATA marker, etc.). - Added a Jest snapshot asserting the exported HTML module output.
Reviewed changes
Copilot reviewed 4 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/configCases/html/full-lexer-integration/webpack.config.js | Enables HTML experiment for the new config case. |
| test/configCases/html/full-lexer-integration/index.js | Imports the HTML module and snapshots the exported string. |
| test/configCases/html/full-lexer-integration/page.html | “Kitchen sink” HTML fixture used by the integration test. |
| test/configCases/html/full-lexer-integration/snapshots/ConfigTest.snap | Snapshot for the non-cache test suite. |
| test/configCases/html/full-lexer-integration/simple.html | Fixture referenced by links in page.html (currently empty). |
| test/configCases/html/full-lexer-integration/image.png | Fixture referenced by <img> tags (currently empty). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is
❌ Your patch check has failed because the patch coverage (64.00%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #20974 +/- ##
==========================================
- Coverage 90.92% 90.90% -0.02%
==========================================
Files 573 573
Lines 58610 58654 +44
Branches 15762 15784 +22
==========================================
+ Hits 53290 53321 +31
- Misses 5320 5333 +13
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Merging this PR will degrade performance by 41.5%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
|
Hii @alexander-akait have updated the snapshots to pass the Basic test can u please re-run it thanks :)) |
|
Can you take a look at copilot review, especially img src resolution |
sorry @alexander-akait I didn’t look deeply at this earlier. It was the bug in the CDATA/RAWTEXT handling inside walkHtmlTokens.js,. The lexer was not returning cleanly to DATA after parser-consumed tag bodies, which affected image src resolution after CDATA/rawtext sections and which the same was caught by the integration test i have fixed it and run the basic test and everything is passing green now thanks ;)) |
|
basic test failed, we need to fix it |
|
@alexander-akait i have ran |
|
@alexander-akait All the required CI are green now thanks :)) |
|
This PR is packaged and the instant preview is available (490684a). Install it locally:
npm i -D webpack@https://pkg.pr.new/webpack@490684a
yarn add -D webpack@https://pkg.pr.new/webpack@490684a
pnpm add -D webpack@https://pkg.pr.new/webpack@490684a |
Summary
This PR introduces a comprehensive end-to-end integration test (
test/configCases/html/full-lexer-integration/) to validate the new HTML Tokenizer.While previous PRs implemented isolated unit tests for individual states, this PR proves that Webpack's compiler can successfully parse a complex, real-world HTML document utilizing all 80 WHATWG lexer states without crashing, dropping characters, or failing AST compilation.
Key highlights:
page.htmlfixture acts as a "kitchen-sink" stress test, explicitly triggering<!DOCTYPE>,<![CDATA[...]]>, deeply nested structures, self-closing void elements, and multi-line comments.RAWTEXT,RCDATA, andSCRIPT_DATAstates by successfully parsing<style>,<textarea>, and<script>blocks.<script>and<style>tags withtype="text/plain"to safely bypass Webpack's downstream JS/CSS chunk extraction pipeline, guaranteeing we strictly test the HTML lexer's stability without triggering circular dependency errors in the AST.part of #536
What kind of change does this PR introduce?
test
Did you add tests for your changes?
Yes. Added a new Webpack config case
test/configCases/html/full-lexer-integration/containing the integration harness, thepage.htmlfixture, and the resulting Jest snapshot.Does this PR introduce a breaking change?
No.
If relevant, what needs to be documented once your changes are merged or what have you already documented?
n/a
Use of AI
partial