perf: use webpack-sources 3.4 buffers() to avoid concat on hashing#20897
Merged
alexander-akait merged 3 commits intoApr 30, 2026
Conversation
🦋 Changeset detectedLatest commit: 1d511f3 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
This PR is packaged and the instant preview is available (9e91057). Install it locally:
npm i -D webpack@https://pkg.pr.new/webpack@9e91057
yarn add -D webpack@https://pkg.pr.new/webpack@9e91057
pnpm add -D webpack@https://pkg.pr.new/webpack@9e91057 |
Codecov Report❌ Patch coverage is
❌ Your patch status has failed because the patch coverage (46.03%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #20897 +/- ##
===========================================
- Coverage 91.29% 80.12% -11.17%
===========================================
Files 562 525 -37
Lines 55615 53586 -2029
Branches 14705 14190 -515
===========================================
- Hits 50772 42938 -7834
- Misses 4843 10648 +5805
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates webpack’s hashing paths to take advantage of webpack-sources@^3.4.1’s new Source.prototype.buffers() API, aiming to reduce peak memory usage and improve performance when hashing large concatenated/modified sources.
Changes:
- Bump
webpack-sourcesfrom^3.3.4to^3.4.1. - Add
updateHashFromSource(hash, source)helper to hashSourcecontent viabuffers()when available (fallback tobuffer()). - Rework
RealContentHashPlugindedup/hash input flow to operate on buffer chunk arrays without concatenating.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| yarn.lock | Updates lockfile entry for webpack-sources@^3.4.1. |
| package.json | Bumps webpack-sources dependency to ^3.4.1. |
| lib/util/source.js | Introduces updateHashFromSource helper and exports it. |
| lib/asset/AssetGenerator.js | Uses updateHashFromSource for full content hashing. |
| lib/dependencies/CssIcssExportDependency.js | Uses updateHashFromSource for CSS [contenthash]-based ident hashing. |
| lib/optimize/RealContentHashPlugin.js | Avoids forced concatenation by deduping/processing Source buffer chunks. |
| .changeset/webpack-sources-buffers.md | Adds a changeset entry documenting the perf/memory improvement. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
502
to
506
| const assetsContent = []; | ||
| for (const chunks of uniqueChunkArrays) { | ||
| for (const c of chunks) assetsContent.push(c); | ||
| } | ||
| let newHash = hooks.updateHash.call(assetsContent, oldHash); |
There was a problem hiding this comment.
assetsContent passed into hooks.updateHash used to contain one Buffer per unique asset source (deduped). This change flattens chunk arrays returned by source.buffers() so a single asset may contribute multiple entries, which changes the observable hook contract (e.g. existing updateHash taps that rely on input.length === 1 for a single unique source will stop working). To keep backwards compatibility, consider preserving the per-source Buffer[] shape when calling the hook (or only using chunked hashing when there are no taps), or explicitly updating the hook API/typedef + in-repo usages/tests to handle chunked inputs as a concatenated byte stream.
Member
Author
There was a problem hiding this comment.
Good catch. Fixed in ec06ae0: the Buffer[] shape passed to hooks.updateHash is preserved (one entry per unique asset, exactly as before), but the array is only built when the hook is actually used (hooks.updateHash.isUsed()). When nobody is tapped — the common case — we feed chunks directly into the internal hash and skip the per-asset Buffer.concat entirely. Contracts seen by existing taps stay identical; the memory win is preserved on the default path.
Generated by Claude Code
3143927 to
ec06ae0
Compare
Bumps webpack-sources to ^3.4.1 and feeds Source bytes into hashes via the new Source.prototype.buffers() API. For ConcatSource/ReplaceSource outputs this removes the intermediate Buffer.concat that source.buffer() performs, eliminating a peak-memory spike equal to the source's total size on each hashed asset: - lib/util/source.js: add updateHashFromSource(hash, source) helper that prefers buffers() and falls back for older sources. - lib/asset/AssetGenerator.js: use the helper in getFullContentHash. - lib/dependencies/CssIcssExportDependency.js: same for [contenthash] in CSS ident generation. - lib/optimize/RealContentHashPlugin.js: replace mapAndDeduplicateBuffers with mapAndDeduplicateSourceBuffers + bufferArraysEqual so dedup and hashing both work on chunk arrays without concatenation. Quick measurement on a 64 MiB ConcatSource: buffer() path: ~121 ms, +64 MiB peak external memory buffers() path: ~64 ms, 0 MiB peak external memory
ec06ae0 to
1d511f3
Compare
Contributor
Types CoverageCoverage after merging claude/update-webpack-sources-memory-UrwI7 into main will be
Coverage Report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps webpack-sources to ^3.4.1 and feeds Source bytes into hashes via
the new Source.prototype.buffers() API. For ConcatSource/ReplaceSource
outputs this removes the intermediate Buffer.concat that source.buffer()
performs, eliminating a peak-memory spike equal to the source's total
size on each hashed asset:
that prefers buffers() and falls back for older sources.
in CSS ident generation.
with mapAndDeduplicateSourceBuffers + bufferArraysEqual so dedup and
hashing both work on chunk arrays without concatenation.
Quick measurement on a 64 MiB ConcatSource:
buffer() path: ~121 ms, +64 MiB peak external memory
buffers() path: ~64 ms, 0 MiB peak external memory