Motivation
Found this upon investigating facebook/docusaurus#1782 and profiling memory usage in Chrome dev tools

In a large project (~> 400 pages), this plugin will consume huge memory and crash it due to the fact that webpack 4 outputs a large amount of data with its stats.toJson() by default
|
this.stats = compilation.getStats().toJson(); |
The error will be
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Even increasing to 8gb heap memory doesn't work.
Upon removing this plugin, the website can build flawlessly (even without increasing memory limit)
Recommendation
Possible improvement
- I believe we dont need full output of stats.toJson.because it can be very resource-heavy. We should minimize the amount of data retrieved from the stats object by passing needed stats https://webpack.js.org/configuration/stats/ instead
Refer to stats.toJson source code.
https://github.com/webpack/webpack/blob/1a773336187ec0218e060d453f801bc73ab98a6e/lib/Stats.js#L104
Maybe something like
this.stats = compilation.getStats().toJson({
all: false,
chunks: true,
entrypoints: true,
chunkModules: true,
reasons: true,
}, true);
- Rework the logic of getting chunk origin
|
/** |
|
* Get application chunk origin |
|
* @method getChunkOrigin |
|
* @param {object} id - Webpack application chunk id |
|
* @param {object} names - Webpack application chunk names |
|
* @param {object} modules - Webpack application chunk modules |
|
* @returns {array} Chunk Keys |
|
*/ |
|
/* eslint-disable class-methods-use-this */ |
|
getChunkOrigin({ id, names, modules }) { |
|
const origins = new Set(); |
|
for (let i = 0; i < modules.length; i += 1) { |
|
const { reasons } = modules[i]; |
|
for (let j = 0; j < reasons.length; j += 1) { |
|
const { type, userRequest } = reasons[j]; |
|
if (type === 'import()') { |
|
origins.add(userRequest); |
|
} |
|
} |
|
} |
|
|
|
if (origins.size === 0) { return [names[0] || id]; } |
|
if (this.entrypoints.has(names[0])) { |
|
origins.add(names[0]); |
|
} |
|
|
|
return Array.from(origins); |
|
} |
reasons stats is very heavy and consume a lot of memory (still crash the site)
Can we use a more memory-friendly stats option and achieve same result ?
Motivation
Found this upon investigating facebook/docusaurus#1782 and profiling memory usage in Chrome dev tools
In a large project (~> 400 pages), this plugin will consume huge memory and crash it due to the fact that webpack 4 outputs a large amount of data with its stats.toJson() by default
react-loadable-ssr-addon/source/ReactLoadableSSRAddon.js
Line 183 in 1b0c0a0
The error will be
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memoryEven increasing to 8gb heap memory doesn't work.
Upon removing this plugin, the website can build flawlessly (even without increasing memory limit)
Recommendation
Possible improvement
Refer to stats.toJson source code.
https://github.com/webpack/webpack/blob/1a773336187ec0218e060d453f801bc73ab98a6e/lib/Stats.js#L104
Maybe something like
react-loadable-ssr-addon/source/ReactLoadableSSRAddon.js
Lines 124 to 151 in 1b0c0a0
reasonsstats is very heavy and consume a lot of memory (still crash the site)Can we use a more memory-friendly stats option and achieve same result ?