Describe the bug
When using ssrLoadModule to load files, the memory required to load files appears to be very large. In my testing about 100x larger than the size of the file itself.
This is problematic in particular when attempting to load large JSON files or text files (using ?raw) for use on the server. On my computer the memory usage is so large that loading a 50 MB file causes node to crash due to running out of memory. On my older computer, which has 8 GB of RAM, it is even worse, with 30 MB and sometimes even 20 MB files causing a crash.
(I originally found this issue when trying to import a large JSON file in a SvelteKit endpoint and filed it as sveltejs/kit#8592)
Reproduction
https://github.com/FeldrinH/vite-oom-repro
Steps to reproduce
Run npm install followed by node --expose-gc test.js
System Info
System:
OS: Windows 10 10.0.19045
CPU: (8) x64 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
Memory: 8.23 GB / 15.69 GB
Binaries:
Node: 18.12.1 - C:\Program Files\nodejs\node.EXE
npm: 9.1.2 - C:\Program Files\nodejs\npm.CMD
Browsers:
Edge: Spartan (44.19041.1266.0), Chromium (108.0.1462.76)
Internet Explorer: 11.0.19041.1566Used Package Manager
npm
Logs
C:\...\vite-oom-repro>node --expose-gc test.js
Loading progressively larger files and observing heap usage:
Loading 10 MB file
Loaded 10 MB file. Heap usage: ~1070 MB
Loading 20 MB file
04:52:16 [vite] page reload largefile_20mb.txt
Loaded 20 MB file. Heap usage: ~1931 MB
Loading 30 MB file
04:52:23 [vite] page reload largefile_30mb.txt
Loaded 30 MB file. Heap usage: ~2896 MB
Loading 40 MB file
04:52:33 [vite] page reload largefile_40mb.txt
Loaded 40 MB file. Heap usage: ~3903 MB
Loading 50 MB file
04:52:55 [vite] page reload largefile_50mb.txt
<--- Last few GCs --->
[8020:0000023BFFAE9880] 65299 ms: Scavenge 4044.7 (4113.7) -> 4043.9 (4136.5) MB, 19.2 / 0.0 ms (average mu = 0.269, current mu = 0.228) allocation failure;
[8020:0000023BFFAE9880] 67745 ms: Mark-sweep 4059.6 (4136.5) -> 4059.6 (4136.5) MB, 2441.3 / 0.0 ms (average mu = 0.167, current mu = 0.031) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 00007FF76E9E1B7F node_api_throw_syntax_error+203775
2: 00007FF76E961556 v8::internal::wasm::WasmCode::safepoint_table_offset+63558
3: 00007FF76E9628C2 v8::internal::wasm::WasmCode::safepoint_table_offset+68530
4: 00007FF76F4047F4 v8::Isolate::ReportExternalAllocationLimitReached+116
5: 00007FF76F3EFB52 v8::Isolate::Exit+674
6: 00007FF76F271BBC v8::internal::EmbedderStackStateScope::ExplicitScopeForTesting+124
7: 00007FF76F27EE9D v8::internal::Heap::PublishPendingAllocations+1117
8: 00007FF76F27BF27 v8::internal::Heap::PageFlagsAreConsistent+3367
9: 00007FF76F26E657 v8::internal::Heap::CollectGarbage+2039
10: 00007FF76F285013 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath+2099
11: 00007FF76F2858BD v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath+93
12: 00007FF76F2950F3 v8::internal::Factory::NewFillerObject+851
13: 00007FF76EF86825 v8::internal::DateCache::Weekday+1349
14: 00007FF76F4A1E81 v8::internal::SetupIsolateDelegate::SetupHeap+558193
15: 00007FF6EF6A1ED8
C:\...\vite-oom-repro>
Validations
Describe the bug
When using
ssrLoadModuleto load files, the memory required to load files appears to be very large. In my testing about 100x larger than the size of the file itself.This is problematic in particular when attempting to load large JSON files or text files (using
?raw) for use on the server. On my computer the memory usage is so large that loading a 50 MB file causes node to crash due to running out of memory. On my older computer, which has 8 GB of RAM, it is even worse, with 30 MB and sometimes even 20 MB files causing a crash.(I originally found this issue when trying to import a large JSON file in a SvelteKit endpoint and filed it as sveltejs/kit#8592)
Reproduction
https://github.com/FeldrinH/vite-oom-repro
Steps to reproduce
Run
npm installfollowed bynode --expose-gc test.jsSystem Info
System: OS: Windows 10 10.0.19045 CPU: (8) x64 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz Memory: 8.23 GB / 15.69 GB Binaries: Node: 18.12.1 - C:\Program Files\nodejs\node.EXE npm: 9.1.2 - C:\Program Files\nodejs\npm.CMD Browsers: Edge: Spartan (44.19041.1266.0), Chromium (108.0.1462.76) Internet Explorer: 11.0.19041.1566Used Package Manager
npm
Logs
Validations