-
-
Notifications
You must be signed in to change notification settings - Fork 3.3k
Out of memory during build #10485
Copy link
Copy link
Closed
Labels
needs discussionIssue needs to be discussedIssue needs to be discussed
Description
Astro Info
Astro v4.5.6
Node v18.18.2
System Linux (x64)
Package Manager unknown (edit: pnpm)
Output static
Adapter none
Integrations none
If this issue only occurs in one browser, which browser is a problem?
No response
Describe the Bug
I have a website with > 100K markdown files I need statically built.
When I played with testing the limits of astro I got:
01:49:57 [build] Building static entrypoints...
<--- Last few GCs --->
[10615:0x71cf750] 39539 ms: Scavenge 4002.0 (4127.2) -> 4002.8 (4144.0) MB, 911.3 / 0.0 ms (average mu = 0.715, current mu = 0.134) allocation failure;
[10615:0x71cf750] 42292 ms: Mark-sweep 4002.9 (4144.0) -> 4000.1 (4156.7) MB, 2752.1 / 0.0 ms (average mu = 0.533, current mu = 0.251) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
1: 0xb85bc0 node::Abort() [node]
2: 0xa94834 [node]
3: 0xd66d10 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xd670b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xf447c5 [node]
6: 0xf56cad v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
7: 0xf313ae v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
8: 0xf32777 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
9: 0xf1394a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x12d8caf v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x1705b39 [node]
Aborted
ELIFECYCLE Command failed with exit code 134.
What's the expected result?
No memory error. Probably loading md files in a buffer one by one and making sure there isn't anything growing in memory for each file. At a minimum give out a warning or error of the limits.
To create the files:
// first create pages/blog dir.
import { writeFile, mkdir } from "fs/promises";
import { join } from "path";
const createMarkdownFile = async (index) => {
const frontMatter = `---
title: "Title ${index}"
date: "2024-03-19"
---\n`;
const content = `# Title ${index}\n\nContent for file ${index}.`;
const fullPath = join("src", "pages", "blog", `file${index}.md`);
await writeFile(fullPath, frontMatter + content, "utf8");
};
const createMarkdownFiles = async () => {
try {
for (let i = 1; i <= 500000; i++) {
await createMarkdownFile(i);
}
console.log("Markdown files have been created.");
} catch (error) {
console.error("An error occurred:", error);
}
};
createMarkdownFiles();
Link to Minimal Reproducible Example
It crashed when I tried recreating 500k markdown files.
Participation
- I am willing to submit a pull request for this issue.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
needs discussionIssue needs to be discussedIssue needs to be discussed