Out of memory during build

[agamm]

Astro Info

Astro                    v4.5.6
Node                     v18.18.2
System                   Linux (x64)
Package Manager          unknown (edit: pnpm)
Output                   static
Adapter                  none
Integrations             none

If this issue only occurs in one browser, which browser is a problem?

No response

Describe the Bug

I have a website with > 100K markdown files I need statically built.

When I played with testing the limits of astro I got:

01:49:57 [build] Building static entrypoints...

<--- Last few GCs --->

[10615:0x71cf750]    39539 ms: Scavenge 4002.0 (4127.2) -> 4002.8 (4144.0) MB, 911.3 / 0.0 ms  (average mu = 0.715, current mu = 0.134) allocation failure; 
[10615:0x71cf750]    42292 ms: Mark-sweep 4002.9 (4144.0) -> 4000.1 (4156.7) MB, 2752.1 / 0.0 ms  (average mu = 0.533, current mu = 0.251) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb85bc0 node::Abort() [node]
 2: 0xa94834  [node]
 3: 0xd66d10 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xd670b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xf447c5  [node]
 6: 0xf56cad v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: 0xf313ae v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0xf32777 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 9: 0xf1394a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x12d8caf v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x1705b39  [node]
Aborted
 ELIFECYCLE  Command failed with exit code 134.

What's the expected result?

No memory error. Probably loading md files in a buffer one by one and making sure there isn't anything growing in memory for each file. At a minimum give out a warning or error of the limits.

To create the files:

// first create pages/blog dir.
import { writeFile, mkdir } from "fs/promises";
import { join } from "path";

const createMarkdownFile = async (index) => {
  const frontMatter = `---
title: "Title ${index}"
date: "2024-03-19"
---\n`;
  const content = `# Title ${index}\n\nContent for file ${index}.`;
  const fullPath = join("src", "pages", "blog", `file${index}.md`);

  await writeFile(fullPath, frontMatter + content, "utf8");
};

const createMarkdownFiles = async () => {
  try {
    for (let i = 1; i <= 500000; i++) {
      await createMarkdownFile(i);
    }
    console.log("Markdown files have been created.");
  } catch (error) {
    console.error("An error occurred:", error);
  }
};

createMarkdownFiles();

Link to Minimal Reproducible Example

It crashed when I tried recreating 500k markdown files.

Participation

• I am willing to submit a pull request for this issue.

[lucamtudor]

I've started seeing this as well. I had to delete "astro check" from the build command.

[mj8327]

I built a site with 10k+ markdown files and have the same error. I use the following command to increase the heap size to 8G
export NODE_OPTIONS="--max-old-space-size=8192"
and it fails too. then I tried 16G, fails too. There was no error until I set it to 24GB, the peak value is 17GB, stable at 16G. It seem that memory usage has no relation with doc file size, I used files 10 times different in size and got the same test results.
by the way, How long does it take to build a website with 100k+ documents? @agamm

[florian-lefebvre]

May be a duplicate of #7241

[ematipico]

I think this is different from #7241; that issue is about content collections.

@agamm I would ask you to create a reproduction, even if you know that it will crash. We just need a way to reproduce it in a minimal way.

[github-actions]

Hello @agamm. Please provide a minimal reproduction using a GitHub repository or StackBlitz. Issues marked with needs repro will be closed if they have no activity within 3 days.

[Princesseuh]

I've started seeing this as well. I had to delete "astro check" from the build command.

That's not the same issue. OP's issue happens after the astro check step would.

[agamm]

@ematipico here you go:
https://github.com/agamm/outofmem-astro

Check the readme on the steps to reproduce (as the markdown files are 700mb+ and I didn't want to commit them directly).

[mingjunlu]

UPDATE:
I realized that the src/pages/blog folder must be created first 💡

---

Hi @agamm,

An error message showed after I executed node populate.js (1st step in README)

An error occurred: [Error: ENOENT: no such file or directory, open 'src/pages/blog/file1.md'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: 'src/pages/blog/file1.md'
}

[agamm]

@mingjunlu did you manage to reproduce or you need any help?

[mingjunlu]

The error is successfully reproduced. Thanks!