fix(core): cap recursive file crawler at 100k entries to prevent OOM#3138
Merged
Conversation
When the @ autocomplete triggers RecursiveFileSearch, the crawler materialises the entire project tree into memory with no upper bound. For very large workspaces (missing .gitignore, huge node_modules, home directory as cwd) this pushes Node.js past its heap limit and crashes. - Add `maxFiles` option to CrawlOptions; use fdir's withMaxFiles() to stop traversal early instead of post-hoc truncation - Apply file-level ignore patterns during crawl via fdir filter() so ignored files don't consume the maxFiles budget - Include maxFiles in the crawl cache key for correctness - Set MAX_CRAWL_FILES = 100 000 in RecursiveFileSearch (caps peak memory at ~50 MB for the file list) Fixes #3130
Contributor
📋 Review SummaryThis PR addresses a critical OOM vulnerability where recursive file crawling with 🔍 General Feedback
🎯 Specific Feedback🟢 Medium
🔵 Low
✅ Highlights
|
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
yiliang114
approved these changes
Apr 11, 2026
xaelistic
pushed a commit
to xaelistic/qwen-code
that referenced
this pull request
Jun 7, 2026
…wenLM#3138) When the @ autocomplete triggers RecursiveFileSearch, the crawler materialises the entire project tree into memory with no upper bound. For very large workspaces (missing .gitignore, huge node_modules, home directory as cwd) this pushes Node.js past its heap limit and crashes. - Add `maxFiles` option to CrawlOptions; use fdir's withMaxFiles() to stop traversal early instead of post-hoc truncation - Apply file-level ignore patterns during crawl via fdir filter() so ignored files don't consume the maxFiles budget - Include maxFiles in the crawl cache key for correctness - Set MAX_CRAWL_FILES = 100 000 in RecursiveFileSearch (caps peak memory at ~50 MB for the file list) Fixes QwenLM#3130
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
When user input contains
@(e.g.@latestin an npm context), the CLI's autocomplete triggersRecursiveFileSearchwhich crawls the entire project tree with no upper bound. For very large workspaces — missing.gitignore, hugenode_modulestrees, or home directory as cwd — this pushes Node.js past its ~4 GB heap limit and crashes with an OOM. This PR caps the crawler at 100,000 entries and applies file-level ignore patterns during traversal so ignored files don't consume the budget.Screenshots / Video Demo
N/A — no user-facing change beyond preventing the OOM crash. The autocomplete behavior is identical for projects under 100k files (virtually all real projects).
Dive Deeper
Root cause:
RecursiveFileSearch.initialize()→crawl()usesfdirto traverse the full tree, storing every path in astring[]. The results are then.map()'d to relative paths, stored in aResultCache, and indexed byAsyncFzf— at least 3–5× the raw file list in memory. With millions of entries this exceeds the V8 heap limit.What changed (4 files):
crawler.tsmaxFilestoCrawlOptions. Usesfdir.withMaxFiles()to stop traversal natively. Addedfdir.filter()to apply file-level ignore patterns during crawl so ignored files (e.g.*.log,*.map) don't eat the budget.crawlCache.tsmaxFilesin the SHA-256 cache key so different limits don't share cached results.fileSearch.tsMAX_CRAWL_FILES = 100_000constant. Passed tocrawl()fromRecursiveFileSearch.initialize().crawler.test.tsbar.mkthat should have been filtered by*.mkignore pattern.Why 100,000? At ~100 bytes/path average, 100k entries ≈ 10 MB for the path array, ~50 MB total with caches and FZF index — well under the heap limit while covering virtually all real projects.
Reviewer Test Plan
cd packages/core && npx vitest run src/utils/filesearch/(71 tests, all pass)node dist/cli.js "check if my configs use @latest" --approval-mode yolo --output-format jsonshould complete with exit code 0 and no OOM@and confirm the autocomplete still works (returns results from the first 100k entries) without crashingTesting Matrix
Linked issues / bugs
Fixes #3130