Skip to content

fix(cli): bump open files limit on macOS and Linux#2106

Merged
mre merged 2 commits into
masterfrom
fix/1248-too-many-open-files-2
Mar 30, 2026
Merged

fix(cli): bump open files limit on macOS and Linux#2106
mre merged 2 commits into
masterfrom
fix/1248-too-many-open-files-2

Conversation

@mre

@mre mre commented Mar 26, 2026

Copy link
Copy Markdown
Member

This automatically bumps the maximum number of open files (NOFILE) to the hard limit.
It prevents 'Too many open files (os error 24)' which happens easily on macOS due to its low default soft limit (256).

It addresses the issue originally mentioned in the comments, where users experience this during initial link extraction or link checking across multiple files.

Fixes #1248

@mre mre force-pushed the fix/1248-too-many-open-files-2 branch from 5e0474d to 36de5a6 Compare March 26, 2026 01:58
thomas-zahner
thomas-zahner approved these changes Mar 26, 2026
View reviewed changes

@thomas-zahner thomas-zahner left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to work on my machine 👍
So just it's my little comment below.

this branch

➜  ~ prlimit --pid $(pgrep lychee) --nofile
RESOURCE DESCRIPTION                SOFT   HARD UNITS
NOFILE   max number of open files 524288 524288 files

master

➜  ~ prlimit --pid $(pgrep lychee) --nofile
RESOURCE DESCRIPTION              SOFT   HARD UNITS
NOFILE   max number of open files 1024 524288 files

Comment thread lychee-bin/src/main.rs Outdated
@katrinafyi

katrinafyi commented Mar 26, 2026

Copy link
Copy Markdown
Member

I was reading the manual and this function increases the soft limit up to the hard limit of no. files, which is usually big enough.

But is it possible that the hard limit is also a low number that would be too small for lychee? If that happens, we should probably lower max-concurrency to keep lychee functional, maybe with a warning. Do we know the relationship between max-concurrency and the number of used files?

@mre mre force-pushed the fix/1248-too-many-open-files-2 branch from 36de5a6 to 0f228c6 Compare March 26, 2026 21:38
@mre

mre commented Mar 26, 2026

Copy link
Copy Markdown
Member Author

But is it possible that the hard limit is also a low number that would be too small for lychee? If that happens, we should probably lower max-concurrency to keep lychee functional, maybe with a warning. Do we know the relationship between max-concurrency and the number of used files?

Yes, it is possible to get a low number and be none the wiser. :) And also yes, we should clamp the max concurrency accordingly. But the tricky part is: clamp to what? I was roughly calculating Number of file descriptors ≈ max_concurrency + baseline_overhead where baseline_overhead is in the low two-digit range (stdin, stdout, stderr, lychee.toml, .lycheeignore, .lycheecache, reqwest + DNS handles), so maybe like 15?

Added a change for that.

@mre mre force-pushed the fix/1248-too-many-open-files-2 branch 3 times, most recently from 53f3c2f to b5f68c2 Compare March 26, 2026 22:28
@mre mre force-pushed the fix/1248-too-many-open-files-2 branch from b5f68c2 to a1f29c3 Compare March 26, 2026 22:30
@mre

mre commented Mar 26, 2026

Copy link
Copy Markdown
Member Author

Also added a regression test.

@mre mre requested review from katrinafyi and thomas-zahner March 26, 2026 22:34
thomas-zahner
thomas-zahner reviewed Mar 27, 2026
View reviewed changes
Comment thread lychee-bin/src/main.rs Outdated
@thomas-zahner

thomas-zahner commented Mar 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

I've double checked the open file descriptors on my machine and it was 18 (in idle mode with lychee -), which is bigger than the advertised 15, so I've bumped it to 20. We could maybe be even more conservative.

I also tried to improve the docs and I now try to read the limit if setting it fails. See 12d4174.

Previously, you divided by 2. let available_fds = limit.saturating_sub(BASELINE_OVERHEAD) / 2; This division was not explained in your formula in the comment. Did you do it mainly to get rid of the clippy warning or to be more conservative? I've now removed it and am now expecting the clippy warning. If you did it for being more conservative I'd rather further increase the BASELINE_OVERHEAD.

@thomas-zahner thomas-zahner force-pushed the fix/1248-too-many-open-files-2 branch from d1b49ac to 12d4174 Compare March 27, 2026 09:30
@mre mre merged commit a250853 into master Mar 30, 2026
8 checks passed
@mre mre deleted the fix/1248-too-many-open-files-2 branch March 30, 2026 16:21
@mre

mre commented Mar 30, 2026

Copy link
Copy Markdown
Member Author

Thanks for the review and for the improvements @thomas-zahner. :)

@mre mre mentioned this pull request Mar 30, 2026
Merged
@ngoldbaum

ngoldbaum commented Mar 30, 2026

Copy link
Copy Markdown

Awesome, thanks for fixing this tricky issue!

@mre

mre commented Mar 30, 2026

Copy link
Copy Markdown
Member Author

Well, let's see if we fixed it. But thanks. :)

@mre mre mentioned this pull request Apr 3, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@thomas-zahner thomas-zahner thomas-zahner approved these changes

@katrinafyi katrinafyi Awaiting requested review from katrinafyi

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Too many open files (os error 24) on macOS

4 participants

@mre @katrinafyi @thomas-zahner @ngoldbaum