ignore: don't process command line arguments in reverse order by th1000s · Pull Request #2849 · BurntSushi/ripgrep

th1000s · 2024-07-03T20:30:08Z

When searching in parallel with many more arguments than threads, the
first arguments are searched last -- unlike in the -j1 case.

This is unexpected for users who know about the parallel nature of rg and
think they can give the scheduler a hint by positioning larger input files
(L1, L2, ..) before smaller ones (█, ██). Instead, this can result in
sub-optimal thread usage and thus longer runtime
(simplified example with 2 threads):

 T1:  █ ██ █ █ █ █ ██ █ █ █ █ █ ██ ╠═════════════L1════════════╣
 T2:  █ █ ██ █ █ ██ █ █ █ ██ █ █ ╠═════L2════╣

                                       ┏━━━━┳━━━━┳━━━━┳━━━━┓
This is caused by assigning work to    ┃ T1 ┃ T2 ┃ T3 ┃ T4 ┃
 per-thread stacks in a round-robin    ┡━━━━╇━━━━╇━━━━╇━━━━┩
              manner, starting here  → │ L1 │ L2 │ L3 │ L4 │ ↵
                                       ├────├────┼────┼────┤
                                       │ s5 │ s6 │ s7 │ s8 │ ↵
                                       ├────┼────┼────┼────┤
                                       ╷ .. ╷ .. ╷ .. ╷ .. ╷
                                       ├────┼────┼────┼────┤
                                       │ st │ su │ sv │ sw │ ↵
                                       ├────┼────┼────┼────┘
                                       │ sx │ sy │ sz │
                                       └────┴────┴────┘
   and then processing them bottom-up:   ↥    ↥    ↥    ↥

                                       ╷ .. ╷ .. ╷ .. ╷ .. ╷
This patch reverses the input order    ├────┼────┼────┼────┤
so the two reversals cancel each other │ s7 │ s6 │ s5 │ L4 │ ↵
out. Now at least the first N          ├────┼────┼────┼────┘
arguments, N=number-of-threads, are    │ L3 │ L2 │ L1 │
processed before any others (then      └────┴────┴────┘
work-stealing may happen):

 T1:  ╠═════════════L1════════════╣ █ ██ █ █ █ █ █ █ ██
 T2:  ╠═════L2════╣ █ █ ██ █ █ ██ █ █ █ ██ █ █ ██ █ █ █

(With some more shuffling T1 could always be assigned L1 etc., but
that would mostly be for optics).

When searching in parallel with many more arguments than threads, the first arguments are searched last -- unlike in the -j1 case. This is unexpected for users who know about the parallel nature of rg and think they can give the scheduler a hint by positioning larger input files (L1, L2, ..) before smaller ones (█, ██). Instead, this can result in sub-optimal thread usage and thus longer runtime (simplified example with 2 threads): T1: █ ██ █ █ █ █ ██ █ █ █ █ █ ██ ╠═════════════L1════════════╣ T2: █ █ ██ █ █ ██ █ █ █ ██ █ █ ╠═════L2════╣ ┏━━━━┳━━━━┳━━━━┳━━━━┓ This is caused by assigning work to ┃ T1 ┃ T2 ┃ T3 ┃ T4 ┃ per-thread stacks in a round-robin ┡━━━━╇━━━━╇━━━━╇━━━━┩ manner, starting here → │ L1 │ L2 │ L3 │ L4 │ ↵ ├────├────┼────┼────┤ │ s5 │ s6 │ s7 │ s8 │ ↵ ├────┼────┼────┼────┤ ╷ .. ╷ .. ╷ .. ╷ .. ╷ ├────┼────┼────┼────┤ │ st │ su │ sv │ sw │ ↵ ├────┼────┼────┼────┘ │ sx │ sy │ sz │ └────┴────┴────┘ and then processing them bottom-up: ↥ ↥ ↥ ↥ ╷ .. ╷ .. ╷ .. ╷ .. ╷ This patch reverses the input order ├────┼────┼────┼────┤ so the two reversals cancel each other │ s7 │ s6 │ s5 │ L4 │ ↵ out. Now at least the first N ├────┼────┼────┼────┘ arguments, N=number-of-threads, are │ L3 │ L2 │ L1 │ processed before any others (then └────┴────┴────┘ work-stealing may happen): T1: ╠═════════════L1════════════╣ █ ██ █ █ █ █ █ █ ██ T2: ╠═════L2════╣ █ █ ██ █ █ ██ █ █ █ ██ █ █ ██ █ █ █ (With some more shuffling T1 could always be assigned L1 etc., but that would mostly be for optics).

When searching in parallel with many more arguments than threads, the first arguments are searched last -- unlike in the -j1 case. This is unexpected for users who know about the parallel nature of rg and think they can give the scheduler a hint by positioning larger input files (L1, L2, ..) before smaller ones (█, ██). Instead, this can result in sub-optimal thread usage and thus longer runtime (simplified example with 2 threads): T1: █ ██ █ █ █ █ ██ █ █ █ █ █ ██ ╠═════════════L1════════════╣ T2: █ █ ██ █ █ ██ █ █ █ ██ █ █ ╠═════L2════╣ ┏━━━━┳━━━━┳━━━━┳━━━━┓ This is caused by assigning work to ┃ T1 ┃ T2 ┃ T3 ┃ T4 ┃ per-thread stacks in a round-robin ┡━━━━╇━━━━╇━━━━╇━━━━┩ manner, starting here → │ L1 │ L2 │ L3 │ L4 │ ↵ ├────├────┼────┼────┤ │ s5 │ s6 │ s7 │ s8 │ ↵ ├────┼────┼────┼────┤ ╷ .. ╷ .. ╷ .. ╷ .. ╷ ├────┼────┼────┼────┤ │ st │ su │ sv │ sw │ ↵ ├────┼────┼────┼────┘ │ sx │ sy │ sz │ └────┴────┴────┘ and then processing them bottom-up: ↥ ↥ ↥ ↥ ╷ .. ╷ .. ╷ .. ╷ .. ╷ This patch reverses the input order ├────┼────┼────┼────┤ so the two reversals cancel each other │ s7 │ s6 │ s5 │ L4 │ ↵ out. Now at least the first N ├────┼────┼────┼────┘ arguments, N=number-of-threads, are │ L3 │ L2 │ L1 │ processed before any others (then └────┴────┴────┘ work-stealing may happen): T1: ╠═════════════L1════════════╣ █ ██ █ █ █ █ █ █ ██ T2: ╠═════L2════╣ █ █ ██ █ █ ██ █ █ █ ██ █ █ ██ █ █ █ (With some more shuffling T1 could always be assigned L1 etc., but that would mostly be for optics). Closes #2849

⚠️ **CAUTION: this is a major update, indicating a breaking change!** ⚠️ This MR contains the following updates: | Package | Update | Change | |---|---|---| | [BurntSushi/ripgrep](https://github.com/BurntSushi/ripgrep) | major | `14.1.1` -> `15.0.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>BurntSushi/ripgrep (BurntSushi/ripgrep)</summary> ### [`v15.0.0`](https://github.com/BurntSushi/ripgrep/blob/HEAD/CHANGELOG.md#1500-2025-10-15) [Compare Source](BurntSushi/ripgrep@14.1.1...15.0.0) \=================== ripgrep 15 is a new major version release of ripgrep that mostly has bug fixes, some minor performance improvements and minor new features. Here are some highlights: - Several bugs around gitignore matching have been fixed. This includes a commonly reported bug related to applying gitignore rules from parent directories. - A memory usage regression when handling very large gitignore files has been fixed. - `rg -vf file`, where `file` is empty, now matches everything. - The `-r/--replace` flag now works with `--json`. - A subset of Jujutsu (`jj`) repositories are now treated as if they were git repositories. That is, ripgrep will respect `jj`'s gitignores. - Globs can now use nested curly braces. Platform support: - `aarch64` for Windows now has release artifacts. - `powerpc64` no longer has release artifacts generated for it. The CI release workflow stopped working, and I didn't deem it worth my time to debug it. If someone wants this and can test it, I'd be happy to add it back. - ripgrep binaries are now compiled with full LTO enabled. You may notice small performance improvements from this and a modest decrease in binary size. Performance improvements: - [PERF #2111](BurntSushi/ripgrep#2111): Don't resolve helper binaries on Windows when `-z/--search-zip` isn't used. - [PERF #2865](BurntSushi/ripgrep#2865): Avoid using path canonicalization on Windows when emitting hyperlinks. - [PERF #3184](BurntSushi/ripgrep#3184): Improve performance of large values with `-A/--after-context`. Bug fixes: - [BUG #829](BurntSushi/ripgrep#829), [BUG #2731](BurntSushi/ripgrep#2731), [BUG #2747](BurntSushi/ripgrep#2747), [BUG #2770](BurntSushi/ripgrep#2770), [BUG #2778](BurntSushi/ripgrep#2778), [BUG #2836](BurntSushi/ripgrep#2836), [BUG #2933](BurntSushi/ripgrep#2933), [BUG #3067](BurntSushi/ripgrep#3067): Fix bug related to gitignores from parent directories. - [BUG #1332](BurntSushi/ripgrep#1332), [BUG #3001](BurntSushi/ripgrep#3001): Make `rg -vf file` where `file` is empty match everything. - [BUG #2177](BurntSushi/ripgrep#2177): Ignore a UTF-8 BOM marker at the start of `.gitignore` (and similar files). - [BUG #2750](BurntSushi/ripgrep#2750): Fix memory usage regression for some truly large gitignore files. - [BUG #2944](BurntSushi/ripgrep#2944): Fix a bug where the "bytes searched" in `--stats` output could be incorrect. - [BUG #2990](BurntSushi/ripgrep#2990): Fix a bug where ripgrep would mishandle globs that ended with a `.`. - [BUG #2094](BurntSushi/ripgrep#2094), [BUG #3076](BurntSushi/ripgrep#3076): Fix bug with `-m/--max-count` and `-U/--multiline` showing too many matches. - [BUG #3100](BurntSushi/ripgrep#3100): Preserve line terminators when using `-r/--replace` flag. - [BUG #3108](BurntSushi/ripgrep#3108): Fix a bug where `-q --files-without-match` inverted the exit code. - [BUG #3131](BurntSushi/ripgrep#3131): Document inconsistency between `-c/--count` and `--files-with-matches`. - [BUG #3135](BurntSushi/ripgrep#3135): Fix rare panic for some classes of large regexes on large haystacks. - [BUG #3140](BurntSushi/ripgrep#3140): Ensure hyphens in flag names are escaped in the roff text for the man page. - [BUG #3155](BurntSushi/ripgrep#3155): Statically compile PCRE2 into macOS release artifacts on `aarch64`. - [BUG #3173](BurntSushi/ripgrep#3173): Fix ancestor ignore filter bug when searching whitelisted hidden files. - [BUG #3178](BurntSushi/ripgrep#3178): Fix bug causing incorrect summary statistics with `--json` flag. - [BUG #3179](BurntSushi/ripgrep#3179): Fix gitignore bug when searching absolute paths with global gitignores. - [BUG #3180](BurntSushi/ripgrep#3180): Fix a panicking bug when using `-U/--multiline` and `-r/--replace`. Feature enhancements: - Many enhancements to the default set of file types available for filtering. - [FEATURE #1872](BurntSushi/ripgrep#1872): Make `-r/--replace` work with `--json`. - [FEATURE #2708](BurntSushi/ripgrep#2708): Completions for the fish shell take ripgrep's config file into account. - [FEATURE #2841](BurntSushi/ripgrep#2841): Add `italic` to the list of available style attributes in `--color`. - [FEATURE #2842](BurntSushi/ripgrep#2842): Directories containing `.jj` are now treated as git repositories. - [FEATURE #2849](BurntSushi/ripgrep#2849): When using multithreading, schedule files to search in order given on CLI. - [FEATURE #2943](BurntSushi/ripgrep#2943): Add `aarch64` release artifacts for Windows. - [FEATURE #3024](BurntSushi/ripgrep#3024): Add `highlight` color type, for styling non-matching text in a matching line. - [FEATURE #3048](BurntSushi/ripgrep#3048): Globs in ripgrep (and the `globset` crate) now support nested alternates. - [FEATURE #3096](BurntSushi/ripgrep#3096): Improve completions for `--hyperlink-format` in bash and fish. - [FEATURE #3102](BurntSushi/ripgrep#3102): Improve completions for `--hyperlink-format` in zsh. </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

15.0.0 (2025-10-15) =================== ripgrep 15 is a new major version release of ripgrep that mostly has bug fixes, some minor performance improvements and minor new features. Here are some highlights: * Several bugs around gitignore matching have been fixed. This includes a commonly reported bug related to applying gitignore rules from parent directories. * A memory usage regression when handling very large gitignore files has been fixed. * `rg -vf file`, where `file` is empty, now matches everything. * The `-r/--replace` flag now works with `--json`. * A subset of Jujutsu (`jj`) repositories are now treated as if they were git repositories. That is, ripgrep will respect `jj`'s gitignores. * Globs can now use nested curly braces. Platform support: * `aarch64` for Windows now has release artifacts. * `powerpc64` no longer has release artifacts generated for it. The CI release workflow stopped working, and I didn't deem it worth my time to debug it. If someone wants this and can test it, I'd be happy to add it back. * ripgrep binaries are now compiled with full LTO enabled. You may notice small performance improvements from this and a modest decrease in binary size. Performance improvements: * [PERF #2111](BurntSushi/ripgrep#2111): Don't resolve helper binaries on Windows when `-z/--search-zip` isn't used. * [PERF #2865](BurntSushi/ripgrep#2865): Avoid using path canonicalization on Windows when emitting hyperlinks. * [PERF #3184](BurntSushi/ripgrep#3184): Improve performance of large values with `-A/--after-context`. Bug fixes: * [BUG #829](BurntSushi/ripgrep#829), [BUG #2731](BurntSushi/ripgrep#2731), [BUG #2747](BurntSushi/ripgrep#2747), [BUG #2770](BurntSushi/ripgrep#2770), [BUG #2778](BurntSushi/ripgrep#2778), [BUG #2836](BurntSushi/ripgrep#2836), [BUG #2933](BurntSushi/ripgrep#2933), [BUG #3067](BurntSushi/ripgrep#3067): Fix bug related to gitignores from parent directories. * [BUG #1332](BurntSushi/ripgrep#1332), [BUG #3001](BurntSushi/ripgrep#3001): Make `rg -vf file` where `file` is empty match everything. * [BUG #2177](BurntSushi/ripgrep#2177): Ignore a UTF-8 BOM marker at the start of `.gitignore` (and similar files). * [BUG #2750](BurntSushi/ripgrep#2750): Fix memory usage regression for some truly large gitignore files. * [BUG #2944](BurntSushi/ripgrep#2944): Fix a bug where the "bytes searched" in `--stats` output could be incorrect. * [BUG #2990](BurntSushi/ripgrep#2990): Fix a bug where ripgrep would mishandle globs that ended with a `.`. * [BUG #2094](BurntSushi/ripgrep#2094), [BUG #3076](BurntSushi/ripgrep#3076): Fix bug with `-m/--max-count` and `-U/--multiline` showing too many matches. * [BUG #3100](BurntSushi/ripgrep#3100): Preserve line terminators when using `-r/--replace` flag. * [BUG #3108](BurntSushi/ripgrep#3108): Fix a bug where `-q --files-without-match` inverted the exit code. * [BUG #3131](BurntSushi/ripgrep#3131): Document inconsistency between `-c/--count` and `--files-with-matches`. * [BUG #3135](BurntSushi/ripgrep#3135): Fix rare panic for some classes of large regexes on large haystacks. * [BUG #3140](BurntSushi/ripgrep#3140): Ensure hyphens in flag names are escaped in the roff text for the man page. * [BUG #3155](BurntSushi/ripgrep#3155): Statically compile PCRE2 into macOS release artifacts on `aarch64`. * [BUG #3173](BurntSushi/ripgrep#3173): Fix ancestor ignore filter bug when searching whitelisted hidden files. * [BUG #3178](BurntSushi/ripgrep#3178): Fix bug causing incorrect summary statistics with `--json` flag. * [BUG #3179](BurntSushi/ripgrep#3179): Fix gitignore bug when searching absolute paths with global gitignores. * [BUG #3180](BurntSushi/ripgrep#3180): Fix a panicking bug when using `-U/--multiline` and `-r/--replace`. Feature enhancements: * Many enhancements to the default set of file types available for filtering. * [FEATURE #1872](BurntSushi/ripgrep#1872): Make `-r/--replace` work with `--json`. * [FEATURE #2708](BurntSushi/ripgrep#2708): Completions for the fish shell take ripgrep's config file into account. * [FEATURE #2841](BurntSushi/ripgrep#2841): Add `italic` to the list of available style attributes in `--color`. * [FEATURE #2842](BurntSushi/ripgrep#2842): Directories containing `.jj` are now treated as git repositories. * [FEATURE #2849](BurntSushi/ripgrep#2849): When using multithreading, schedule files to search in order given on CLI. * [FEATURE #2943](BurntSushi/ripgrep#2943): Add `aarch64` release artifacts for Windows. * [FEATURE #3024](BurntSushi/ripgrep#3024): Add `highlight` color type, for styling non-matching text in a matching line. * [FEATURE #3048](BurntSushi/ripgrep#3048): Globs in ripgrep (and the `globset` crate) now support nested alternates. * [FEATURE #3096](BurntSushi/ripgrep#3096): Improve completions for `--hyperlink-format` in bash and fish. * [FEATURE #3102](BurntSushi/ripgrep#3102): Improve completions for `--hyperlink-format` in zsh.

BurntSushi added the rollup A PR that has been merged with many others in a rollup. label Jul 26, 2025

BurntSushi closed this in 75970fd Sep 20, 2025

moonfruit mentioned this pull request Oct 16, 2025

ripgrep 15.0.0 Homebrew/homebrew-core#249827

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ignore: don't process command line arguments in reverse order#2849

ignore: don't process command line arguments in reverse order#2849
th1000s wants to merge 1 commit intoBurntSushi:masterfrom
th1000s:revrev

th1000s commented Jul 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

th1000s commented Jul 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants