New benchmarking using hyperfine by dimitris-m · Pull Request #557 · opengrep/opengrep

dimitris-m · 2026-01-30T03:39:46Z

Changes

Must merge opengrep/semgrep-interfaces#4 first -- this is now DONE.

See perf/opengrep-scripts/README.md for details.

Benchmarking

Replace Python bench scripts with shell-based scripts using hyperfine.
Add bench.sh, bench-against-sha.sh, compare-outputs.sh, and supporting utilities.
New opengrep-diff tool for comparing scan outputs between versions.

Performance improvements

Remove unnecessary system calls. In some cases --time was making scans > 2 times slower... now the performance is almost the same.
Make --time output more compact and space-efficient. In large repos like grafana the output json with --time was around 20M and now it's around 3M (close to the size without --time).

Small fixes and technical debt

Fix Python CLI --time output.
Remove unused Common.jsoo (JS target dropped).
Remove unused and commented out CLI arguments.
Fix opengrep email in cli/setup.py.
Add missing opam/parser_vbnet.opam file.

perf/opengrep-scripts/.semgrepignore

When `--time` is passed, we had a lot of system calls to filesize when in fact the information is already available: we parsed targets and can calculate `String.length` on the lazy content, which is equivalent.

- We no longer store long sequences of 0s, only rule_id * match_time entries per file. This applies to json output. - We estimate more accurately the parsing time (parsing + prefiltering in fact). This can be further refined but for now it's already an improvement. In a scan of grafana with the tainting rules from opengrep-rules and the --time flag, the size of the json dropped from ~20M to ~3M. NOTE: The ATD of semgrep_interfaces has been adapted.

NOTE: These will be deleted at a later stage.

- added a tool that compares json results so we can track differences between opengrep versions and between opengrep and semgrep. - added a set of scripts that can be used to run benchmarks and produce some useful reports.

New standalone script to generate benchmark reports for all configurations in a results directory. Supports both multi-config and single-config layouts. run-benchmarks.sh now calls this after completing benchmarks.

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [opengrep/opengrep](https://github.com/opengrep/opengrep) | patch | `v1.16.0` → `v1.16.1` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>opengrep/opengrep (opengrep/opengrep)</summary> ### [`v1.16.1`](https://github.com/opengrep/opengrep/releases/tag/v1.16.1): Opengrep 1.16.1 [Compare Source](opengrep/opengrep@v1.16.0...v1.16.1) #### Improvements - Pin Nuitka to 2.8.9 across all build workflows by [@dimitris-m](https://github.com/dimitris-m) in [#594](opengrep/opengrep#594) - Remove redundant pip and Nuitka dependencies by [@dimitris-m](https://github.com/dimitris-m) in [#573](opengrep/opengrep#573) - Support split rule/target directories in test subcommand by [@qkaiser](https://github.com/qkaiser) in [#576](opengrep/opengrep#576) #### Benchmarking - New benchmarking using hyperfine by [@dimitris-m](https://github.com/dimitris-m) in [#557](opengrep/opengrep#557) and [#579](opengrep/opengrep#579) #### Bug fixes - Allow multiple logical operators in metavariable comparison by [@maciejpirog](https://github.com/maciejpirog) in [#590](opengrep/opengrep#590) - In `--experimental`, don't report git untracked files as skipped with `--use-git-ignore` by [@maciejpirog](https://github.com/maciejpirog) in [#577](opengrep/opengrep#577) - C#: Add primary constructor arguments to base class by [@maciejpirog](https://github.com/maciejpirog) in [#589](opengrep/opengrep#589) - Dockerfile: Add missing buildkit constructs by [@maciejpirog](https://github.com/maciejpirog) in [#581](opengrep/opengrep#581) - Dockerfile: Fix CRLF and comment-in-continuation parsing by [@abezdina](https://github.com/abezdina) in [#586](opengrep/opengrep#586) - Rust: Fix taint propagation through variable shadowing by [@dimitris-m](https://github.com/dimitris-m) in [#572](opengrep/opengrep#572) - TS/TSX: Add support for the `satisfies` construct by [@maciejpirog](https://github.com/maciejpirog) in [#592](opengrep/opengrep#592) #### Installation - Add Windows install script (pwsh) by [@dimitris-m](https://github.com/dimitris-m) in [#569](opengrep/opengrep#569) - Ensure that install.ps1 works on ARM by [@dimitris-m](https://github.com/dimitris-m) in [#571](opengrep/opengrep#571) - Fix: handle unparseable cosign version in install.sh by [@dimitris-m](https://github.com/dimitris-m) in [#580](opengrep/opengrep#580) #### Documentation - Improve the README by [@dimitris-m](https://github.com/dimitris-m) in [#570](opengrep/opengrep#570) #### New Contributors - [@qkaiser](https://github.com/qkaiser) made their first contribution in [#576](opengrep/opengrep#576) - [@abezdina](https://github.com/abezdina) made their first contribution in [#586](opengrep/opengrep#586) **Full Changelog**: <opengrep/opengrep@v1.16.0...v1.16.1> </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever MR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

dimitris-m requested review from corneliuhoffman, maciejpirog and willem-delbare as code owners January 30, 2026 03:39

dimitris-m force-pushed the dm/bench-hyperfine branch 3 times, most recently from 8db6106 to 64b8662 Compare January 30, 2026 03:58

dimitris-m mentioned this pull request Jan 30, 2026

new bench.py script #532

Closed

corneliuhoffman reviewed Jan 30, 2026

View reviewed changes

perf/opengrep-scripts/.semgrepignore Show resolved Hide resolved

dimitris-m force-pushed the dm/bench-hyperfine branch 3 times, most recently from cc5d66b to 1856805 Compare February 4, 2026 00:21

dimitris-m removed the request for review from willem-delbare February 4, 2026 01:00

dimitris-m force-pushed the dm/bench-hyperfine branch from 1856805 to 3ea4798 Compare February 4, 2026 21:32

dimitris-m added 12 commits February 4, 2026 23:42

add missing opam/parser_vbnet.opam

c4a1638

python: fix --time output

0aa6fae

remove Common.jsoo, we dropped js long ago

fdc1eaa

remove unused cli arguments

fc7b81b

perf: remove a ton of unneeded system calls

faac7ae

When `--time` is passed, we had a lot of system calls to filesize when in fact the information is already available: we parsed targets and can calculate `String.length` on the lazy content, which is equivalent.

delete our previous python bench scripts

08ebc7c

fix typo only

a8f4301

adapt semgrep's perf scripts

a5aeb93

NOTE: These will be deleted at a later stage.

fix opengrep email in cli/setup.py

82f079a

add new output compare tool and bench scripts

8952416

- added a tool that compares json results so we can track differences between opengrep versions and between opengrep and semgrep. - added a set of scripts that can be used to run benchmarks and produce some useful reports.

add diff of targets scanned to bench report

7a455db

dimitris-m force-pushed the dm/bench-hyperfine branch from 3ea4798 to 7a455db Compare February 5, 2026 18:30

add generate-reports.sh and integrate into run-benchmarks.sh

b7edc70

New standalone script to generate benchmark reports for all configurations in a results directory. Supports both multi-config and single-config layouts. run-benchmarks.sh now calls this after completing benchmarks.

dimitris-m force-pushed the dm/bench-hyperfine branch from 8385662 to b7edc70 Compare February 5, 2026 18:57

dimitris-m added 2 commits February 5, 2026 19:01

add mention of opengrep-diff in bench README.md

938abb1

update semgrep-interfaces: point to opengrep/main

22ddcc9

dimitris-m merged commit 7601289 into main Feb 5, 2026
3 checks passed

dimitris-m deleted the dm/bench-hyperfine branch February 5, 2026 19:11

maciejpirog mentioned this pull request Feb 17, 2026

Release v1.16.1 #593

Merged

dimitris-m mentioned this pull request Feb 17, 2026

Release v1.16.1 #595

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New benchmarking using hyperfine#557

New benchmarking using hyperfine#557
dimitris-m merged 15 commits intomainfrom
dm/bench-hyperfine

dimitris-m commented Jan 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dimitris-m commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Benchmarking

Performance improvements

Small fixes and technical debt

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dimitris-m commented Jan 30, 2026 •

edited

Loading