docs: add benchmarks page and reproducible benchmark suite#766
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a new documentation page and a robust benchmark suite to demonstrate Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces an excellent and comprehensive benchmark suite to compare hk with other tools, along with detailed documentation. The scripts for generating the test project and running the benchmarks are well-structured. My review focuses on ensuring the correctness of the benchmark setup and improving script portability. I've identified a critical issue in the lefthook configuration that would lead to inaccurate results, a bug in the benchmark reporting script, and a couple of opportunities to enhance script portability.
| - name: black | ||
| run: black {staged_files} | ||
| glob: "*.py" | ||
| - name: ruff-format | ||
| run: ruff format --quiet --force-exclude {staged_files} | ||
| glob: "*.{py,pyi}" | ||
| - name: ruff-check | ||
| run: ruff check --force-exclude --fix {staged_files} | ||
| glob: "*.py" | ||
| - name: jq | ||
| run: > | ||
| for f in {staged_files}; do | ||
| tmp=$(mktemp); jq . "$f" > "$tmp" && mv "$tmp" "$f"; | ||
| done | ||
| glob: "*.json" | ||
| - name: yq | ||
| run: > | ||
| for f in {staged_files}; do | ||
| tmp=$(mktemp); yq eval '.' "$f" > "$tmp" && mv "$tmp" "$f"; | ||
| done | ||
| glob: "*.{yml,yaml}" | ||
| - name: shfmt | ||
| run: shfmt -w {staged_files} | ||
| glob: "*.{sh,bash}" | ||
| - name: trailing-whitespace | ||
| run: sed -i 's/[[:space:]]*$//' {staged_files} | ||
| - name: newlines | ||
| run: > | ||
| for f in {staged_files}; do | ||
| [ -s "$f" ] && [ -n "$(tail -c 1 "$f")" ] && echo >> "$f"; | ||
| done | ||
| - name: prettier | ||
| run: prettier --write {staged_files} | ||
| glob: "*.{js,ts,css,md}" | ||
| - name: eslint | ||
| run: eslint --fix {staged_files} | ||
| glob: "*.{js,ts}" |
There was a problem hiding this comment.
There are two issues in this lefthook configuration:
-
Non-recursive globs (Critical): The
globpatterns are not recursive (e.g.,*.pyonly matches files in the root directory). The synthetic project creates files in subdirectories, so these linters will not run on most files. This will lead to incorrect and misleadingly fast benchmark results forlefthook. You should use recursive globs (e.g.,**/*.py). -
Non-portable
sedcommand (Medium): Thesed -icommand fortrailing-whitespaceis not portable and will fail on macOS/BSD systems. Usingperlis a more portable alternative.
The suggestion below fixes both issues.
- name: black
run: black {staged_files}
glob: "**/*.py"
- name: ruff-format
run: ruff format --quiet --force-exclude {staged_files}
glob: "**/*.{py,pyi}"
- name: ruff-check
run: ruff check --force-exclude --fix {staged_files}
glob: "**/*.py"
- name: jq
run: >
for f in {staged_files}; do
tmp=$(mktemp); jq . "$f" > "$tmp" && mv "$tmp" "$f";
done
glob: "**/*.json"
- name: yq
run: >
for f in {staged_files}; do
tmp=$(mktemp); yq eval '.' "$f" > "$tmp" && mv "$tmp" "$f";
done
glob: "**/*.{yml,yaml}"
- name: shfmt
run: shfmt -w {staged_files}
glob: "**/*.{sh,bash}"
- name: trailing-whitespace
run: perl -pi -e 's/[[:space:]]*$//' {staged_files}
- name: newlines
run: >
for f in {staged_files}; do
[ -s "$f" ] && [ -n "$(tail -c 1 "$f")" ] && echo >> "$f";
done
- name: prettier
run: prettier --write {staged_files}
glob: "**/*.{js,ts,css,md}"
- name: eslint
run: eslint --fix {staged_files}
glob: "**/*.{js,ts}"| if other_time > 0: | ||
| speedup = other_time / hk_time | ||
| if speedup > 1.1: | ||
| print(f"{scenarios.keys().__iter__().__next__()}: hk is {speedup:.1f}x faster than {tool}") |
There was a problem hiding this comment.
There's a bug in how the scenario name is printed in the speedup annotations. scenarios.keys().__iter__().__next__() will always return the first scenario's name. To fix this, you should get the scenario name corresponding to the current loop iteration i.
| print(f"{scenarios.keys().__iter__().__next__()}: hk is {speedup:.1f}x faster than {tool}") | |
| print(f"{list(scenarios.keys())[i].replace('\n', ' ')}: hk is {speedup:.1f}x faster than {tool}") |
| types: [shell] | ||
| - id: trailing-whitespace | ||
| name: trailing-whitespace | ||
| entry: bash -c 'sed -i "s/[[:space:]]*$//" "$@"' |
There was a problem hiding this comment.
Greptile SummaryThis PR adds a reproducible benchmarking suite ( Issues found:
Confidence Score: 4/5
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[benchmark/generate-project.sh] -->|generates ~6,000 files| B[/tmp/hk-bench/]
B --> C[benchmark/run.sh]
C -->|hyperfine --prepare| D{Reset repo state}
D -->|each iteration| E[Run hk / lefthook / pre-commit / prek]
E --> F[all-files.json]
E --> G[staged-changes.json]
C -->|uv run matplotlib| H[docs/public/benchmark.png]
C -->|json.dump| I[docs/public/benchmark-data.json]
F --> J[benchmark/results/]
G --> J
K[no-changes.json] -.->|committed but not generated by run.sh| J
I --> L[docs/benchmarks.md]
H --> L
L -->|VitePress renders| M[Published Docs Site]
|
| @@ -0,0 +1 @@ | |||
|
|
|||
There was a problem hiding this comment.
Accidentally committed empty scripts/task_1.sh file
Low Severity
An empty scripts/task_1.sh file was added to the main repository's scripts/ directory. This appears to be a leftover from running benchmark/generate-project.sh (which creates scripts/task_${i}.sh files in the target directory). It doesn't belong in the source repo.
e0603c6 to
1cd0c74
Compare
|
|
||
| # Stage all files | ||
| git add -A | ||
| git commit -q -m "generated benchmark project" |
There was a problem hiding this comment.
Generate script fails on re-run, missing --allow-empty
Medium Severity
Running generate-project.sh twice on the same directory fails because git commit on line 402 lacks --allow-empty. The generated content is deterministic, so re-running produces identical files. git add -A stages nothing new, and git commit exits with code 1 ("nothing to commit"), which triggers set -euo pipefail and aborts the script before printing the summary.
b041968 to
e849d66
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
…mark suite Add comprehensive benchmarks comparing hk vs lefthook, pre-commit, and prek with 10 linters on a ~6000-file synthetic project. hk's safe parallel execution via file-level read/write locks gives it a major advantage: - All files (6158): hk 3.1s vs prek 10.5s (3.3x) vs pre-commit 10.6s (3.4x) vs lefthook 20.8s (6.6x) - Staged (500 files, 50 dirty): hk 416ms vs lefthook 717ms (1.7x) vs prek 740ms (1.8x) vs pre-commit 807ms (1.9x) Includes: - benchmark/generate-project.sh: codegen for synthetic projects - benchmark/run.sh: hyperfine runner with chart and JSON data generation - benchmark/parallel/: configs for hk, lefthook, pre-commit (10 linters) - docs/benchmarks.md: results page with dynamic Vue tables from benchmark-data.json - docs/why-hk.md: comparisons page covering parallelism, smart stashing, check_diff, plugin security, built-in Rust utilities - docs/.vitepress/config.mts: sidebar entries Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
### 🚀 Features - **(betterleaks)** add betterleaks config to hk builtin config by [@hituzi-no-sippo](https://github.com/hituzi-no-sippo) in [#750](#750) - **(builtins)** add google-java-format to builtins by [@timothysparg](https://github.com/timothysparg) in [#777](#777) - **(builtins)** add dclint to builtins by [@timothysparg](https://github.com/timothysparg) in [#779](#779) - **(config)** set default value for exclude to List() by [@timothysparg](https://github.com/timothysparg) in [#781](#781) - **(core)** add required field to prevent unconfigured steps from running by [@timothysparg](https://github.com/timothysparg) in [#785](#785) - **(gitleaks)** add gitleaks config to hk builtin config by [@hituzi-no-sippo](https://github.com/hituzi-no-sippo) in [#749](#749) - **(mdschema)** add mdschema config to hk builtin config by [@hituzi-no-sippo](https://github.com/hituzi-no-sippo) in [#748](#748) - **(pkl)** add pklr as opt-in pkl backend by [@jdx](https://github.com/jdx) in [#769](#769) - add pklr as opt-in pkl backend by [@jdx](https://github.com/jdx) in [#768](#768) ### 🐛 Bug Fixes - **(docs)** replace invalid /latest/ pkl package URIs with versioned format by [@jdx](https://github.com/jdx) in [#770](#770) - **(stage)** do not stage pre-existing untracked files by [@jdx](https://github.com/jdx) in [#788](#788) ### 📚 Documentation - add benchmarks page and reproducible benchmark suite by [@jdx](https://github.com/jdx) in [#766](#766) - add recommended setup section to mise integration by [@timothysparg](https://github.com/timothysparg) in [#780](#780) ### 📦️ Dependency Updates - lock file maintenance by [@renovate[bot]](https://github.com/renovate[bot]) in [#762](#762) - update rust crate pklr to 0.4 by [@renovate[bot]](https://github.com/renovate[bot]) in [#776](#776) - update apple-actions/import-codesign-certs digest to fe74d46 by [@renovate[bot]](https://github.com/renovate[bot]) in [#774](#774) - update anthropics/claude-code-action digest to 094bd24 by [@renovate[bot]](https://github.com/renovate[bot]) in [#773](#773) - update taiki-e/upload-rust-binary-action digest to 0e34102 by [@renovate[bot]](https://github.com/renovate[bot]) in [#775](#775) - bump usage to 3.2.0 and pkl to 0.31.1, add windows platforms by [@jdx](https://github.com/jdx) in [#787](#787) - lock file maintenance by [@renovate[bot]](https://github.com/renovate[bot]) in [#786](#786) ### New Contributors - @timothysparg made their first contribution in [#781](#781) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Primarily a version/documentation bump, but it also updates the Rust dependency lockfile (e.g., `hyper` and `windows-sys`), which could introduce build/runtime regressions. > > **Overview** > Bumps hk to **v1.40.0** and publishes the corresponding release notes in `CHANGELOG.md`. > > Updates generated CLI/docs and all Pkl package URL references in docs/examples to point at `v1.40.0`, and refreshes `Cargo.lock` with dependency updates/removals consistent with the new release. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit da00ab8. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: mise-en-dev <123107610+mise-en-dev@users.noreply.github.com>
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [hk](https://github.com/jdx/hk) | minor | `1.39.0` → `1.40.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>jdx/hk (hk)</summary> ### [`v1.40.0`](https://github.com/jdx/hk/blob/HEAD/CHANGELOG.md#1400---2026-04-01) [Compare Source](jdx/hk@v1.39.0...v1.40.0) ##### 🚀 Features - **(betterleaks)** add betterleaks config to hk builtin config by [@​hituzi-no-sippo](https://github.com/hituzi-no-sippo) in [#​750](jdx/hk#750) - **(builtins)** add google-java-format to builtins by [@​timothysparg](https://github.com/timothysparg) in [#​777](jdx/hk#777) - **(builtins)** add dclint to builtins by [@​timothysparg](https://github.com/timothysparg) in [#​779](jdx/hk#779) - **(config)** set default value for exclude to List() by [@​timothysparg](https://github.com/timothysparg) in [#​781](jdx/hk#781) - **(core)** add required field to prevent unconfigured steps from running by [@​timothysparg](https://github.com/timothysparg) in [#​785](jdx/hk#785) - **(gitleaks)** add gitleaks config to hk builtin config by [@​hituzi-no-sippo](https://github.com/hituzi-no-sippo) in [#​749](jdx/hk#749) - **(mdschema)** add mdschema config to hk builtin config by [@​hituzi-no-sippo](https://github.com/hituzi-no-sippo) in [#​748](jdx/hk#748) - **(pkl)** add pklr as opt-in pkl backend by [@​jdx](https://github.com/jdx) in [#​769](jdx/hk#769) - add pklr as opt-in pkl backend by [@​jdx](https://github.com/jdx) in [#​768](jdx/hk#768) ##### 🐛 Bug Fixes - **(docs)** replace invalid /latest/ pkl package URIs with versioned format by [@​jdx](https://github.com/jdx) in [#​770](jdx/hk#770) - **(stage)** do not stage pre-existing untracked files by [@​jdx](https://github.com/jdx) in [#​788](jdx/hk#788) ##### 📚 Documentation - add benchmarks page and reproducible benchmark suite by [@​jdx](https://github.com/jdx) in [#​766](jdx/hk#766) - add recommended setup section to mise integration by [@​timothysparg](https://github.com/timothysparg) in [#​780](jdx/hk#780) ##### 📦️ Dependency Updates - lock file maintenance by [@​renovate\[bot\]](https://github.com/renovate\[bot]) in [#​762](jdx/hk#762) - update rust crate pklr to 0.4 by [@​renovate\[bot\]](https://github.com/renovate\[bot]) in [#​776](jdx/hk#776) - update apple-actions/import-codesign-certs digest to [`fe74d46`](jdx/hk@fe74d46) by [@​renovate\[bot\]](https://github.com/renovate\[bot]) in [#​774](jdx/hk#774) - update anthropics/claude-code-action digest to [`094bd24`](jdx/hk@094bd24) by [@​renovate\[bot\]](https://github.com/renovate\[bot]) in [#​773](jdx/hk#773) - update taiki-e/upload-rust-binary-action digest to [`0e34102`](jdx/hk@0e34102) by [@​renovate\[bot\]](https://github.com/renovate\[bot]) in [#​775](jdx/hk#775) - bump usage to 3.2.0 and pkl to 0.31.1, add windows platforms by [@​jdx](https://github.com/jdx) in [#​787](jdx/hk#787) - lock file maintenance by [@​renovate\[bot\]](https://github.com/renovate\[bot]) in [#​786](jdx/hk#786) ##### New Contributors - [@​timothysparg](https://github.com/timothysparg) made their first contribution in [#​781](jdx/hk#781) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMDIuMTAiLCJ1cGRhdGVkSW5WZXIiOiI0My4xMDIuMTAiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbIlJlbm92YXRlIEJvdCIsImF1dG9tYXRpb246Ym90LWF1dGhvcmVkIiwiZGVwZW5kZW5jeS10eXBlOjptaW5vciJdfQ==-->


Summary
check_diffas the fastest fix path unique to hkFiles
benchmark/generate-project.sh— generates synthetic project with JS, Python, JSON, YAML, CSS, MD, and shell filesbenchmark/run.sh— hyperfine-based runner with chart generationbenchmark/parallel/— config files for hk, lefthook, and pre-commitdocs/benchmarks.md— documentation page with methodology and resultsdocs/.vitepress/config.mts— sidebar entryTest plan
benchmark/generate-project.sh /tmp/hk-benchto verify codegenbenchmark/run.sh /tmp/hk-benchto verify benchmarks execute🤖 Generated with Claude Code
Note
Low Risk
Low risk: changes are isolated to new benchmarking scripts/configs and documentation, with no modifications to core hook execution logic. Main risk is repo noise/bloat (committed benchmark outputs and an apparently stray empty
scripts/task_1.sh) and the maintenance cost of keeping benchmark numbers current.Overview
Adds a reproducible benchmarking suite under
benchmark/that generates a large synthetic multi-language repo and runshyperfinecomparisons acrosshk,lefthook,pre-commit, andprek, exporting JSON results and generatingdocs/public/benchmark.pngplusdocs/public/benchmark-data.json.Introduces benchmark configs for each tool (
benchmark/parallel/hk.pkl,lefthook.yml,.pre-commit-config.yaml) targeting overlapping linters (including all-files whitespace/newline fixers) to demonstrate safe parallelism.Updates docs by adding a new
Benchmarkspage (dynamic table driven bybenchmark-data.json) and aWhy hk?explainer, and wires both into the VitePress sidebar.Written by Cursor Bugbot for commit 4dca868. This will update automatically on new commits. Configure here.