Conversation
This should fixed the two flaky test, `color` and `run_with_stdin_closed`.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1768 +/- ##
==========================================
- Coverage 91.71% 91.67% -0.04%
==========================================
Files 98 98
Lines 19847 19879 +32
==========================================
+ Hits 18203 18225 +22
- Misses 1644 1654 +10 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR adjusts Cmd’s PTY execution path to better capture trailing subprocess output after a fast exit, aiming to eliminate flakiness in tests that rely on color/TTY behavior.
Changes:
- Refactors PTY output collection into
pty_output_innerand adds a post-exit drain step. - Introduces
drain_ready_ptyto opportunistically read remaining bytes afterchild.wait(). - Adds a non-Windows async unit test to validate trailing output capture for fast-exiting commands.
📦 Cargo Bloat ComparisonBinary size change: +0.40% (24.7 MiB → 24.8 MiB) Expand for cargo-bloat outputHead Branch ResultsBase Branch Results |
⚡️ Hyperfine PerformanceEnvironment
CLI CommandsBenchmarking basic commands in the main repo:
|
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base --version |
2.3 ± 0.1 | 2.2 | 2.6 | 1.01 ± 0.05 |
prek-head --version |
2.3 ± 0.1 | 2.2 | 2.6 | 1.00 |
prek list
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base list |
8.8 ± 0.2 | 8.5 | 10.7 | 1.00 ± 0.03 |
prek-head list |
8.8 ± 0.1 | 8.6 | 9.1 | 1.00 |
prek validate-config .pre-commit-config.yaml
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base validate-config .pre-commit-config.yaml |
3.1 ± 0.0 | 3.0 | 3.3 | 1.03 ± 0.02 |
prek-head validate-config .pre-commit-config.yaml |
3.0 ± 0.0 | 2.9 | 3.2 | 1.00 |
prek sample-config
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base sample-config |
2.6 ± 0.1 | 2.5 | 2.7 | 1.00 |
prek-head sample-config |
2.6 ± 0.0 | 2.6 | 2.7 | 1.02 ± 0.02 |
Cold vs Warm Runs
Comparing first run (cold) vs subsequent runs (warm cache):
prek run --all-files (cold - no cache)
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run --all-files |
149.2 ± 2.3 | 147.1 | 155.0 | 1.00 |
prek-head run --all-files |
149.3 ± 1.5 | 147.1 | 152.4 | 1.00 ± 0.02 |
prek run --all-files (warm - with cache)
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run --all-files |
148.2 ± 2.4 | 144.6 | 153.5 | 1.00 |
prek-head run --all-files |
148.3 ± 2.0 | 144.8 | 154.2 | 1.00 ± 0.02 |
Full Hook Suite
Running the builtin hook suite on the benchmark workspace:
prek run --all-files (full builtin hook suite)
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run --all-files |
148.8 ± 1.7 | 145.7 | 152.8 | 1.00 |
prek-head run --all-files |
151.3 ± 18.4 | 144.5 | 276.9 | 1.02 ± 0.12 |
Individual Hook Performance
Benchmarking each hook individually on the test repo:
prek run trailing-whitespace --all-files
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run trailing-whitespace --all-files |
21.4 ± 0.4 | 20.9 | 22.8 | 1.01 ± 0.03 |
prek-head run trailing-whitespace --all-files |
21.2 ± 0.4 | 20.4 | 22.0 | 1.00 |
prek run end-of-file-fixer --all-files
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run end-of-file-fixer --all-files |
28.0 ± 1.5 | 25.3 | 31.0 | 1.01 ± 0.08 |
prek-head run end-of-file-fixer --all-files |
27.6 ± 1.7 | 25.1 | 30.9 | 1.00 |
prek run check-json --all-files
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run check-json --all-files |
12.1 ± 0.5 | 11.5 | 13.1 | 1.01 ± 0.05 |
prek-head run check-json --all-files |
12.0 ± 0.3 | 11.3 | 12.6 | 1.00 |
prek run check-yaml --all-files
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run check-yaml --all-files |
11.8 ± 0.2 | 11.5 | 12.2 | 1.01 ± 0.02 |
prek-head run check-yaml --all-files |
11.7 ± 0.1 | 11.4 | 12.0 | 1.00 |
prek run check-toml --all-files
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run check-toml --all-files |
12.0 ± 0.3 | 11.4 | 12.5 | 1.00 |
prek-head run check-toml --all-files |
12.0 ± 0.3 | 11.4 | 12.6 | 1.00 ± 0.03 |
prek run check-xml --all-files
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run check-xml --all-files |
12.0 ± 0.3 | 11.6 | 12.6 | 1.00 ± 0.03 |
prek-head run check-xml --all-files |
12.0 ± 0.3 | 11.6 | 12.8 | 1.00 |
Installation Performance
Benchmarking hook installation (fast path hooks skip Python setup):
prek install-hooks (cold - no cache)
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base install-hooks |
4.7 ± 0.1 | 4.7 | 4.8 | 1.01 ± 0.02 |
prek-head install-hooks |
4.7 ± 0.1 | 4.7 | 4.8 | 1.00 |
prek install-hooks (warm - with cache)
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base install-hooks |
4.7 ± 0.0 | 4.7 | 4.8 | 1.00 |
prek-head install-hooks |
4.7 ± 0.1 | 4.7 | 4.8 | 1.00 ± 0.01 |
File Filtering/Scoping Performance
Testing different file selection modes:
prek run (staged files only)
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run |
18.4 ± 0.1 | 18.2 | 18.6 | 1.00 ± 0.01 |
prek-head run |
18.3 ± 0.1 | 18.1 | 18.5 | 1.00 |
prek run --files '*.json' (specific file type)
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run --files '*.json' |
7.3 ± 0.1 | 7.2 | 7.5 | 1.00 ± 0.01 |
prek-head run --files '*.json' |
7.3 ± 0.1 | 7.2 | 7.5 | 1.00 |
Workspace Discovery & Initialization
Benchmarking hook discovery and initialization overhead:
prek run --dry-run --all-files (measures init overhead)
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run --dry-run --all-files |
12.3 ± 0.1 | 12.0 | 12.4 | 1.00 ± 0.01 |
prek-head run --dry-run --all-files |
12.2 ± 0.1 | 12.0 | 12.5 | 1.00 |
Meta Hooks Performance
Benchmarking meta hooks separately:
prek run check-hooks-apply --all-files
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run check-hooks-apply --all-files |
13.1 ± 0.8 | 12.3 | 14.0 | 1.05 ± 0.06 |
prek-head run check-hooks-apply --all-files |
12.5 ± 0.1 | 12.3 | 12.8 | 1.00 |
prek run check-useless-excludes --all-files
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run check-useless-excludes --all-files |
12.3 ± 0.1 | 12.2 | 12.5 | 1.00 |
prek-head run check-useless-excludes --all-files |
12.4 ± 0.1 | 12.2 | 12.5 | 1.00 ± 0.01 |
prek run identity --all-files
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|---|---|---|---|---|
prek-base run identity --all-files |
10.8 ± 0.1 | 10.7 | 11.0 | 1.00 |
prek-head run identity --all-files |
11.0 ± 0.3 | 10.7 | 11.8 | 1.02 ± 0.03 |
This should also fix the two flaky tests,
colorandrun_with_stdin_closed.