Benchmark diffs not being output on threshold regression#3515
Merged
rnro merged 1 commit intoapple:mainfrom Feb 20, 2026
Merged
Benchmark diffs not being output on threshold regression#3515rnro merged 1 commit intoapple:mainfrom
rnro merged 1 commit intoapple:mainfrom
Conversation
Motivation * `check_benchmark_thresholds.sh` was updated to check for exit code 2 from `swift package benchmark thresholds check` to distinguish threshold regression from build errors. * It seems SwiftPM's `CommandPlugin` infrastructure always exits with code 1 when `performCommand` throws, regardless of the error's raw value. So the exit code is never 2, causing all regressions to fall through to the `else` branch and be misreported as build errors, with no diff output. Modifications * Remove the `rc == 2` check and instead attempt `thresholds update` for any non-zero `rc` from `thresholds check`. * Use the result of `thresholds update` to distinguish regression (success) from build error (failure). * Remove `--exit-code` from `git diff` to prevent `set -uo pipefail` from aborting the script before output is fully flushed. Result * Benchmark threshold regressions correctly output the `=== BEGIN DIFF ===` section again. * Actual build errors are still correctly detected and reported.
fabianfett
approved these changes
Feb 20, 2026
Member
fabianfett
left a comment
There was a problem hiding this comment.
This is definitely more resilient than my solution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
check_benchmark_thresholds.shwas updated to check for exit code 2 fromswift package benchmark thresholds checkto distinguish threshold regression from build errors.CommandPlugininfrastructure always exits with code 1 whenperformCommandthrows, regardless of the error's raw value. So the exit code is never 2, causing all regressions to fall through to theelsebranch and be misreported as build errors, with no diff output.Modifications
rc == 2check and instead attemptthresholds updatefor any non-zerorcfromthresholds check.thresholds updateto distinguish regression (success) from build error (failure).--exit-codefromgit diffto preventset -uo pipefailfrom aborting the script before output is fully flushed.Result
=== BEGIN DIFF ===section again.