but we can get negative percentage in normal circumstances it is an average time so no guarantee that it will be always greater than the Baseline

example: https://github.com/checkstyle/checkstyle/actions/runs/9910196565/job/27380011798?pr=15199

we can get negative percentage in normal circumstances it is an average time so no guarantee that it will be always greater than the Baseline

Just like we can high positives in "normal" circumstances. We have no guarantees it will be within the threshold and won't randomly take longer for unexplained reasons. We could hit some PC lag for random reasons.

example: https://github.com/checkstyle/checkstyle/actions/runs/9910196565/job/27380011798?pr=15199
Difference is within the maximum allowed difference (-.8100% <= 10%).

This actually proves we have no problems using abs normally. Even +0.81% (8/10s of a percentage) is still less than 10% (actually less than 1%) threshold and wouldn't falsely trigger.

Make sense, I got it. After taking the abs value now it will trigger:

• if it goes below the threshold on the negative side. An example of this would be the string template removal PR, performance improved so we go negative. (CI will be red so we need to decrease the baseline)
• if it goes above the threshold on the positive side. An example of this would be a syntax update PR that affects performance badly (CI will be red so we need to redo PR to think about how to optimize parsing)

We will determine if it is case 1 or 2 based on the PR activity either removal or introducing syntax.

@nrmancuso @rnveach what do you feel about decreasing the threshold percentage for example from 10% to 7%

what do you feel about decreasing the threshold percentage for example from 10% to 7%

I am not sure what analysis went into for creating these numbers. @nrmancuso was mostly in control of that.

I would like to know what the values have generally been. Like the average of the last 10-20 runs, and the percentages they were off. If we can gather that from master runs (I assume this runs all the time), then we can use those as our basis. Your example CI run showed <1%, so I wouldn't be too surprised if it could be low, however, we need to also take into account if we want to support local runs which could be faster and slower.

Failed to execute goal org.pitest:pitest-maven:1.16.1:mutationCoverage (default-cli) on project checkstyle: Execution default-cli of goal org.pitest:pitest-maven:1.16.1:mutationCoverage failed: Licence expired on 2024-06-13 and grace period ended.

Let me contact them to renew .
This PR can be merged if only pitest is failing.

Yes only pitest