Skip to content

[Perf] Linux/x64: 142 Regressions on 12/7/2022 10:10:38 AM #79607

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.TryGetValueFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedDictionary - Duration of single invocation 607.25 μs 692.09 μs 1.14 0.14 False
ImmutableSortedDictionary - Duration of single invocation 586.01 μs 648.61 μs 1.11 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueFalse<Int32, Int32>.SortedDictionary(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 692.0862336956523 > 653.6626083312499.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -11.19507384735334 (T) = (0 -689448.823493863) / Math.Sqrt((484235957.5897148 / (31)) + (249520970.04535538 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.08809694114354276 = (633628.1239512374 - 689448.823493863) / 633628.1239512374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;.ImmutableSortedDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 648.6063610491071 > 624.197248987611.
IsChangePoint: Marked as a change because one of 9/28/2022 3:03:11 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 2:57:03 PM, 12/7/2022 10:10:38 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.201833885530117 (T) = (0 -641639.1238436721) / Math.Sqrt((252093372.5097592 / (32)) + (245576912.4583364 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.06347841706644324 = (603340.0523666521 - 641639.1238436721) / 603340.0523666521 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in Benchstone.MDBenchI.MDGeneralArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 938.11 ms 6.29 secs 6.71 0.15 False
Test2 - Duration of single invocation 934.69 ms 6.31 secs 6.75 0.12 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.MDBenchI.MDGeneralArray*'
Details

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchI.MDGeneralArray.Test


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.294134878285714 > 974.7867649853847.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -332.67252279848014 (T) = (0 -6324614195.630578) / Math.Sqrt((906468370819308.8 / (31)) + (6261031663079951 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -5.666072090617283 = (948776747.3341132 - 6324614195.630578) / 948776747.3341132 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Benchstone.MDBenchI.MDGeneralArray.Test2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.309850924928572 > 979.834198875.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -341.4035925796587 (T) = (0 -6285254191.324551) / Math.Sqrt((714591220814518.2 / (31)) + (5974629693366718 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -5.625040301109063 = (948711842.5939794 - 6285254191.324551) / 948711842.5939794 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.CreateAddAndClear<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
List - Duration of single invocation 13.03 μs 23.47 μs 1.80 0.07 False
HashSet - Duration of single invocation 74.27 μs 86.80 μs 1.17 0.05 False
Array - Duration of single invocation 5.16 μs 5.72 μs 1.11 0.04 False
SortedSet - Duration of single invocation 586.86 μs 640.27 μs 1.09 0.05 False
ImmutableArray - Duration of single invocation 261.26 μs 1.09 ms 4.17 0.02 False
ConcurrentBag - Duration of single invocation 72.90 μs 83.50 μs 1.15 0.06 False
IDictionary - Duration of single invocation 101.90 μs 115.97 μs 1.14 0.01 False
Queue - Duration of single invocation 25.43 μs 37.61 μs 1.48 0.11 False
SortedList - Duration of single invocation 452.70 μs 1.24 ms 2.73 0.00 False
Dictionary - Duration of single invocation 99.86 μs 111.87 μs 1.12 0.02 False
ICollection - Duration of single invocation 17.96 μs 29.35 μs 1.63 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;Int32&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<Int32>.List(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.469186698893516 > 12.916729737048087.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -161.81748973999427 (T) = (0 -23729.039701293033) / Math.Sqrt((97733.2729470524 / (31)) + (45991.36346664534 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.9055328450654487 = (12452.70568950913 - 23729.039701293033) / 12452.70568950913 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.HashSet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 86.8032005704365 > 78.426903206159.
IsChangePoint: Marked as a change because one of 9/23/2022 1:16:35 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -36.40434044961095 (T) = (0 -86958.13508971513) / Math.Sqrt((2217138.4295794037 / (31)) + (824857.8523305526 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.15440119547874265 = (75327.48184105323 - 86958.13508971513) / 75327.48184105323 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.Array(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.716856630642543 > 5.399100894675782.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -18.48518076366622 (T) = (0 -5567.102932469006) / Math.Sqrt((6380.931601219815 / (31)) + (6821.039358783926 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07653819458003623 = (5171.300898098432 - 5567.102932469006) / 5171.300898098432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.SortedSet(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 640.2655003720239 > 616.7127253262997.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/8/2022 3:49:24 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.01220410996348 (T) = (0 -643854.4397887815) / Math.Sqrt((398689911.64089644 / (36)) + (46558078.043755606 / (22))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (22) - 2, .025) and -0.07268521834314194 = (600226.8221643551 - 643854.4397887815) / 600226.8221643551 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.088982142857143 > 276.2522915484152.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -376.8357690560496 (T) = (0 -1090594.941926711) / Math.Sqrt((18660112.45729481 / (31)) + (113333022.77851015 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -3.1148981078260696 = (265035.71008296 - 1090594.941926711) / 265035.71008296 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ConcurrentBag(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 83.5045912403475 > 76.60009291431061.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -14.055342969139994 (T) = (0 -82297.57627283916) / Math.Sqrt((6139535.682714304 / (31)) + (2690300.2627694877 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.10275834182771805 = (74628.84038259795 - 82297.57627283916) / 74628.84038259795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.IDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 115.9718737132353 > 107.65308224498571.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -25.98727133750973 (T) = (0 -115309.30339318319) / Math.Sqrt((3628105.640189915 / (31)) + (2778664.8968289616 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.11818768059609838 = (103121.60059902696 - 115309.30339318319) / 103121.60059902696 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.612352560240964 > 26.790697540098424.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -45.925900069924374 (T) = (0 -37014.780575339915) / Math.Sqrt((1434411.4025058036 / (31)) + (250737.15594331396 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.41333275150141474 = (26189.714018880757 - 37014.780575339915) / 26189.714018880757 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.SortedList(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2351747310897436 > 481.12689615472993.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -301.61076696726565 (T) = (0 -1229816.142481173) / Math.Sqrt((55609320.484936915 / (31)) + (126180305.55516766 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.6573410010932557 = (462799.5210156373 - 1229816.142481173) / 462799.5210156373 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.Dictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 111.87108105654761 > 104.90005048467356.
IsChangePoint: Marked as a change because one of 11/15/2022 10:02:59 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -39.77603222684615 (T) = (0 -112061.74309436271) / Math.Sqrt((1971209.5081188811 / (31)) + (634294.283447297 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.11699701161321148 = (100324.12077138746 - 112061.74309436271) / 100324.12077138746 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ICollection(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.352911646110055 > 18.873768375699232.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -47.78328987979911 (T) = (0 -29802.996069191984) / Math.Sqrt((344930.27369587903 / (31)) + (1243141.1594205475 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.6216680859074272 = (18377.987658624545 - 29802.996069191984) / 18377.987658624545 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Tests.Perf_Array

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ArrayRetrieve1D - Duration of single invocation 56.81 ms 92.47 ms 1.63 0.01 False
ArrayRetrieve3D - Duration of single invocation 79.96 ms 195.12 ms 2.44 0.01 False
ArrayAssign3D - Duration of single invocation 69.93 ms 171.79 ms 2.46 0.01 False
Clear - Duration of single invocation 802.50 ns 1.28 μs 1.60 0.04 False
ArrayCopy3D - Duration of single invocation 895.19 ns 2.51 μs 2.81 0.01 False
ArrayAssign1D - Duration of single invocation 54.61 ms 89.41 ms 1.64 0.01 False
ArrayAssign2D - Duration of single invocation 65.28 ms 132.76 ms 2.03 0.01 False
ArrayRetrieve2D - Duration of single invocation 67.16 ms 137.49 ms 2.05 0.01 False
ArrayCopy2D - Duration of single invocation 890.28 ns 2.52 μs 2.84 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Array*'
Details

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Array.ArrayRetrieve1D


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 92.47110893333334 > 59.529360575000005.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -21.223216619456945 (T) = (0 -100087510.99728836) / Math.Sqrt((944226172326.4021 / (31)) + (107004198809894.9 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.7353657290510317 = (57675168.59516424 - 100087510.99728836) / 57675168.59516424 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayRetrieve3D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.11992573333333 > 83.9455244775.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -54.40797583460208 (T) = (0 -202777833.6621286) / Math.Sqrt((1706486419546.1235 / (31)) + (135365602452713.36 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.5256843560719324 = (80286292.77234729 - 202777833.6621286) / 80286292.77234729 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign3D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 171.7920578 > 73.9266687490625.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -55.45285415461456 (T) = (0 -182884483.0767128) / Math.Sqrt((1110423131065.4656 / (31)) + (108905512668653.58 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.5750620889404228 = (71021387.74135944 - 182884483.0767128) / 71021387.74135944 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.Clear

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2804935073141204 > 850.2766919782395.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -127.1820164510616 (T) = (0 -1281.975518770727) / Math.Sqrt((237.5497501364831 / (31)) + (146.7495100171736 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.5601731699878728 = (821.6879660741068 - 1281.975518770727) / 821.6879660741068 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayCopy3D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.514441045422554 > 940.3953264838125.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -326.36903495578684 (T) = (0 -2511.4497234478554) / Math.Sqrt((92.51337904364681 / (31)) + (576.4641480733283 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.785979880695033 = (901.4601077525767 - 2511.4497234478554) / 901.4601077525767 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign1D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 89.40596875 > 57.93329311.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -21.44143410887711 (T) = (0 -97473520.43416937) / Math.Sqrt((760556966515.2164 / (31)) + (103192246062875.11 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.7587614009574852 = (55421685.04556896 - 97473520.43416937) / 55421685.04556896 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayAssign2D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 132.7614871153846 > 67.79270777999999.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -40.95992139453779 (T) = (0 -142767105.53070477) / Math.Sqrt((921759610052.5035 / (31)) + (96164588354940.81 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.1915565594332844 = (65144157.43284443 - 142767105.53070477) / 65144157.43284443 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayRetrieve2D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.49438343333333 > 70.714509425.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -38.38776010535931 (T) = (0 -144161378.61725006) / Math.Sqrt((573856769846.9851 / (31)) + (107603613011374.25 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.14050916337911 = (67349106.03684126 - 144161378.61725006) / 67349106.03684126 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Array.ArrayCopy2D

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5241001749029435 > 937.8620140077794.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -324.6576433641876 (T) = (0 -2498.1992456941894) / Math.Sqrt((178.34028851076172 / (31)) + (501.1030098691688 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.7838377726305519 = (897.3939754160128 - 2498.1992456941894) / 897.3939754160128 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline b8584841ffed04ec8559629d0392eee9879f5dcb
Compare 59d56cc05b791bfb477ffc44cbef362287bd1054
Diff Diff

Regressions in System.Collections.CreateAddAndRemove<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedList - Duration of single invocation 2.67 ms 4.41 ms 1.65 0.02 False
Queue - Duration of single invocation 51.88 μs 64.29 μs 1.24 0.09 False
List - Duration of single invocation 167.39 μs 570.13 μs 3.41 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove&lt;String&gt;*'
Details

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndRemove<String>.SortedList(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.4123604302083335 > 2.8308888640625005.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -85.38138212497053 (T) = (0 -4277671.000971078) / Math.Sqrt((5664652178.633984 / (31)) + (4049769195.1352 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.5725245280355129 = (2720257.0927876006 - 4277671.000971078) / 2720257.0927876006 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;String&gt;.Queue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 64.28818438333332 > 54.06705439008348.
IsChangePoint: Marked as a change because one of 9/28/2022 9:30:44 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -24.59478322082287 (T) = (0 -63755.51149809263) / Math.Sqrt((4958665.781557593 / (31)) + (1047886.1980280007 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.20771189180473193 = (52790.33180903786 - 63755.51149809263) / 52790.33180903786 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndRemove&lt;String&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 570.1297382716049 > 175.73679156375115.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -260.4511859800968 (T) = (0 -560716.7040482993) / Math.Sqrt((16396587.887146804 / (31)) + (46053772.84641648 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -2.2718187228020907 = (171377.68059719505 - 560716.7040482993) / 171377.68059719505 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions