-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Closed
Labels
arch-x64area-Codegen-Interpreter-monoos-linuxLinux OS (any supported distro)Linux OS (any supported distro)
Milestone
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | b8584841ffed04ec8559629d0392eee9879f5dcb |
| Compare | 59d56cc05b791bfb477ffc44cbef362287bd1054 |
| Diff | Diff |
Regressions in System.Collections.TryGetValueFalse<Int32, Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| SortedDictionary - Duration of single invocation | 607.25 μs | 692.09 μs | 1.14 | 0.14 | False | |||||
| ImmutableSortedDictionary - Duration of single invocation | 586.01 μs | 648.61 μs | 1.11 | 0.05 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse<Int32, Int32>*'Details
Payloads
Histogram
System.Collections.TryGetValueFalse<Int32, Int32>.SortedDictionary(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 692.0862336956523 > 653.6626083312499.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -11.19507384735334 (T) = (0 -689448.823493863) / Math.Sqrt((484235957.5897148 / (31)) + (249520970.04535538 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.08809694114354276 = (633628.1239512374 - 689448.823493863) / 633628.1239512374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.TryGetValueFalse<Int32, Int32>.ImmutableSortedDictionary(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 648.6063610491071 > 624.197248987611.
IsChangePoint: Marked as a change because one of 9/28/2022 3:03:11 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 2:57:03 PM, 12/7/2022 10:10:38 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -9.201833885530117 (T) = (0 -641639.1238436721) / Math.Sqrt((252093372.5097592 / (32)) + (245576912.4583364 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.06347841706644324 = (603340.0523666521 - 641639.1238436721) / 603340.0523666521 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | b8584841ffed04ec8559629d0392eee9879f5dcb |
| Compare | 59d56cc05b791bfb477ffc44cbef362287bd1054 |
| Diff | Diff |
Regressions in Benchstone.MDBenchI.MDGeneralArray
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Test - Duration of single invocation | 938.11 ms | 6.29 secs | 6.71 | 0.15 | False | |||||
| Test2 - Duration of single invocation | 934.69 ms | 6.31 secs | 6.75 | 0.12 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.MDBenchI.MDGeneralArray*'Details
Payloads
Histogram
Benchstone.MDBenchI.MDGeneralArray.Test
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.294134878285714 > 974.7867649853847.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -332.67252279848014 (T) = (0 -6324614195.630578) / Math.Sqrt((906468370819308.8 / (31)) + (6261031663079951 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -5.666072090617283 = (948776747.3341132 - 6324614195.630578) / 948776747.3341132 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Benchstone.MDBenchI.MDGeneralArray.Test2
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.309850924928572 > 979.834198875.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -341.4035925796587 (T) = (0 -6285254191.324551) / Math.Sqrt((714591220814518.2 / (31)) + (5974629693366718 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -5.625040301109063 = (948711842.5939794 - 6285254191.324551) / 948711842.5939794 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | b8584841ffed04ec8559629d0392eee9879f5dcb |
| Compare | 59d56cc05b791bfb477ffc44cbef362287bd1054 |
| Diff | Diff |
Regressions in System.Collections.CreateAddAndClear<Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| List - Duration of single invocation | 13.03 μs | 23.47 μs | 1.80 | 0.07 | False | |||||
| HashSet - Duration of single invocation | 74.27 μs | 86.80 μs | 1.17 | 0.05 | False | |||||
| Array - Duration of single invocation | 5.16 μs | 5.72 μs | 1.11 | 0.04 | False | |||||
| SortedSet - Duration of single invocation | 586.86 μs | 640.27 μs | 1.09 | 0.05 | False | |||||
| ImmutableArray - Duration of single invocation | 261.26 μs | 1.09 ms | 4.17 | 0.02 | False | |||||
| ConcurrentBag - Duration of single invocation | 72.90 μs | 83.50 μs | 1.15 | 0.06 | False | |||||
| IDictionary - Duration of single invocation | 101.90 μs | 115.97 μs | 1.14 | 0.01 | False | |||||
| Queue - Duration of single invocation | 25.43 μs | 37.61 μs | 1.48 | 0.11 | False | |||||
| SortedList - Duration of single invocation | 452.70 μs | 1.24 ms | 2.73 | 0.00 | False | |||||
| Dictionary - Duration of single invocation | 99.86 μs | 111.87 μs | 1.12 | 0.02 | False | |||||
| ICollection - Duration of single invocation | 17.96 μs | 29.35 μs | 1.63 | 0.06 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear<Int32>*'Details
Payloads
Histogram
System.Collections.CreateAddAndClear<Int32>.List(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.469186698893516 > 12.916729737048087.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -161.81748973999427 (T) = (0 -23729.039701293033) / Math.Sqrt((97733.2729470524 / (31)) + (45991.36346664534 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.9055328450654487 = (12452.70568950913 - 23729.039701293033) / 12452.70568950913 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<Int32>.HashSet(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 86.8032005704365 > 78.426903206159.
IsChangePoint: Marked as a change because one of 9/23/2022 1:16:35 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -36.40434044961095 (T) = (0 -86958.13508971513) / Math.Sqrt((2217138.4295794037 / (31)) + (824857.8523305526 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.15440119547874265 = (75327.48184105323 - 86958.13508971513) / 75327.48184105323 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<Int32>.Array(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.716856630642543 > 5.399100894675782.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -18.48518076366622 (T) = (0 -5567.102932469006) / Math.Sqrt((6380.931601219815 / (31)) + (6821.039358783926 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.07653819458003623 = (5171.300898098432 - 5567.102932469006) / 5171.300898098432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<Int32>.SortedSet(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 640.2655003720239 > 616.7127253262997.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/22/2022 11:54:51 PM, 11/15/2022 8:48:03 PM, 11/22/2022 11:24:47 AM, 12/8/2022 3:49:24 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -12.01220410996348 (T) = (0 -643854.4397887815) / Math.Sqrt((398689911.64089644 / (36)) + (46558078.043755606 / (22))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (36) + (22) - 2, .025) and -0.07268521834314194 = (600226.8221643551 - 643854.4397887815) / 600226.8221643551 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<Int32>.ImmutableArray(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.088982142857143 > 276.2522915484152.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -376.8357690560496 (T) = (0 -1090594.941926711) / Math.Sqrt((18660112.45729481 / (31)) + (113333022.77851015 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -3.1148981078260696 = (265035.71008296 - 1090594.941926711) / 265035.71008296 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<Int32>.ConcurrentBag(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 83.5045912403475 > 76.60009291431061.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -14.055342969139994 (T) = (0 -82297.57627283916) / Math.Sqrt((6139535.682714304 / (31)) + (2690300.2627694877 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.10275834182771805 = (74628.84038259795 - 82297.57627283916) / 74628.84038259795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<Int32>.IDictionary(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 115.9718737132353 > 107.65308224498571.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -25.98727133750973 (T) = (0 -115309.30339318319) / Math.Sqrt((3628105.640189915 / (31)) + (2778664.8968289616 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.11818768059609838 = (103121.60059902696 - 115309.30339318319) / 103121.60059902696 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<Int32>.Queue(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.612352560240964 > 26.790697540098424.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -45.925900069924374 (T) = (0 -37014.780575339915) / Math.Sqrt((1434411.4025058036 / (31)) + (250737.15594331396 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.41333275150141474 = (26189.714018880757 - 37014.780575339915) / 26189.714018880757 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<Int32>.SortedList(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2351747310897436 > 481.12689615472993.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -301.61076696726565 (T) = (0 -1229816.142481173) / Math.Sqrt((55609320.484936915 / (31)) + (126180305.55516766 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.6573410010932557 = (462799.5210156373 - 1229816.142481173) / 462799.5210156373 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<Int32>.Dictionary(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 111.87108105654761 > 104.90005048467356.
IsChangePoint: Marked as a change because one of 11/15/2022 10:02:59 PM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -39.77603222684615 (T) = (0 -112061.74309436271) / Math.Sqrt((1971209.5081188811 / (31)) + (634294.283447297 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.11699701161321148 = (100324.12077138746 - 112061.74309436271) / 100324.12077138746 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<Int32>.ICollection(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.352911646110055 > 18.873768375699232.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -47.78328987979911 (T) = (0 -29802.996069191984) / Math.Sqrt((344930.27369587903 / (31)) + (1243141.1594205475 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.6216680859074272 = (18377.987658624545 - 29802.996069191984) / 18377.987658624545 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | b8584841ffed04ec8559629d0392eee9879f5dcb |
| Compare | 59d56cc05b791bfb477ffc44cbef362287bd1054 |
| Diff | Diff |
Regressions in System.Tests.Perf_Array
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ArrayRetrieve1D - Duration of single invocation | 56.81 ms | 92.47 ms | 1.63 | 0.01 | False | |||||
| ArrayRetrieve3D - Duration of single invocation | 79.96 ms | 195.12 ms | 2.44 | 0.01 | False | |||||
| ArrayAssign3D - Duration of single invocation | 69.93 ms | 171.79 ms | 2.46 | 0.01 | False | |||||
| Clear - Duration of single invocation | 802.50 ns | 1.28 μs | 1.60 | 0.04 | False | |||||
| ArrayCopy3D - Duration of single invocation | 895.19 ns | 2.51 μs | 2.81 | 0.01 | False | |||||
| ArrayAssign1D - Duration of single invocation | 54.61 ms | 89.41 ms | 1.64 | 0.01 | False | |||||
| ArrayAssign2D - Duration of single invocation | 65.28 ms | 132.76 ms | 2.03 | 0.01 | False | |||||
| ArrayRetrieve2D - Duration of single invocation | 67.16 ms | 137.49 ms | 2.05 | 0.01 | False | |||||
| ArrayCopy2D - Duration of single invocation | 890.28 ns | 2.52 μs | 2.84 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Array*'Details
Payloads
Histogram
System.Tests.Perf_Array.ArrayRetrieve1D
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 92.47110893333334 > 59.529360575000005.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -21.223216619456945 (T) = (0 -100087510.99728836) / Math.Sqrt((944226172326.4021 / (31)) + (107004198809894.9 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.7353657290510317 = (57675168.59516424 - 100087510.99728836) / 57675168.59516424 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Array.ArrayRetrieve3D
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.11992573333333 > 83.9455244775.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -54.40797583460208 (T) = (0 -202777833.6621286) / Math.Sqrt((1706486419546.1235 / (31)) + (135365602452713.36 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.5256843560719324 = (80286292.77234729 - 202777833.6621286) / 80286292.77234729 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Array.ArrayAssign3D
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 171.7920578 > 73.9266687490625.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -55.45285415461456 (T) = (0 -182884483.0767128) / Math.Sqrt((1110423131065.4656 / (31)) + (108905512668653.58 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.5750620889404228 = (71021387.74135944 - 182884483.0767128) / 71021387.74135944 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Array.Clear
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2804935073141204 > 850.2766919782395.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -127.1820164510616 (T) = (0 -1281.975518770727) / Math.Sqrt((237.5497501364831 / (31)) + (146.7495100171736 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.5601731699878728 = (821.6879660741068 - 1281.975518770727) / 821.6879660741068 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Array.ArrayCopy3D
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.514441045422554 > 940.3953264838125.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -326.36903495578684 (T) = (0 -2511.4497234478554) / Math.Sqrt((92.51337904364681 / (31)) + (576.4641480733283 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.785979880695033 = (901.4601077525767 - 2511.4497234478554) / 901.4601077525767 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Array.ArrayAssign1D
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 89.40596875 > 57.93329311.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -21.44143410887711 (T) = (0 -97473520.43416937) / Math.Sqrt((760556966515.2164 / (31)) + (103192246062875.11 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.7587614009574852 = (55421685.04556896 - 97473520.43416937) / 55421685.04556896 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Array.ArrayAssign2D
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 132.7614871153846 > 67.79270777999999.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -40.95992139453779 (T) = (0 -142767105.53070477) / Math.Sqrt((921759610052.5035 / (31)) + (96164588354940.81 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.1915565594332844 = (65144157.43284443 - 142767105.53070477) / 65144157.43284443 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Array.ArrayRetrieve2D
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.49438343333333 > 70.714509425.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -38.38776010535931 (T) = (0 -144161378.61725006) / Math.Sqrt((573856769846.9851 / (31)) + (107603613011374.25 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.14050916337911 = (67349106.03684126 - 144161378.61725006) / 67349106.03684126 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_Array.ArrayCopy2D
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5241001749029435 > 937.8620140077794.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -324.6576433641876 (T) = (0 -2498.1992456941894) / Math.Sqrt((178.34028851076172 / (31)) + (501.1030098691688 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -1.7838377726305519 = (897.3939754160128 - 2498.1992456941894) / 897.3939754160128 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | b8584841ffed04ec8559629d0392eee9879f5dcb |
| Compare | 59d56cc05b791bfb477ffc44cbef362287bd1054 |
| Diff | Diff |
Regressions in System.Collections.CreateAddAndRemove<String>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| SortedList - Duration of single invocation | 2.67 ms | 4.41 ms | 1.65 | 0.02 | False | |||||
| Queue - Duration of single invocation | 51.88 μs | 64.29 μs | 1.24 | 0.09 | False | |||||
| List - Duration of single invocation | 167.39 μs | 570.13 μs | 3.41 | 0.05 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove<String>*'Details
Payloads
Histogram
System.Collections.CreateAddAndRemove<String>.SortedList(Size: 512)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.4123604302083335 > 2.8308888640625005.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -85.38138212497053 (T) = (0 -4277671.000971078) / Math.Sqrt((5664652178.633984 / (31)) + (4049769195.1352 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.5725245280355129 = (2720257.0927876006 - 4277671.000971078) / 2720257.0927876006 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndRemove<String>.Queue(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 64.28818438333332 > 54.06705439008348.
IsChangePoint: Marked as a change because one of 9/28/2022 9:30:44 PM, 11/15/2022 8:48:03 PM, 12/2/2022 5:21:30 AM, 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -24.59478322082287 (T) = (0 -63755.51149809263) / Math.Sqrt((4958665.781557593 / (31)) + (1047886.1980280007 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -0.20771189180473193 = (52790.33180903786 - 63755.51149809263) / 52790.33180903786 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndRemove<String>.List(Size: 512)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 570.1297382716049 > 175.73679156375115.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/13/2022 2:17:54 AM falls between 12/4/2022 6:53:17 AM and 12/13/2022 2:17:54 AM.
IsRegressionStdDev: Marked as regression because -260.4511859800968 (T) = (0 -560716.7040482993) / Math.Sqrt((16396587.887146804 / (31)) + (46053772.84641648 / (27))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (27) - 2, .025) and -2.2718187228020907 = (171377.68059719505 - 560716.7040482993) / 171377.68059719505 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Metadata
Metadata
Assignees
Labels
arch-x64area-Codegen-Interpreter-monoos-linuxLinux OS (any supported distro)Linux OS (any supported distro)