-
Notifications
You must be signed in to change notification settings - Fork 3
Closed
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | a077f271c3b5a8487f94111dc00a404a01fdc86b |
| Compare | 9342649430c19af4bc9fe0d870609778f244a721 |
| Diff | Diff |
Improvements in Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| GetHit - Duration of single invocation | 43.70 ns | 37.31 ns | 0.85 | 0.08 | False | |||||
| AddThenRemove_SlidingExpiration - Duration of single invocation | 28.63 μs | 26.17 μs | 0.91 | 0.55 | False | |||||
| GetMiss - Duration of single invocation | 37.03 ns | 28.20 ns | 0.76 | 0.05 | False | |||||
| TryGetValueHit - Duration of single invocation | 44.76 ns | 40.04 ns | 0.89 | 0.10 | False | |||||
| TryGetValueMiss - Duration of single invocation | 37.74 ns | 28.82 ns | 0.76 | 0.04 | True | |||||
| AddThenRemove_RelativeExpiration - Duration of single invocation | 25.20 μs | 22.41 μs | 0.89 | 0.51 | False | |||||
| CreateEntry - Duration of single invocation | 16.23 ns | 12.68 ns | 0.78 | 0.07 | False |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests*'Details
Payloads
Histogram
Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.GetHit
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 37.31027505932753 < 42.00445957479231.
IsChangePoint: Marked as a change because one of 4/27/2022 1:24:17 PM, 5/3/2022 11:42:39 AM falls between 4/24/2022 10:52:31 PM and 5/3/2022 11:42:39 AM.
IsImprovementStdDev: Marked as improvement because 21.45360784284862 (T) = (0 -38.54739073238311) / Math.Sqrt((1.8476141075196961 / (299)) + (1.7590565536599623 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.12754965818806716 = (44.18290518670283 - 38.54739073238311) / 44.18290518670283 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.AddThenRemove_SlidingExpiration
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 26.169790058523414 < 27.220558833350573.
IsChangePoint: Marked as a change because one of 4/26/2022 10:28:11 AM, 5/3/2022 11:42:39 AM falls between 4/24/2022 10:52:31 PM and 5/3/2022 11:42:39 AM.
IsImprovementStdDev: Marked as improvement because 5.047171050104607 (T) = (0 -24182.001126146224) / Math.Sqrt((6804491.578825965 / (299)) + (6637774.310751563 / (34))) is greater than 1.9671567996108528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (34) - 2, .975) and 0.08879459433000436 = (26538.474174618797 - 24182.001126146224) / 26538.474174618797 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.GetMiss
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 28.200068443259003 < 35.223835442204525.
IsChangePoint: Marked as a change because one of 4/27/2022 1:24:17 PM, 5/3/2022 11:42:39 AM falls between 4/24/2022 10:52:31 PM and 5/3/2022 11:42:39 AM.
IsImprovementStdDev: Marked as improvement because 32.20842107340933 (T) = (0 -29.11481803357828) / Math.Sqrt((0.5980157011605819 / (299)) + (1.7908671723533844 / (27))) is greater than 1.9673127716992511 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (27) - 2, .975) and 0.22430829680903439 = (37.53400727867599 - 29.11481803357828) / 37.53400727867599 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.TryGetValueHit
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 40.03653287361666 < 41.91067128244847.
IsChangePoint: Marked as a change because one of 4/27/2022 1:24:17 PM, 5/3/2022 11:42:39 AM falls between 4/24/2022 10:52:31 PM and 5/3/2022 11:42:39 AM.
IsImprovementStdDev: Marked as improvement because 25.274043830482675 (T) = (0 -38.51935027890317) / Math.Sqrt((1.8168730059820162 / (299)) + (1.2873051722288822 / (27))) is greater than 1.9673127716992511 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (27) - 2, .975) and 0.13203927680962663 = (44.37913980406526 - 38.51935027890317) / 44.37913980406526 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.TryGetValueMiss
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 28.81805650298544 < 35.775865940316415.
IsChangePoint: Marked as a change because one of 4/27/2022 1:24:17 PM, 5/3/2022 11:42:39 AM falls between 4/24/2022 10:52:31 PM and 5/3/2022 11:42:39 AM.
IsImprovementStdDev: Marked as improvement because 65.02261345489205 (T) = (0 -28.85070121681417) / Math.Sqrt((0.6436343190354133 / (299)) + (0.4594966189094766 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.23492890695296287 = (37.70983047067027 - 28.85070121681417) / 37.70983047067027 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.AddThenRemove_RelativeExpiration
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 22.410148246951216 < 24.07052892497983.
IsChangePoint: Marked as a change because one of 5/1/2022 2:35:06 PM, 5/3/2022 11:42:39 AM falls between 4/24/2022 10:52:31 PM and 5/3/2022 11:42:39 AM.
IsImprovementStdDev: Marked as improvement because 6.751456677593773 (T) = (0 -22803.56864283538) / Math.Sqrt((6765299.197270015 / (299)) + (5443556.171667827 / (10))) is greater than 1.9677212881555053 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (10) - 2, .975) and 0.18229541665939508 = (27887.29463845604 - 22803.56864283538) / 27887.29463845604 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### Microsoft.Extensions.Caching.Memory.Tests.MemoryCacheTests.CreateEntry
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 12.680601288254895 < 15.458961376699245.
IsChangePoint: Marked as a change because one of 3/30/2022 6:37:35 PM, 4/27/2022 1:24:17 PM, 5/3/2022 11:42:39 AM falls between 4/24/2022 10:52:31 PM and 5/3/2022 11:42:39 AM.
IsImprovementStdDev: Marked as improvement because 38.142440560353634 (T) = (0 -12.830243334632183) / Math.Sqrt((0.2727619366915838 / (299)) + (0.17828611497755684 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.20232649741340006 = (16.084580085746623 - 12.830243334632183) / 16.084580085746623 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | a077f271c3b5a8487f94111dc00a404a01fdc86b |
| Compare | 9342649430c19af4bc9fe0d870609778f244a721 |
| Diff | Diff |
Improvements in System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| CreateNew - Duration of single invocation | 23.54 μs | 17.90 μs | 0.76 | 0.09 | True | |||||
| CreateNew - Duration of single invocation | 23.78 μs | 18.76 μs | 0.79 | 0.09 | True | |||||
| CreateNew - Duration of single invocation | 26.55 μs | 18.14 μs | 0.68 | 0.07 | True | |||||
| CreateNew - Duration of single invocation | 24.66 μs | 18.77 μs | 0.76 | 0.11 | True |
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile*'Details
Payloads
Histogram
System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateNew(capacity: 10000000)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 17.902582178381962 < 22.884319063453184.
IsChangePoint: Marked as a change because one of 4/27/2022 1:24:17 PM, 5/3/2022 11:42:39 AM falls between 4/24/2022 10:52:31 PM and 5/3/2022 11:42:39 AM.
IsImprovementStdDev: Marked as improvement because 67.94175076121805 (T) = (0 -18556.618538282186) / Math.Sqrt((495908.59244409844 / (299)) + (143449.46593162837 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.23166461039630803 = (24151.71654120176 - 18556.618538282186) / 24151.71654120176 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateNew(capacity: 10000)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 18.763510520957485 < 22.976482527070452.
IsChangePoint: Marked as a change because one of 4/27/2022 8:46:50 AM, 5/3/2022 11:42:39 AM falls between 4/24/2022 10:52:31 PM and 5/3/2022 11:42:39 AM.
IsImprovementStdDev: Marked as improvement because 76.48391372820173 (T) = (0 -18455.86576776111) / Math.Sqrt((564028.1044781969 / (299)) + (103742.09064416958 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.23657320184355735 = (24175.03002557569 - 18455.86576776111) / 24175.03002557569 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateNew(capacity: 100000)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 18.144305307825086 < 22.7836403507761.
IsChangePoint: Marked as a change because one of 4/27/2022 1:24:17 PM, 5/3/2022 11:42:39 AM falls between 4/24/2022 10:52:31 PM and 5/3/2022 11:42:39 AM.
IsImprovementStdDev: Marked as improvement because 68.92513791875366 (T) = (0 -18512.312398627426) / Math.Sqrt((650103.0703313223 / (299)) + (126532.70253332735 / (28))) is greater than 1.967290077109849 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (28) - 2, .975) and 0.23348427284149614 = (24151.249273453406 - 18512.312398627426) / 24151.249273453406 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
```#### System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateNew(capacity: 1000000)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 18.76678380114771 < 23.129448238391113.
IsChangePoint: Marked as a change because one of 4/27/2022 8:46:50 AM, 5/3/2022 11:42:39 AM falls between 4/24/2022 10:52:31 PM and 5/3/2022 11:42:39 AM.
IsImprovementStdDev: Marked as improvement because 58.93411999890581 (T) = (0 -18496.651072240715) / Math.Sqrt((605545.2004190284 / (299)) + (210996.6244818188 / (29))) is greater than 1.9672675222597202 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (29) - 2, .975) and 0.23505449551657476 = (24180.35136337153 - 18496.651072240715) / 24180.35136337153 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Reactions are currently unavailable