Data

This time we have covered following configs:

• architectures: x64, x86, arm64, arm
• Unix: Ubuntu 18.04, Ubuntu 20.04, pop 20.04, Alpine 3.13, Debian 11, macOS 11.6.3, macOS 12.2.1
• Windows: 10 & 11

Most of the benchmarks were run on bare-metal machines, some were executed via WSL.

This would not be possible without the help from: @AndyAyersMS @carlossanlop @danmoseley @jeffhandley @sblom and @dakersnar who contributed their results and time.

The full report generated by the tool is available here. The full report contains also improvements, so if you read it from the end you can see the biggest perf improvements. There are plenty of them!

Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. Preview1 report can be found here.

Regressions

By design

• Multiple Microsoft.Extensions.DependencyInjection.TimeToFirstService* regressions
  • detected by the bot: Regression in Microsoft.Extensions.DependencyInjection.TimeToFirstService #57146
  • explained: Regression in Microsoft.Extensions.DependencyInjection.TimeToFirstService #57146 (comment)
• System.Numerics.Tests.Perf_VectorConvert.Convert_ulong_double
  • seems to be missed by the bot
  • opened: System.Numerics.Tests.Perf_VectorConvert.Convert_ulong_double has regressed a lot on x86 #66832
  • specific to Windows x86
  • explained: System.Numerics.Tests.Perf_VectorConvert.Convert_ulong_double has regressed a lot on x86 #66832 (comment)
• System.Security.Cryptography.Tests.Perf_Rfc2898DeriveBytes.DeriveBytes
  • I've missed Regressions in System.Security.Cryptography.Tests.Perf_Rfc2898DeriveBytes #65719 and opened a duplicate: System.Security.Cryptography.Tests.Perf_Rfc2898DeriveBytes.DeriveBytes regressed on Unix-like OSes #66830
  • specific to Unix, Windows seems to be unaffected
  • explained: Regressions in System.Security.Cryptography.Tests.Perf_Rfc2898DeriveBytes #65719 (comment)
• System.Numerics.Tests.Perf_BigInteger.Add(arguments: 65536,65536 bits)
  • regression was reported by the bot (Regressions in System.Numerics.Tests.Perf_BigInteger #60293), but not for this particular argument
  • added a comment: Regressions in System.Numerics.Tests.Perf_BigInteger #60293 (comment)
  • explained: Regressions in System.Numerics.Tests.Perf_BigInteger #60293 (comment)

Investigation in progress

• System.Collections.Sort<BigStruct>.LinqQuery(Size: 512), System.Collections.Sort<BigStruct>.Array_Comparison(Size: 512), System.Collections.Sort<BigStruct>.LinqQuery(Size: 512)
  • seems to be missed by the bot (cc @DrewScoggins)
  • opened: System.Collections.Sort<BigStruct>.LinqQuery has regressed on all configs except Windows 64 bit #66776
  • all configs except 64 bit Windows are affected
• System.Threading.Tests.Perf_Timer.SynchronousContention
  • not reported by the bot because we don't have macOS machines in our perf lab (yet)
  • opened: System.Threading.Tests.Perf_Timer.SynchronousContention has regressed on macOS x64 #66774
  • specific to macOS x64 12.2.1 (arm64 and macOS 11.6.3 seem to be unaffected)
  • it might be a GC issue: System.Threading.Tests.Perf_Timer.SynchronousContention has regressed on macOS x64 #66774 (comment)
• System.Threading.Tasks.Tests.Perf_AsyncMethods.Yield
  • not reported by the bot because the data from Reporting System does not show it
  • opened: System.Threading.Tasks.Tests.Perf_AsyncMethods.Yield regressed on ARM64 #66837
  • specific to ARM64
• System.Text.Json.Tests.Perf_Get.GetSingle
  • seems to be missed by the bot (cc @DrewScoggins)
  • opened: float parsing benchmarks have regressed on 32 bit platforms #66838
  • specific to 32 bit
• System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString
  • seems to be missed by the bot (cc @DrewScoggins)
  • opened: System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString regressed on 32 bit platforms #66840
  • specific to 32 bit
• there was quite a lot of allocation-heavy benchmarks that have regressed on macOS x64, I've added some of them to GC.AllocateUninitializedArray has regressed on macOS #65198 which was discovered in Preview1 report.

Noise, flaky or multimodal

The following benchmarks showed up in the report generated by the tool, but were not actual regressions:

• System.Threading.Tests.Perf_Interlocked.CompareExchange_long, System.Threading.Tests.Perf_Volatile.Read_double on x86: CompareExchange_long benchmark sometimes reports very long execution time on x86 performance#1497 (comment)
• MicroBenchmarks.Serializers.Json_ToString<LoginViewModel>.Jil_ - just noisy
• PerfLabTests.EnumPerf.EnumEquals - noisy

Big thanks to everyone involved!