Data

This time we have covered following configs:

• architectures: x64, x86, arm64, arm
• Unix: Alpine 3.13, CentOS 7, Ubuntu 18.04, Ubuntu 20.04, macOS 11.4, macOS 11.6.3
• Windows: 10 & 11

ARM64 results are not 100% complete because we have hit two ARM bugs: #64657 (AV) and #64980 (hang).

Most of the benchmarks were run on bare-metal machines, some were executed via WSL.

This would not be possible without the help from: @AndyAyersMS @carlossanlop @danmoseley @jeffhandley and @janvorli who contributed their results and time.

The full report generated by the tool is available here. The full report contains also improvements, so if you read it from the end you can see the biggest perf improvements. There are plenty of them!

Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871.

Regressions

By design

• System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 0]), System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 1, 1]), System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 1, 1]):
  • reading 0-1 char long lines: S.IO.StringReader: Use ReadOnlySpan.IndexOfAny in ReadLine() for performance #60463
• System.Numerics.Tests.Perf_BigInteger.Subtract(arguments: 65536,65536 bits):
  • acceptable trade-off Regressions in System.Numerics.Tests.Perf_BigInteger #60293 (comment)
• System.Numerics.Tests.Perf_VectorConvert.Convert_double_long:
  • seems to be missed by the bot
  • opened: System.Numerics.Tests.Perf_VectorConvert.Convert_double_long has regressed #65189
  • explained: System.Numerics.Tests.Perf_VectorConvert.Convert_double_long has regressed #65189 (comment)
• Multiple System.Text.RegularExpressions* regressions: System.Text.RegularExpressions performance regressions #65187
  • not detected by the bot, as it seems that benchmarks were added after the regression happened
  • opened: System.Text.RegularExpressions performance regressions #65187
  • trade-offs explained: System.Text.RegularExpressions performance regressions #65187 (comment)

Investigation in progress

• System.Numerics.Tests.Perf_Matrix4x4.NegationOperatorBenchmark, System.Collections.Sort<BigStruct>.Array_Comparison(Size: 512), System.Collections.Sort<BigStruct>.Array_ComparerClass(Size: 512), System.Collections.Sort<BigStruct>.LinqOrderByExtension(Size: 512), System.Collections.Sort<BigStruct>.Array_ComparerStruct(Size: 512)
  • detected by the bot: Perf regressions in Perf_Matrix benchmarks #59415
  • added a comment: Perf regressions in Perf_Matrix benchmarks #59415 (comment)
• System.Collections.IterateForEach<String>.SortedSet
  • it's not visible in the Reporting System, but I can reproduce it locally.
  • opened: System.Collections.IterateForEach<String>.SortedSet has regressed #65190
• System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark, System.Numerics.Tests.Perf_Matrix4x4.Transpose, System.Numerics.Tests.Perf_Matrix4x4.NegateBenchmark, System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarBenchmark
  • seems to be missed by the bot
  • opened System.Numerics.Tests.Perf_Matrix4x4.MultiplyByScalarOperatorBenchmark has regressed #65191
• PerfLabTests.GetMember.GetMethod*, PerfLabTests.GetMember.GetField
  • opened: Few performance regressions in Reflection APIs #65192
  • it turned out to be a duplicate of [Perf] Changes at 12/26/2021 10:58:11 AM #63456
• System.Collections.TryGetValueFalse<Int32, Int32>.Dictionary(Size: 512)
  • detected by the bot: [Perf] Regressions in System.Collections.TryGetValueFalse<Int32, Int32> #63745
  • added a comment: [Perf] Regressions in System.Collections.TryGetValueFalse<Int32, Int32> #63745 (comment)
• System.Tests.Perf_Version.Parse2
  • seems to be missed by the bot
  • opened: System.Tests.Perf_Version.Parse2 has regressed on Unix #65193
  • Unix specific
  • most likely caused by a PGO data update
• System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: False)
  • not reported by the bot because we don't have macOS machines in our perf lab (yet)
  • opened: GC.AllocateUninitializedArray has regressed on macOS #65198
  • macOS-specific

Fixed in the meantime

• System.Text.Json.Tests.Perf_Ctor.Ctor(Formatted: False, SkipValidation: False)
  • was there when we started the manual runs, bot got fixed very recently
• System.Text.Json.Tests.Perf_Base64.WriteByteArrayAsBase64_NoEscaping(NumberOfBytes: 100)
  • was there when we started the manual runs, bot got fixed recently

Noise, flaky or multimodal

The following benchmarks showed up in the report generated by the tool, but were not actual regressions:

• System.Threading.Tests.Perf_Interlocked.Decrement_long, System.Threading.Tests.Perf_Interlocked.Increment_long on x86: CompareExchange_long benchmark sometimes reports very long execution time on x86 performance#1497 (comment)
• System.Diagnostics.Perf_Process.GetProcessesByName I was unable to reproduce it despite my machine reported it previously
• System.Memory.Span<Char>.Fill(Size: 512) - memory alignment
• PerfLabTests.CastingPerf.CheckArrayIsInterfaceNo - very noisy

Big thanks to everyone involved!