Skip to content

Comment out new test agent on Windows#7692

Merged
bouwkast merged 2 commits intomasterfrom
steven/fix-ci
Oct 21, 2025
Merged

Comment out new test agent on Windows#7692
bouwkast merged 2 commits intomasterfrom
steven/fix-ci

Conversation

@bouwkast
Copy link
Collaborator

@bouwkast bouwkast commented Oct 21, 2025 •
edited
Loading

Summary of changes

Comments out the start / stop test agent as we are hitting unauthenticated Docker rate limits as it pulls in a later version that the VM doesn't yet have generalized on it.

The tests do not yet run on Windows due to flake we were seeing, so should be no harm.

Reason for change

This is to get CI green again sooner rather than later

Mode                 LastWriteTime         Length Name                                                                 
----                 -------------         ------ ----                                                                 
d-----        10/21/2025   6:15 PM                snapshots                                                            
 test-agent.windows Pulling 
 test-agent.windows Warning 
Sending build context to Docker daemon  9.166kB

Step 1/5 : FROM python:3.10.5-windowsservercore-ltsc2022
toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit

Implementation details

Commented out the docker steps in the pipeline.

Test coverage

The tests run on Linux only at the moment, so we should have the same level of testing.
Have not tested this, will see if this resolves it hopefully.

Other details

I thought that #7656 would just cause the tests to be a bit slower as they'd need to pull from Docker for a bit as we worked on updating the VMs, but didn't realize we'd hit an unauthenticated rate limit.

Will prioritize updating the VMs with this image to get this working again.

EDIT I forgot that we skipped running on Windows for the above reason and for other flake we were attempting to address as well, so we don't have as much of a push

@bouwkast
Comment out new test agent and skip SubmitsOtlpMetrics
967502e 
Requires a VM update and we are hitting an unauthenticated Docker rate
limit which has broken all Windows Jobs
@bouwkast bouwkast requested review from a team as code owners October 21, 2025 20:31
link04
link04 approved these changes Oct 21, 2025
View reviewed changes
Copy link
Contributor

@link04 link04 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dam, sorry about this one team.

@github-actions github-actions bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Oct 21, 2025
@bouwkast bouwkast changed the title Comment out new test agent and skip SubmitsOtlpMetrics Comment out new test agent on Windows Oct 21, 2025
@bouwkast bouwkast enabled auto-merge (squash) October 21, 2025 21:35
@pr-commenter
Copy link

pr-commenter bot commented Oct 21, 2025

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #7692 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.121
  • 2 benchmarks are slower, with geometric mean 1.168
  • 3 benchmarks have fewer allocations
  • 3 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7692

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑netcoreapp3.1 5.67 KB 5.7 KB 36 B 0.64%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 10.6μs 58.9ns 358ns 0 0 0 5.51 KB
master StartStopWithChild netcoreapp3.1 15.3μs 73.3ns 311ns 0 0 0 5.67 KB
master StartStopWithChild net472 21.6μs 118ns 706ns 1.01 0.45 0.112 6 KB
#7692 StartStopWithChild net6.0 11μs 51.8ns 214ns 0 0 0 5.52 KB
#7692 StartStopWithChild netcoreapp3.1 15.1μs 28.6ns 107ns 0 0 0 5.7 KB
#7692 StartStopWithChild net472 22.2μs 125ns 810ns 0.977 0.293 0.0977 6.03 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 942μs 385ns 1.49μs 0 0 0 2.71 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 1.02ms 319ns 1.24μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 1.19ms 179ns 692ns 0 0 0 3.31 KB
#7692 WriteAndFlushEnrichedTraces net6.0 937μs 111ns 400ns 0 0 0 2.71 KB
#7692 WriteAndFlushEnrichedTraces netcoreapp3.1 1.03ms 190ns 736ns 0 0 0 2.7 KB
#7692 WriteAndFlushEnrichedTraces net472 1.19ms 96.7ns 362ns 0 0 0 3.31 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 347μs 1.73μs 7.34μs 0 0 0 179.56 KB
master AllCycleSimpleBody netcoreapp3.1 505μs 2.3μs 8.61μs 0 0 0 185.32 KB
master AllCycleSimpleBody net472 467μs 97.1ns 376ns 32.4 0 0 205.41 KB
master AllCycleMoreComplexBody net6.0 359μs 1.35μs 5.21μs 0 0 0 183.08 KB
master AllCycleMoreComplexBody netcoreapp3.1 545μs 498ns 1.93μs 0 0 0 188.75 KB
master AllCycleMoreComplexBody net472 479μs 149ns 539ns 32.4 0 0 208.94 KB
master ObjectExtractorSimpleBody net6.0 333ns 0.115ns 0.431ns 0 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 406ns 0.208ns 0.779ns 0 0 0 272 B
master ObjectExtractorSimpleBody net472 304ns 0.0386ns 0.145ns 0.0445 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 6.3μs 30.3ns 128ns 0 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 7.87μs 34.5ns 134ns 0 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 6.76μs 5.92ns 22.9ns 0.574 0 0 3.8 KB
#7692 AllCycleSimpleBody net6.0 348μs 1.71μs 7.23μs 0 0 0 179.56 KB
#7692 AllCycleSimpleBody netcoreapp3.1 502μs 1.41μs 5.28μs 0 0 0 185.32 KB
#7692 AllCycleSimpleBody net472 468μs 99.2ns 384ns 32.4 0 0 205.41 KB
#7692 AllCycleMoreComplexBody net6.0 358μs 621ns 2.41μs 0 0 0 183.09 KB
#7692 AllCycleMoreComplexBody netcoreapp3.1 508μs 1.48μs 5.72μs 0 0 0 188.75 KB
#7692 AllCycleMoreComplexBody net472 479μs 71.9ns 278ns 32.4 0 0 208.94 KB
#7692 ObjectExtractorSimpleBody net6.0 323ns 1.75ns 9.07ns 0 0 0 280 B
#7692 ObjectExtractorSimpleBody netcoreapp3.1 400ns 0.155ns 0.599ns 0 0 0 272 B
#7692 ObjectExtractorSimpleBody net472 310ns 0.113ns 0.409ns 0.0444 0 0 281 B
#7692 ObjectExtractorMoreComplexBody net6.0 6.39μs 28.6ns 111ns 0 0 0 3.78 KB
#7692 ObjectExtractorMoreComplexBody netcoreapp3.1 7.8μs 36.7ns 142ns 0 0 0 3.69 KB
#7692 ObjectExtractorMoreComplexBody net472 6.63μs 4.74ns 18.4ns 0.598 0 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 76.8μs 235ns 911ns 0 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 97μs 202ns 784ns 0 0 0 32.4 KB
master EncodeArgs net472 109μs 23.6ns 91.3ns 4.89 0 0 32.51 KB
master EncodeLegacyArgs net6.0 142μs 77.6ns 301ns 0 0 0 2.15 KB
master EncodeLegacyArgs netcoreapp3.1 196μs 204ns 790ns 0 0 0 2.14 KB
master EncodeLegacyArgs net472 262μs 42.9ns 166ns 0 0 0 2.16 KB
#7692 EncodeArgs net6.0 76.7μs 49.1ns 190ns 0 0 0 32.4 KB
#7692 EncodeArgs netcoreapp3.1 96μs 280ns 1.05μs 0 0 0 32.4 KB
#7692 EncodeArgs net472 108μs 37.8ns 146ns 4.88 0 0 32.51 KB
#7692 EncodeLegacyArgs net6.0 142μs 22.8ns 88.2ns 0 0 0 2.15 KB
#7692 EncodeLegacyArgs netcoreapp3.1 199μs 112ns 435ns 0 0 0 2.15 KB
#7692 EncodeLegacyArgs net472 263μs 40.9ns 159ns 0 0 0 2.16 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7692

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark‑netcoreapp3.1 4.48 KB 4.69 KB 211 B 4.71%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 392μs 37.7ns 146ns 0 0 0 4.55 KB
master RunWafRealisticBenchmark netcoreapp3.1 853μs 4.92μs 40.5μs 0 0 0 4.48 KB
master RunWafRealisticBenchmark net472 429μs 166ns 643ns 0 0 0 4.66 KB
master RunWafRealisticBenchmarkWithAttack net6.0 288μs 78.2ns 293ns 0 0 0 2.24 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 298μs 34.1ns 132ns 0 0 0 2.22 KB
master RunWafRealisticBenchmarkWithAttack net472 321μs 29.2ns 113ns 0 0 0 2.28 KB
#7692 RunWafRealisticBenchmark net6.0 396μs 89.7ns 347ns 0 0 0 4.55 KB
#7692 RunWafRealisticBenchmark netcoreapp3.1 846μs 3.82μs 14.8μs 0 0 0 4.69 KB
#7692 RunWafRealisticBenchmark net472 431μs 34.2ns 133ns 0 0 0 4.66 KB
#7692 RunWafRealisticBenchmarkWithAttack net6.0 286μs 23.6ns 88.3ns 0 0 0 2.24 KB
#7692 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 303μs 297ns 1.07μs 0 0 0 2.22 KB
#7692 RunWafRealisticBenchmarkWithAttack net472 308μs 28.2ns 109ns 0 0 0 2.29 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 61.9μs 283ns 1.13μs 0 0 0 14.52 KB
master SendRequest netcoreapp3.1 71.4μs 355ns 1.67μs 0 0 0 17.42 KB
master SendRequest net472 0.00428ns 0.00183ns 0.00707ns 0 0 0 0 b
#7692 SendRequest net6.0 62.4μs 67ns 241ns 0 0 0 14.52 KB
#7692 SendRequest netcoreapp3.1 71.6μs 372ns 1.74μs 0 0 0 17.42 KB
#7692 SendRequest net472 0.00401ns 0.00178ns 0.00689ns 0 0 0 0 b
Benchmarks.Trace.CharSliceBenchmark - Faster 🎉 More allocations ⚠️

Faster 🎉 in #7692

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net6.0 1.121 1,497,541.67 1,335,602.78

More allocations ⚠️ in #7692

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net6.0 2 B 7 B 5 B 250.00%

Fewer allocations 🎉 in #7692

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net472 48 B 47 B -1 B -2.08%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master OriginalCharSlice net6.0 1.94ms 497ns 1.86μs 0 0 0 640.01 KB
master OriginalCharSlice netcoreapp3.1 2.11ms 7.02μs 27.2μs 0 0 0 640 KB
master OriginalCharSlice net472 2.66ms 312ns 1.12μs 100 0 0 641.95 KB
master OptimizedCharSlice net6.0 1.5ms 232ns 899ns 0 0 0 2 B
master OptimizedCharSlice netcoreapp3.1 1.75ms 389ns 1.51μs 0 0 0 1 B
master OptimizedCharSlice net472 1.99ms 347ns 1.34μs 0 0 0 73 B
master OptimizedCharSliceWithPool net6.0 843μs 78.1ns 292ns 0 0 0 3 B
master OptimizedCharSliceWithPool netcoreapp3.1 837μs 141ns 547ns 0 0 0 1 B
master OptimizedCharSliceWithPool net472 1.16ms 124ns 464ns 0 0 0 48 B
#7692 OriginalCharSlice net6.0 1.93ms 1.81μs 7μs 0 0 0 640.01 KB
#7692 OriginalCharSlice netcoreapp3.1 2.09ms 2μs 6.93μs 0 0 0 640 KB
#7692 OriginalCharSlice net472 2.73ms 547ns 1.97μs 100 0 0 641.95 KB
#7692 OptimizedCharSlice net6.0 1.34ms 396ns 1.48μs 0 0 0 7 B
#7692 OptimizedCharSlice netcoreapp3.1 1.73ms 342ns 1.32μs 0 0 0 1 B
#7692 OptimizedCharSlice net472 1.91ms 210ns 814ns 0 0 0 73 B
#7692 OptimizedCharSliceWithPool net6.0 834μs 53.4ns 207ns 0 0 0 3 B
#7692 OptimizedCharSliceWithPool netcoreapp3.1 832μs 109ns 424ns 0 0 0 1 B
#7692 OptimizedCharSliceWithPool net472 1.17ms 122ns 456ns 0 0 0 47 B
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #7692

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 1.214 663,143.33 805,356.25 several?

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 770μs 3.19μs 12.3μs 0 0 0 41.82 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 674μs 3.84μs 26.6μs 0 0 0 42.04 KB
master WriteAndFlushEnrichedTraces net472 944μs 4.39μs 21.1μs 8.33 0 0 56.12 KB
#7692 WriteAndFlushEnrichedTraces net6.0 708μs 1.59μs 6.16μs 0 0 0 41.96 KB
#7692 WriteAndFlushEnrichedTraces netcoreapp3.1 781μs 4.58μs 44.1μs 0 0 0 41.88 KB
#7692 WriteAndFlushEnrichedTraces net472 910μs 2.37μs 9.5μs 8.33 0 0 56.25 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.9μs 8.33ns 32.3ns 0 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 2.6μs 8.41ns 31.5ns 0 0 0 1.02 KB
master ExecuteNonQuery net472 2.89μs 3.21ns 12.4ns 0.156 0.0142 0 987 B
#7692 ExecuteNonQuery net6.0 1.92μs 1.77ns 6.84ns 0 0 0 1.02 KB
#7692 ExecuteNonQuery netcoreapp3.1 2.6μs 10.4ns 40.4ns 0 0 0 1.02 KB
#7692 ExecuteNonQuery net472 2.86μs 3.44ns 13.3ns 0.156 0.0142 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.79μs 6.62ns 25.7ns 0 0 0 1.03 KB
master CallElasticsearch netcoreapp3.1 2.34μs 11.4ns 51ns 0 0 0 1.03 KB
master CallElasticsearch net472 3.74μs 2.6ns 9.71ns 0.15 0 0 1.04 KB
master CallElasticsearchAsync net6.0 1.84μs 4.03ns 15.1ns 0 0 0 1.01 KB
master CallElasticsearchAsync netcoreapp3.1 2.44μs 11.2ns 43.2ns 0 0 0 1.08 KB
master CallElasticsearchAsync net472 3.74μs 4.53ns 17.5ns 0.168 0 0 1.1 KB
#7692 CallElasticsearch net6.0 1.85μs 4.37ns 16.9ns 0 0 0 1.03 KB
#7692 CallElasticsearch netcoreapp3.1 2.32μs 8.47ns 32.8ns 0 0 0 1.03 KB
#7692 CallElasticsearch net472 3.52μs 4.46ns 17.3ns 0.158 0 0 1.04 KB
#7692 CallElasticsearchAsync net6.0 1.84μs 7.61ns 29.5ns 0 0 0 1.01 KB
#7692 CallElasticsearchAsync netcoreapp3.1 2.36μs 9.01ns 33.7ns 0 0 0 1.08 KB
#7692 CallElasticsearchAsync net472 3.66μs 3.06ns 11.8ns 0.164 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.8μs 5.94ns 22.2ns 0 0 0 952 B
master ExecuteAsync netcoreapp3.1 2.39μs 1.43ns 5.53ns 0 0 0 952 B
master ExecuteAsync net472 2.62μs 5.9ns 22.1ns 0.145 0 0 915 B
#7692 ExecuteAsync net6.0 1.91μs 4.05ns 15.7ns 0 0 0 952 B
#7692 ExecuteAsync netcoreapp3.1 2.34μs 11.4ns 49.8ns 0 0 0 952 B
#7692 ExecuteAsync net472 2.6μs 2.74ns 10.6ns 0.143 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 6.76μs 6.94ns 26ns 0 0 0 2.36 KB
master SendAsync netcoreapp3.1 8.6μs 10.9ns 42.3ns 0 0 0 2.9 KB
master SendAsync net472 13μs 12.5ns 48.3ns 0.456 0 0 3.18 KB
#7692 SendAsync net6.0 6.72μs 8.61ns 33.3ns 0 0 0 2.36 KB
#7692 SendAsync netcoreapp3.1 8.81μs 31ns 120ns 0 0 0 2.9 KB
#7692 SendAsync net472 12.6μs 11.8ns 45.7ns 0.504 0 0 3.18 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Slower ⚠️ Fewer allocations 🎉

Slower ⚠️ in #7692

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net6.0 1.123 41,600.00 46,700.00

Fewer allocations 🎉 in #7692

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net6.0 45.27 KB 43.64 KB -1.63 KB -3.60%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 274.21 KB 259.28 KB -14.93 KB -5.44%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 41.7μs 227ns 1.2μs 0 0 0 45.27 KB
master StringConcatBenchmark netcoreapp3.1 46.3μs 243ns 1.69μs 0 0 0 43.28 KB
master StringConcatBenchmark net472 55.9μs 285ns 1.3μs 0 0 0 57.34 KB
master StringConcatAspectBenchmark net6.0 447μs 1.56μs 5.61μs 0 0 0 259.62 KB
master StringConcatAspectBenchmark netcoreapp3.1 524μs 1.04μs 3.87μs 0 0 0 274.21 KB
master StringConcatAspectBenchmark net472 400μs 1.66μs 7.25μs 0 0 0 287.98 KB
#7692 StringConcatBenchmark net6.0 47.7μs 302ns 2.9μs 0 0 0 43.64 KB
#7692 StringConcatBenchmark netcoreapp3.1 48.6μs 270ns 1.77μs 0 0 0 43.28 KB
#7692 StringConcatBenchmark net472 57.3μs 272ns 1.02μs 0 0 0 57.34 KB
#7692 StringConcatAspectBenchmark net6.0 445μs 2.09μs 7.82μs 0 0 0 258.38 KB
#7692 StringConcatAspectBenchmark netcoreapp3.1 519μs 898ns 3.11μs 0 0 0 259.28 KB
#7692 StringConcatAspectBenchmark net472 406μs 2.11μs 10.3μs 0 0 0 286.72 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.62μs 1.37ns 5.3ns 0 0 0 1.7 KB
master EnrichedLog netcoreapp3.1 3.64μs 15.9ns 61.5ns 0 0 0 1.7 KB
master EnrichedLog net472 3.92μs 5.63ns 21.8ns 0.249 0 0 1.64 KB
#7692 EnrichedLog net6.0 2.55μs 13.8ns 75.7ns 0 0 0 1.7 KB
#7692 EnrichedLog netcoreapp3.1 3.72μs 14.6ns 56.5ns 0 0 0 1.7 KB
#7692 EnrichedLog net472 3.8μs 3.63ns 14.1ns 0.247 0 0 1.64 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 123μs 42.4ns 153ns 0 0 0 4.31 KB
master EnrichedLog netcoreapp3.1 130μs 425ns 1.65μs 0 0 0 4.31 KB
master EnrichedLog net472 167μs 136ns 528ns 0 0 0 4.52 KB
#7692 EnrichedLog net6.0 124μs 43.3ns 162ns 0 0 0 4.31 KB
#7692 EnrichedLog netcoreapp3.1 126μs 49.5ns 185ns 0 0 0 4.31 KB
#7692 EnrichedLog net472 167μs 34.5ns 133ns 0 0 0 4.52 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.94μs 4.58ns 17.7ns 0 0 0 2.26 KB
master EnrichedLog netcoreapp3.1 6.72μs 23.9ns 92.6ns 0 0 0 2.26 KB
master EnrichedLog net472 7.89μs 5.75ns 21.5ns 0.316 0 0 2.08 KB
#7692 EnrichedLog net6.0 5.02μs 3.76ns 14.6ns 0 0 0 2.26 KB
#7692 EnrichedLog netcoreapp3.1 6.83μs 27.3ns 106ns 0 0 0 2.26 KB
#7692 EnrichedLog net472 7.69μs 5.24ns 19.6ns 0.309 0 0 2.08 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 2μs 1.43ns 5.36ns 0 0 0 1.2 KB
master SendReceive netcoreapp3.1 2.58μs 13ns 58.2ns 0 0 0 1.2 KB
master SendReceive net472 3.26μs 4.35ns 16.3ns 0.178 0 0 1.2 KB
#7692 SendReceive net6.0 1.96μs 8.41ns 32.6ns 0 0 0 1.2 KB
#7692 SendReceive netcoreapp3.1 2.63μs 3.33ns 12.5ns 0 0 0 1.2 KB
#7692 SendReceive net472 3.22μs 5.47ns 21.2ns 0.177 0 0 1.2 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.27μs 16.9ns 65.5ns 0 0 0 1.58 KB
master EnrichedLog netcoreapp3.1 5.57μs 16.1ns 58.1ns 0 0 0 1.63 KB
master EnrichedLog net472 6.59μs 6.93ns 26.9ns 0.295 0 0 2.03 KB
#7692 EnrichedLog net6.0 4.3μs 3.76ns 14.6ns 0 0 0 1.58 KB
#7692 EnrichedLog netcoreapp3.1 5.78μs 15.5ns 60.1ns 0 0 0 1.63 KB
#7692 EnrichedLog net472 6.66μs 6.66ns 25.8ns 0.299 0 0 2.03 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 756ns 3.58ns 13.9ns 0 0 0 576 B
master StartFinishSpan netcoreapp3.1 970ns 4.32ns 16.7ns 0 0 0 576 B
master StartFinishSpan net472 976ns 1.59ns 6.16ns 0.0884 0 0 578 B
master StartFinishScope net6.0 892ns 4.86ns 28.8ns 0 0 0 696 B
master StartFinishScope netcoreapp3.1 1.21μs 6.13ns 28.1ns 0 0 0 696 B
master StartFinishScope net472 1.16μs 1.32ns 4.95ns 0.0996 0 0 658 B
#7692 StartFinishSpan net6.0 748ns 3.78ns 16.5ns 0 0 0 576 B
#7692 StartFinishSpan netcoreapp3.1 935ns 4.24ns 17ns 0 0 0 576 B
#7692 StartFinishSpan net472 939ns 0.38ns 1.42ns 0.0894 0 0 578 B
#7692 StartFinishScope net6.0 904ns 0.591ns 2.29ns 0 0 0 696 B
#7692 StartFinishScope netcoreapp3.1 1.17μs 6.4ns 35.1ns 0 0 0 696 B
#7692 StartFinishScope net472 1.12μs 0.0776ns 0.29ns 0.101 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 1.08μs 2.04ns 7.9ns 0 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 1.41μs 6.7ns 27.6ns 0 0 0 696 B
master RunOnMethodBegin net472 1.49μs 0.871ns 3.37ns 0.105 0 0 658 B
#7692 RunOnMethodBegin net6.0 1.05μs 5.08ns 20.3ns 0 0 0 696 B
#7692 RunOnMethodBegin netcoreapp3.1 1.47μs 5.36ns 20.8ns 0 0 0 696 B
#7692 RunOnMethodBegin net472 1.4μs 0.659ns 2.47ns 0.0991 0 0 658 B

@dd-trace-dotnet-ci-bot
Copy link

dd-trace-dotnet-ci-bot bot commented Oct 21, 2025

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7692) - mean (74ms)  : 72, 75
     .   : milestone, 74,
    master - mean (74ms)  : 72, 76
     .   : milestone, 74,

    section Baseline
    This PR (7692) - mean (70ms)  : 68, 72
     .   : milestone, 70,
    master - mean (70ms)  : 68, 72
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (7692) - mean (1,067ms)  : 1001, 1133
     .   : milestone, 1067,
    master - mean (1,059ms)  : 1002, 1116
     .   : milestone, 1059,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7692) - mean (109ms)  : 107, 111
     .   : milestone, 109,
    master - mean (110ms)  : 107, 112
     .   : milestone, 110,

    section Baseline
    This PR (7692) - mean (109ms)  : 106, 111
     .   : milestone, 109,
    master - mean (109ms)  : 107, 111
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (7692) - mean (752ms)  : 727, 778
     .   : milestone, 752,
    master - mean (755ms)  : 734, 776
     .   : milestone, 755,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7692) - mean (97ms)  : 95, 98
     .   : milestone, 97,
    master - mean (98ms)  : 96, 99
     .   : milestone, 98,

    section Baseline
    This PR (7692) - mean (96ms)  : 94, 98
     .   : milestone, 96,
    master - mean (97ms)  : 95, 100
     .   : milestone, 97,

    section CallTarget+Inlining+NGEN
    This PR (7692) - mean (732ms)  : 663, 800
     .   : milestone, 732,
    master - mean (713ms)  : 693, 733
     .   : milestone, 713,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7692) - mean (96ms)  : 93, 98
     .   : milestone, 96,
    master - mean (96ms)  : 94, 98
     .   : milestone, 96,

    section Baseline
    This PR (7692) - mean (95ms)  : 92, 98
     .   : milestone, 95,
    master - mean (95ms)  : 93, 98
     .   : milestone, 95,

    section CallTarget+Inlining+NGEN
    This PR (7692) - mean (668ms)  : 649, 687
     .   : milestone, 668,
    master - mean (678ms)  : 664, 692
     .   : milestone, 678,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7692) - mean (196ms)  : 194, 198
     .   : milestone, 196,
    master - mean (197ms)  : 194, 199
     .   : milestone, 197,

    section Baseline
    This PR (7692) - mean (194ms)  : 190, 197
     .   : milestone, 194,
    master - mean (194ms)  : 189, 198
     .   : milestone, 194,

    section CallTarget+Inlining+NGEN
    This PR (7692) - mean (1,168ms)  : 1110, 1227
     .   : milestone, 1168,
    master - mean (1,176ms)  : 1101, 1250
     .   : milestone, 1176,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7692) - mean (278ms)  : 274, 282
     .   : milestone, 278,
    master - mean (277ms)  : 273, 282
     .   : milestone, 277,

    section Baseline
    This PR (7692) - mean (276ms)  : 272, 281
     .   : milestone, 276,
    master - mean (276ms)  : 272, 281
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (7692) - mean (947ms)  : 896, 998
     .   : milestone, 947,
    master - mean (946ms)  : 906, 986
     .   : milestone, 946,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7692) - mean (271ms)  : 266, 275
     .   : milestone, 271,
    master - mean (276ms)  : 269, 284
     .   : milestone, 276,

    section Baseline
    This PR (7692) - mean (270ms)  : 266, 274
     .   : milestone, 270,
    master - mean (275ms)  : 268, 282
     .   : milestone, 275,

    section CallTarget+Inlining+NGEN
    This PR (7692) - mean (927ms)  : 884, 970
     .   : milestone, 927,
    master - mean (934ms)  : 885, 984
     .   : milestone, 934,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7692) - mean (270ms)  : 267, 273
     .   : milestone, 270,
    master - mean (270ms)  : 266, 274
     .   : milestone, 270,

    section Baseline
    This PR (7692) - mean (270ms)  : 266, 274
     .   : milestone, 270,
    master - mean (269ms)  : 264, 275
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (7692) - mean (852ms)  : 835, 869
     .   : milestone, 852,
    master - mean (860ms)  : 839, 882
     .   : milestone, 860,
Loading

@bouwkast bouwkast merged commit bd65b17 into master Oct 21, 2025
71 of 74 checks passed
@bouwkast bouwkast deleted the steven/fix-ci branch October 21, 2025 22:05
@github-actions github-actions bot added this to the vNext-v3 milestone Oct 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewlock andrewlock andrewlock approved these changes

@link04 link04 link04 approved these changes

Assignees

No one assigned

Labels

area:builds project files, build scripts, pipelines, versioning, releases, packages type:flake-fix

Projects

None yet

Milestone

3.30.0

Development

Successfully merging this pull request may close these issues.

3 participants

@bouwkast @andrewlock @link04