Skip to content

Benchmarks: Disable Benchmarks.OpenTelemetry.Api from running in the RunBenchmarks Nuke build target#6980

Merged
zacharycmontoya merged 1 commit intomasterfrom
zach.montoya/update-otel-span-api-benchmark
May 15, 2025
Merged

Benchmarks: Disable Benchmarks.OpenTelemetry.Api from running in the RunBenchmarks Nuke build target#6980
zacharycmontoya merged 1 commit intomasterfrom
zach.montoya/update-otel-span-api-benchmark

Conversation

@zacharycmontoya
Copy link
Contributor

@zacharycmontoya zacharycmontoya commented May 15, 2025

Summary of changes

Self-titled.

Reason for change

The previous PR #6968 added two new projects to measure the overhead we introduce when instrumenting the OpenTelemetry Tracing API: Benchmarks.OpenTelemetry.Api and Benchmarks.OpenTelemetry.InstrumentedApi. This exceeds the timeout of our current microbenchmarks CI stage so disable Benchmarks.OpenTelemetry.Api since that project doesn't run any Datadog components and won't be changing any time soon.

Implementation details

Test coverage

Other details

@zacharycmontoya
Benchmarks: Temporarily disable the OpenTelemetry.Api benchmark, but …
4b8976d 
…still keep the OpenTelemetry.InstrumentedApi benchmark enabled
@zacharycmontoya zacharycmontoya requested a review from a team as a code owner May 15, 2025 18:48
@github-actions github-actions bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label May 15, 2025
@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented May 15, 2025 •
edited
Loading

Datadog Report

Branch report: zach.montoya/update-otel-span-api-benchmark
Commit report: 4b8976d
Test service: dd-trace-dotnet

✅ 0 Failed, 259276 Passed, 2341 Skipped, 17h 45m 34.56s Total Time

@andrewlock
Copy link
Member

andrewlock commented May 15, 2025

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6980) - mean (68ms)  : 65, 71
     .   : milestone, 68,
    master - mean (68ms)  : 63, 73
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (6980) - mean (1,013ms)  : 979, 1047
     .   : milestone, 1013,
    master - mean (1,006ms)  : 983, 1029
     .   : milestone, 1006,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6980) - mean (103ms)  : 101, 104
     .   : milestone, 103,
    master - mean (103ms)  : 100, 105
     .   : milestone, 103,

    section CallTarget+Inlining+NGEN
    This PR (6980) - mean (698ms)  : 680, 715
     .   : milestone, 698,
    master - mean (693ms)  : 673, 714
     .   : milestone, 693,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6980) - mean (89ms)  : 87, 91
     .   : milestone, 89,
    master - mean (89ms)  : 88, 91
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (6980) - mean (655ms)  : 636, 674
     .   : milestone, 655,
    master - mean (656ms)  : 629, 684
     .   : milestone, 656,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6980) - mean (189ms)  : 186, 192
     .   : milestone, 189,
    master - mean (191ms)  : 185, 196
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6980) - mean (1,112ms)  : 1088, 1137
     .   : milestone, 1112,
    master - mean (1,108ms)  : 1080, 1137
     .   : milestone, 1108,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6980) - mean (268ms)  : 263, 274
     .   : milestone, 268,
    master - mean (268ms)  : 264, 271
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (6980) - mean (877ms)  : 850, 905
     .   : milestone, 877,
    master - mean (876ms)  : 847, 906
     .   : milestone, 876,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6980) - mean (261ms)  : 258, 265
     .   : milestone, 261,
    master - mean (261ms)  : 258, 264
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (6980) - mean (868ms)  : 840, 895
     .   : milestone, 868,
    master - mean (864ms)  : 837, 892
     .   : milestone, 864,
Loading

@pr-commenter
Copy link

pr-commenter bot commented May 15, 2025

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #99 compared to master:

  • 2 benchmarks are slower, with geometric mean 1.190
  • 2 benchmarks have fewer allocations
  • 6 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #99

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net472 6.1 KB 6.18 KB 74 B 1.21%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 10.8μs 57.6ns 310ns 0 0 0 5.55 KB
master StartStopWithChild netcoreapp3.1 13.5μs 65.1ns 284ns 0 0 0 5.75 KB
master StartStopWithChild net472 21.5μs 117ns 693ns 0.987 0.329 0.11 6.1 KB
#99 StartStopWithChild net6.0 10.7μs 59.9ns 384ns 0 0 0 5.53 KB
#99 StartStopWithChild netcoreapp3.1 14.9μs 76.6ns 375ns 0 0 0 5.77 KB
#99 StartStopWithChild net472 22.4μs 121ns 663ns 0.968 0.215 0 6.18 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #99

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 3.31 KB 3.35 KB 46 B 1.39%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 937μs 167ns 579ns 0 0 0 2.71 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 1.02ms 62.4ns 233ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 1.21ms 693ns 2.68μs 0 0 0 3.31 KB
#99 WriteAndFlushEnrichedTraces net6.0 947μs 68.3ns 246ns 0 0 0 2.71 KB
#99 WriteAndFlushEnrichedTraces netcoreapp3.1 1.04ms 110ns 395ns 0 0 0 2.7 KB
#99 WriteAndFlushEnrichedTraces net472 1.24ms 352ns 1.36μs 0 0 0 3.35 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #99

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net472 236.35 KB 237.89 KB 1.54 KB 0.65%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net472 239.88 KB 241.41 KB 1.54 KB 0.64%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 329μs 1.33μs 5.14μs 0 0 0 197.06 KB
master AllCycleSimpleBody netcoreapp3.1 483μs 1.89μs 7.34μs 0 0 0 204.77 KB
master AllCycleSimpleBody net472 438μs 147ns 529ns 36.6 2.16 0 236.35 KB
master AllCycleMoreComplexBody net6.0 339μs 361ns 1.4μs 0 0 0 200.56 KB
master AllCycleMoreComplexBody netcoreapp3.1 490μs 2.28μs 8.81μs 0 0 0 208.18 KB
master AllCycleMoreComplexBody net472 447μs 129ns 499ns 37.9 2.23 0 239.88 KB
master ObjectExtractorSimpleBody net6.0 309ns 1.81ns 12.7ns 0 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 394ns 2.06ns 10.1ns 0 0 0 272 B
master ObjectExtractorSimpleBody net472 302ns 0.0754ns 0.282ns 0.0441 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 6.25μs 29.6ns 115ns 0 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 7.71μs 37.2ns 144ns 0 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 6.81μs 7.65ns 29.6ns 0.578 0 0 3.8 KB
#99 AllCycleSimpleBody net6.0 327μs 1.55μs 6.21μs 0 0 0 197.06 KB
#99 AllCycleSimpleBody netcoreapp3.1 474μs 2.16μs 8.36μs 0 0 0 204.77 KB
#99 AllCycleSimpleBody net472 442μs 111ns 399ns 36.6 2.16 0 237.89 KB
#99 AllCycleMoreComplexBody net6.0 335μs 210ns 814ns 0 0 0 200.56 KB
#99 AllCycleMoreComplexBody netcoreapp3.1 482μs 970ns 3.63μs 0 0 0 208.18 KB
#99 AllCycleMoreComplexBody net472 445μs 133ns 517ns 37.9 2.23 0 241.41 KB
#99 ObjectExtractorSimpleBody net6.0 316ns 1.63ns 8.13ns 0 0 0 280 B
#99 ObjectExtractorSimpleBody netcoreapp3.1 401ns 2.09ns 10ns 0 0 0 272 B
#99 ObjectExtractorSimpleBody net472 302ns 0.012ns 0.0448ns 0.0441 0 0 281 B
#99 ObjectExtractorMoreComplexBody net6.0 6.29μs 33.5ns 183ns 0 0 0 3.78 KB
#99 ObjectExtractorMoreComplexBody netcoreapp3.1 7.65μs 33ns 128ns 0 0 0 3.69 KB
#99 ObjectExtractorMoreComplexBody net472 6.69μs 1.84ns 7.13ns 0.601 0 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 N/A N/A N/A NaN NaN NaN 0 b
master EncodeArgs netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
master EncodeArgs net472 N/A N/A N/A NaN NaN NaN 0 b
master EncodeLegacyArgs net6.0 N/A N/A N/A NaN NaN NaN 0 b
master EncodeLegacyArgs netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
master EncodeLegacyArgs net472 N/A N/A N/A NaN NaN NaN 0 b
#99 EncodeArgs net6.0 N/A N/A N/A NaN NaN NaN 0 b
#99 EncodeArgs netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
#99 EncodeArgs net472 N/A N/A N/A NaN NaN NaN 0 b
#99 EncodeLegacyArgs net6.0 N/A N/A N/A NaN NaN NaN 0 b
#99 EncodeLegacyArgs netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
#99 EncodeLegacyArgs net472 N/A N/A N/A NaN NaN NaN 0 b
Benchmarks.Trace.Asm.AppSecWafBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 N/A N/A N/A NaN NaN NaN 0 b
master RunWafRealisticBenchmark netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
master RunWafRealisticBenchmark net472 N/A N/A N/A NaN NaN NaN 0 b
master RunWafRealisticBenchmarkWithAttack net6.0 N/A N/A N/A NaN NaN NaN 0 b
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
master RunWafRealisticBenchmarkWithAttack net472 N/A N/A N/A NaN NaN NaN 0 b
#99 RunWafRealisticBenchmark net6.0 N/A N/A N/A NaN NaN NaN 0 b
#99 RunWafRealisticBenchmark netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
#99 RunWafRealisticBenchmark net472 N/A N/A N/A NaN NaN NaN 0 b
#99 RunWafRealisticBenchmarkWithAttack net6.0 N/A N/A N/A NaN NaN NaN 0 b
#99 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 N/A N/A N/A NaN NaN NaN 0 b
#99 RunWafRealisticBenchmarkWithAttack net472 N/A N/A N/A NaN NaN NaN 0 b
Benchmarks.Trace.AspNetCoreBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #99

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AspNetCoreBenchmark.SendRequest‑net6.0 1.217 61,051.53 74,311.06

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 61.1μs 116ns 401ns 0 0 0 14.53 KB
master SendRequest netcoreapp3.1 72.9μs 314ns 1.33μs 0 0 0 17.42 KB
master SendRequest net472 0.024ns 0.00267ns 0.00998ns 0 0 0 0 b
#99 SendRequest net6.0 74.3μs 27.1ns 93.7ns 0 0 0 14.53 KB
#99 SendRequest netcoreapp3.1 70.8μs 168ns 606ns 0 0 0 17.43 KB
#99 SendRequest net472 0.0224ns 0.00286ns 0.0111ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 659μs 1.34μs 5.19μs 0 0 0 41.55 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 635μs 3.08μs 12.7μs 0 0 0 41.97 KB
master WriteAndFlushEnrichedTraces net472 877μs 2.87μs 11.1μs 4.46 0 0 55.72 KB
#99 WriteAndFlushEnrichedTraces net6.0 732μs 846ns 3.28μs 0 0 0 41.73 KB
#99 WriteAndFlushEnrichedTraces netcoreapp3.1 664μs 3.3μs 14.4μs 0 0 0 41.94 KB
#99 WriteAndFlushEnrichedTraces net472 886μs 1.41μs 5.45μs 4.46 0 0 55.82 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.97μs 5.61ns 21ns 0 0 0 1.03 KB
master ExecuteNonQuery netcoreapp3.1 2.44μs 8.35ns 31.2ns 0 0 0 1.02 KB
master ExecuteNonQuery net472 2.74μs 3.3ns 12.8ns 0.151 0.0137 0 995 B
#99 ExecuteNonQuery net6.0 1.88μs 8.83ns 33ns 0 0 0 1.03 KB
#99 ExecuteNonQuery netcoreapp3.1 2.38μs 3.27ns 12.6ns 0 0 0 1.02 KB
#99 ExecuteNonQuery net472 2.79μs 2.17ns 8.39ns 0.151 0.0137 0 995 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.77μs 8.81ns 38.4ns 0 0 0 1.04 KB
master CallElasticsearch netcoreapp3.1 2.33μs 8.72ns 33.8ns 0 0 0 1.04 KB
master CallElasticsearch net472 3.45μs 2.91ns 11.3ns 0.154 0 0 1.05 KB
master CallElasticsearchAsync net6.0 1.79μs 3.34ns 13ns 0 0 0 1.02 KB
master CallElasticsearchAsync netcoreapp3.1 2.36μs 9.7ns 37.6ns 0 0 0 1.09 KB
master CallElasticsearchAsync net472 3.8μs 3.3ns 12.4ns 0.172 0 0 1.11 KB
#99 CallElasticsearch net6.0 1.81μs 9.04ns 42.4ns 0 0 0 1.04 KB
#99 CallElasticsearch netcoreapp3.1 2.25μs 3.64ns 13.6ns 0 0 0 1.04 KB
#99 CallElasticsearch net472 3.57μs 4.42ns 17.1ns 0.159 0 0 1.05 KB
#99 CallElasticsearchAsync net6.0 1.88μs 8.54ns 33.1ns 0 0 0 1.02 KB
#99 CallElasticsearchAsync netcoreapp3.1 2.32μs 10.4ns 40.2ns 0 0 0 1.09 KB
#99 CallElasticsearchAsync net472 3.75μs 3.02ns 11.7ns 0.17 0 0 1.11 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.88μs 1.31ns 5.06ns 0 0 0 960 B
master ExecuteAsync netcoreapp3.1 2.27μs 1.37ns 4.73ns 0 0 0 960 B
master ExecuteAsync net472 2.56μs 1.83ns 7.08ns 0.141 0 0 923 B
#99 ExecuteAsync net6.0 1.82μs 6.25ns 23.4ns 0 0 0 960 B
#99 ExecuteAsync netcoreapp3.1 2.46μs 8.31ns 32.2ns 0 0 0 960 B
#99 ExecuteAsync net472 2.64μs 0.922ns 3.57ns 0.144 0 0 923 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 7.02μs 12.3ns 44.4ns 0 0 0 2.37 KB
master SendAsync netcoreapp3.1 8.54μs 4.89ns 18.3ns 0 0 0 2.9 KB
master SendAsync net472 12.2μs 5.24ns 20.3ns 0.487 0 0 3.19 KB
#99 SendAsync net6.0 6.85μs 13.5ns 52.4ns 0 0 0 2.37 KB
#99 SendAsync netcoreapp3.1 8.86μs 10.9ns 42.2ns 0 0 0 2.9 KB
#99 SendAsync net472 12.4μs 12.9ns 48.4ns 0.496 0 0 3.19 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Slower ⚠️ More allocations ⚠️

Slower ⚠️ in #99

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net6.0 1.164 42,700.00 49,700.00

More allocations ⚠️ in #99

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 257.33 KB 272.44 KB 15.11 KB 5.87%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0 276.46 KB 280.53 KB 4.07 KB 1.47%

Fewer allocations 🎉 in #99

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net472 281.47 KB 278.53 KB -2.94 KB -1.05%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 65.54 KB 57.34 KB -8.19 KB -12.50%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 43.2μs 230ns 1.2μs 0 0 0 43.82 KB
master StringConcatBenchmark netcoreapp3.1 50.9μs 289ns 2.63μs 0 0 0 42.72 KB
master StringConcatBenchmark net472 56.4μs 253ns 948ns 0 0 0 65.54 KB
master StringConcatAspectBenchmark net6.0 469μs 2.06μs 7.72μs 0 0 0 276.46 KB
master StringConcatAspectBenchmark netcoreapp3.1 499μs 2.15μs 7.45μs 0 0 0 257.33 KB
master StringConcatAspectBenchmark net472 410μs 1.73μs 7.91μs 0 0 0 281.47 KB
#99 StringConcatBenchmark net6.0 49.9μs 284ns 1.84μs 0 0 0 43.81 KB
#99 StringConcatBenchmark netcoreapp3.1 45.8μs 224ns 951ns 0 0 0 42.64 KB
#99 StringConcatBenchmark net472 56.5μs 254ns 951ns 0 0 0 57.34 KB
#99 StringConcatAspectBenchmark net6.0 513μs 1.57μs 5.67μs 0 0 0 280.53 KB
#99 StringConcatAspectBenchmark netcoreapp3.1 464μs 6.88μs 68.1μs 0 0 0 272.44 KB
#99 StringConcatAspectBenchmark net472 416μs 2.18μs 11.5μs 0 0 0 278.53 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.62μs 13ns 59.5ns 0 0 0 1.76 KB
master EnrichedLog netcoreapp3.1 3.55μs 13.1ns 50.6ns 0 0 0 1.76 KB
master EnrichedLog net472 3.84μs 4.48ns 17.3ns 0.251 0 0 1.69 KB
#99 EnrichedLog net6.0 2.56μs 2.75ns 10.6ns 0 0 0 1.76 KB
#99 EnrichedLog netcoreapp3.1 3.55μs 17.9ns 79.9ns 0 0 0 1.76 KB
#99 EnrichedLog net472 3.87μs 2.13ns 8.26ns 0.25 0 0 1.69 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 125μs 97.3ns 364ns 0 0 0 4.37 KB
master EnrichedLog netcoreapp3.1 128μs 244ns 945ns 0 0 0 4.37 KB
master EnrichedLog net472 168μs 130ns 502ns 0 0 0 4.57 KB
#99 EnrichedLog net6.0 123μs 94.5ns 366ns 0 0 0 4.37 KB
#99 EnrichedLog netcoreapp3.1 128μs 380ns 1.47μs 0 0 0 4.37 KB
#99 EnrichedLog net472 167μs 150ns 540ns 0 0 0 4.57 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.93μs 6.24ns 24.2ns 0 0 0 2.32 KB
master EnrichedLog netcoreapp3.1 6.77μs 12.1ns 47ns 0 0 0 2.32 KB
master EnrichedLog net472 7.23μs 6.23ns 23.3ns 0.325 0 0 2.14 KB
#99 EnrichedLog net6.0 4.91μs 23.3ns 90.4ns 0 0 0 2.32 KB
#99 EnrichedLog netcoreapp3.1 6.47μs 10.5ns 40.6ns 0 0 0 2.32 KB
#99 EnrichedLog net472 7.26μs 6.76ns 26.2ns 0.328 0 0 2.14 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 2.02μs 2.25ns 8.7ns 0 0 0 1.21 KB
master SendReceive netcoreapp3.1 2.6μs 12.2ns 47.2ns 0 0 0 1.21 KB
master SendReceive net472 3.05μs 2.22ns 8.61ns 0.183 0 0 1.21 KB
#99 SendReceive net6.0 1.99μs 9.59ns 39.5ns 0 0 0 1.21 KB
#99 SendReceive netcoreapp3.1 2.57μs 12.7ns 56.8ns 0 0 0 1.21 KB
#99 SendReceive net472 3.06μs 4.35ns 16.9ns 0.183 0 0 1.21 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.17μs 10.8ns 38.9ns 0 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 5.63μs 19.1ns 73.8ns 0 0 0 1.69 KB
master EnrichedLog net472 6.56μs 5.63ns 21.8ns 0.328 0 0 2.08 KB
#99 EnrichedLog net6.0 4.22μs 16.4ns 63.6ns 0 0 0 1.64 KB
#99 EnrichedLog netcoreapp3.1 5.66μs 6.99ns 27.1ns 0 0 0 1.69 KB
#99 EnrichedLog net472 6.53μs 7.49ns 29ns 0.327 0 0 2.08 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 747ns 0.338ns 1.31ns 0 0 0 584 B
master StartFinishSpan netcoreapp3.1 929ns 4.78ns 22.9ns 0 0 0 584 B
master StartFinishSpan net472 920ns 0.364ns 1.41ns 0.0918 0 0 586 B
master StartFinishScope net6.0 870ns 4.97ns 27.2ns 0 0 0 704 B
master StartFinishScope netcoreapp3.1 1.19μs 5.84ns 24.8ns 0 0 0 704 B
master StartFinishScope net472 1.14μs 0.169ns 0.609ns 0.102 0 0 666 B
#99 StartFinishSpan net6.0 737ns 4.06ns 23.7ns 0 0 0 584 B
#99 StartFinishSpan netcoreapp3.1 950ns 4.52ns 18.1ns 0 0 0 584 B
#99 StartFinishSpan net472 925ns 0.44ns 1.7ns 0.0924 0 0 586 B
#99 StartFinishScope net6.0 891ns 0.325ns 1.22ns 0 0 0 704 B
#99 StartFinishScope netcoreapp3.1 1.17μs 5.78ns 21.6ns 0 0 0 704 B
#99 StartFinishScope net472 1.12μs 0.143ns 0.554ns 0.101 0 0 666 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 1.06μs 0.417ns 1.61ns 0 0 0 704 B
master RunOnMethodBegin netcoreapp3.1 1.33μs 6.52ns 26.1ns 0 0 0 704 B
master RunOnMethodBegin net472 1.37μs 1.03ns 4.01ns 0.103 0 0 666 B
#99 RunOnMethodBegin net6.0 1.03μs 0.213ns 0.797ns 0 0 0 704 B
#99 RunOnMethodBegin netcoreapp3.1 1.37μs 6.55ns 27ns 0 0 0 704 B
#99 RunOnMethodBegin net472 1.36μs 0.641ns 2.48ns 0.101 0 0 666 B

@zacharycmontoya zacharycmontoya merged commit 56e25f3 into master May 15, 2025
126 of 127 checks passed
@zacharycmontoya zacharycmontoya deleted the zach.montoya/update-otel-span-api-benchmark branch May 15, 2025 20:19
@github-actions github-actions bot added this to the vNext-v3 milestone May 15, 2025
@andrewlock
Copy link
Member

andrewlock commented May 16, 2025

Similarly, still no benchmarks posted? 😕

chojomok pushed a commit that referenced this pull request Jul 15, 2025
@zacharycmontoya @chojomok
Benchmarks: Disable Benchmarks.OpenTelemetry.Api from running in the …
bf68816 
…`RunBenchmarks` Nuke build target (#6980)

The previous PR #6968
added two new projects to measure the overhead we introduce when
instrumenting the OpenTelemetry Tracing API:
`Benchmarks.OpenTelemetry.Api` and
`Benchmarks.OpenTelemetry.InstrumentedApi`. This exceeds the timeout of
our current microbenchmarks CI stage so disable
`Benchmarks.OpenTelemetry.Api` since that project doesn't run any
Datadog components and won't be changing any time soon.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bouwkast bouwkast bouwkast approved these changes

Assignees

No one assigned

Labels

area:builds project files, build scripts, pipelines, versioning, releases, packages

Projects

None yet

Milestone

3.17.0

Development

Successfully merging this pull request may close these issues.

3 participants

@zacharycmontoya @andrewlock @bouwkast