Add new benchmark projects to microbenchmark OpenTelemetry Tracing APIs by zacharycmontoya · Pull Request #6968 · DataDog/dd-trace-dotnet

zacharycmontoya · 2025-05-13T22:11:37Z

Summary of changes

Adds two new projects to microbenchmark the OpenTelemetry Tracing APIs, configured to run with the existing RunBenchmarks Nuke task

Benchmarks.OpenTelemetry.Api - Baseline behavior using the OpenTelemetry APIs with the OpenTelemetry SDK
Benchmarks.OpenTelemetry.InstrumentedApi - New behavior using the OpenTelemetry APIs with the Datadog SDK

Both projects test the following APIs:

System.Diagnostics.Activity (built-in .NET) and OpenTelemetry.Trace.TelemetrySpan (OTEL API):
- (baseline)
- AddEvent
- GetContext
- RecordException (only implemented in Activity for .NET 9+)
- SetStatus
- SetAttributes
- UpdateName
OpenTelemetry.Trace.Tracer
- StartSpan
- StartActiveSpan
- StartRootSpan
- GetCurrentSpan
- SetActive

Reason for change

We want to begin measuring the relative overhead we introduce over the OpenTelemetry SDK so we can track and improve it.

Implementation details

Adds the two BenchmarkDotNet projects Benchmarks.OpenTelemetry.Api and Benchmarks.OpenTelemetry.InstrumentedApi. Both share the same benchmark test code by importing the source files in the second project, which means the code under test is identical and only uses the same OpenTelemetry Tracing API.

When the BenchmarkDotNet GlobalSetup is called for the OpenTelemetry SDK test classes, the OpenTelemetry SDK is configured with specific TracerProviders.

When the BenchmarkDotNet GlobalSetup is called for the Datadog SDK test classes, we call Datadog.Trace.Activity.ActivityListener.Initialize() to ensure that our ActivityListener (which converts Activity objects to Datadog spans) is initialized.

Test coverage

This is the test coverage 😉

Other details

Internal dashboard with results from local machine can be found here where the corrected setup begins reporting on May 12.

APMAPI-1405

1. OTEL Tracing API with the OTEL SDK (Benchmarks.OpenTelemetry.Api) 2. OTEL Tracing API with the Datadog SDK (Benchmarks.OpenTelemetry.InstrumentedApi) So far there are two benchmarks whose code is shared in each project (since they only reference the OTEL Tracing API): - TracerBenchmark for operations on the OTEL Tracer class - TelemetrySpanBenchmark for operations on the OTEL Span class

…rn off Datadog Agent communications so we can perform more controlled API microbenchmarks

… Benchmark types with different fullnames, even though the source is shared between the Benchmarks.OpenTelemetry.Api and Benchmarks.OpenTelemetry.InstrumentedApi. The result is that BenchmarkDotNet now reports the test cases with different full names so we can differentiate them in the Datadog backend.

…rojects

…ce I had to remove DD_TRACE_ENABLED=false, so this may skew results

…penTelemetry SDK, we are really only concerned with the common case of recorded spans Delete NotSampled tests since our implementation of the Tracing API will always mark Activity objects to be sampled, then we do our own sampling decision from the Datadog side. While this does create more overhead when NotSampled, this isn't currently a comparison that provides valuable insight.

…overyService client when CI Visibility is enabled

…eaching the DiscoveryService and RemoteConfig when using CI Visibility

…rget

datadog-datadog-prod-us1 · 2025-05-13T22:43:40Z

Datadog Report

All test runs c58b24b 🔗

✅ 2 Total Test Services: 0 Failed, 2 Passed

Test Services

Service Name	Failed	Known Flaky	New Flaky	Passed	Skipped	Total Time	Test Service View
`dd-trace-dotnet`	0	0	0	248323	2263	17h 36m 57.18s	Link
`exploration_tests`	0	0	0	22085	3	2m 9.73s	Link

andrewlock · 2025-05-13T22:47:18Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6968) - mean (68ms)  : 66, 70
     .   : milestone, 68,
    master - mean (68ms)  : 64, 72
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (6968) - mean (1,004ms)  : 978, 1030
     .   : milestone, 1004,
    master - mean (1,004ms)  : 978, 1029
     .   : milestone, 1004,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6968) - mean (102ms)  : 100, 103
     .   : milestone, 102,
    master - mean (102ms)  : 100, 104
     .   : milestone, 102,

    section CallTarget+Inlining+NGEN
    This PR (6968) - mean (688ms)  : 672, 705
     .   : milestone, 688,
    master - mean (693ms)  : 675, 711
     .   : milestone, 693,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6968) - mean (89ms)  : 86, 91
     .   : milestone, 89,
    master - mean (89ms)  : 87, 91
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (6968) - mean (646ms)  : 624, 668
     .   : milestone, 646,
    master - mean (653ms)  : 628, 677
     .   : milestone, 653,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6968) - mean (189ms)  : 182, 196
     .   : milestone, 189,
    master - mean (189ms)  : 185, 192
     .   : milestone, 189,

    section CallTarget+Inlining+NGEN
    This PR (6968) - mean (1,109ms)  : 1082, 1137
     .   : milestone, 1109,
    master - mean (1,108ms)  : 1078, 1137
     .   : milestone, 1108,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6968) - mean (268ms)  : 265, 271
     .   : milestone, 268,
    master - mean (268ms)  : 264, 272
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (6968) - mean (870ms)  : 836, 904
     .   : milestone, 870,
    master - mean (877ms)  : 848, 906
     .   : milestone, 877,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6968) - mean (262ms)  : 258, 266
     .   : milestone, 262,
    master - mean (261ms)  : 258, 263
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (6968) - mean (859ms)  : 827, 892
     .   : milestone, 859,
    master - mean (864ms)  : 838, 891
     .   : milestone, 864,

bouwkast

Nice LGTM!

tracer/test/benchmarks/Benchmarks.OpenTelemetry.Api/Trace/ActivityBenchmark.cs

… if running on .NET 9+ since the built-in Activity has a new AddException API

…`RunBenchmarks` Nuke build target (#6980) The previous PR #6968 added two new projects to measure the overhead we introduce when instrumenting the OpenTelemetry Tracing API: `Benchmarks.OpenTelemetry.Api` and `Benchmarks.OpenTelemetry.InstrumentedApi`. This exceeds the timeout of our current microbenchmarks CI stage so disable `Benchmarks.OpenTelemetry.Api` since that project doesn't run any Datadog components and won't be changing any time soon.

andrewlock · 2025-05-16T07:53:21Z

...rks/Benchmarks.OpenTelemetry.InstrumentedApi/Benchmarks.OpenTelemetry.InstrumentedApi.csproj

+    <RootNamespace>Benchmarks.OpenTelemetry.InstrumentedApi</RootNamespace>
+
+    <!-- benchmarkdotnet only support numberic values, not "latest"-->
+    <LangVersion>11</LangVersion>


y no 13? 😢

Oh I copy/pasted this from the Benchmark.Trace project, I can update this later

andrewlock · 2025-05-16T07:54:37Z

The fact that the results of the benchmarks weren't posted to this PR is concerning 😬

…Is (#6968) Adds two new projects to microbenchmark the OpenTelemetry Tracing APIs configured to run with the existing `RunBenchmarks` Nuke task: - `Benchmarks.OpenTelemetry.Api` - Baseline behavior using the OpenTelemetry APIs with the OpenTelemetry SDK - `Benchmarks.OpenTelemetry.InstrumentedApi` - New behavior using the OpenTelemetry APIs with the Datadog SDK Both projects test the following APIs: - `System.Diagnostics.Activity` (built-in .NET) and `OpenTelemetry.Trace.TelemetrySpan` (OTEL API): - (baseline) - `AddEvent` - `GetContext` - `RecordException` (only implemented in Activity for .NET 9+) - `SetStatus` - `SetAttributes` - `UpdateName` - `OpenTelemetry.Trace.Tracer` - `StartSpan` - `StartActiveSpan` - `StartRootSpan` - `GetCurrentSpan` - `SetActive`

…`RunBenchmarks` Nuke build target (#6980) The previous PR #6968 added two new projects to measure the overhead we introduce when instrumenting the OpenTelemetry Tracing API: `Benchmarks.OpenTelemetry.Api` and `Benchmarks.OpenTelemetry.InstrumentedApi`. This exceeds the timeout of our current microbenchmarks CI stage so disable `Benchmarks.OpenTelemetry.Api` since that project doesn't run any Datadog components and won't be changing any time soon.

) ## Summary of changes Make sure we run the "OTel API only" benchmarks on master ## Reason for change We created "OTel API only" benchmarks in #6968 to compare with our "DD instrumented OTel API" benchmarks, but we're currently never running them, so we have no data. This ensures we run them on master (only) so we have comparison data. ## Implementation details Check if we're on a PR (which we are already doing). If not, then run the OTel API only benchmarks. ## Test coverage By its nature, hard to confirm it's working exactly as expected. Will confirm after merge

zacharycmontoya added 11 commits May 13, 2025 10:59

Add a DD_INTERNAL_AGENT_STANDALONE_MODE_ENABLED key to allow us to tu…

f0c0248

…rn off Datadog Agent communications so we can perform more controlled API microbenchmarks

Add DatadogExporter to benchmarks

8b18284

Add RunOpenTelemetryBenchmarks nuke target to run the new benchmark p…

706849a

…rojects

Get the benchmarks running, but the agent writer is still running sin…

399e673

…ce I had to remove DD_TRACE_ENABLED=false, so this may skew results

Everything seems to be working for a subset of tests, don't touch

7d2f722

Add DD_CIVISIBILITY_FORCE_AGENT_EVP_PROXY=V4 to avoid creating a Disc…

e7a12bc

…overyService client when CI Visibility is enabled

Remove the InternalStandaloneModeEnabled setting since we can avoid r…

3de8a2b

…eaching the DiscoveryService and RemoteConfig when using CI Visibility

Combine RunOpenTelemetryBenchmarks into the existing RunBenchmarks ta…

7d30098

…rget

zacharycmontoya requested review from a team as code owners May 13, 2025 22:11

lucaspimentel added the type:performance Performance, speed, latency, resource usage (CPU, memory) label May 14, 2025

bouwkast approved these changes May 14, 2025

View reviewed changes

tracer/test/benchmarks/Benchmarks.OpenTelemetry.Api/Trace/ActivityBenchmark.cs Show resolved Hide resolved

Add StartSpan_RecordException_Sampled test case for ActivityBenchmark…

c58b24b

… if running on .NET 9+ since the built-in Activity has a new AddException API

zacharycmontoya merged commit 8a6606e into master May 15, 2025
125 of 129 checks passed

zacharycmontoya deleted the zach.montoya/q2-otel-okr-research branch May 15, 2025 16:56

github-actions bot added this to the vNext-v3 milestone May 15, 2025

zacharycmontoya mentioned this pull request May 15, 2025

Benchmarks: Disable Benchmarks.OpenTelemetry.Api from running in the RunBenchmarks Nuke build target #6980

Merged

andrewlock reviewed May 16, 2025

View reviewed changes

andrewlock mentioned this pull request Dec 24, 2025

Ensure we run the BenchmarksOpenTelemetryApi benchmarks on master #8012

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new benchmark projects to microbenchmark OpenTelemetry Tracing APIs#6968

Add new benchmark projects to microbenchmark OpenTelemetry Tracing APIs#6968
zacharycmontoya merged 12 commits intomasterfrom
zach.montoya/q2-otel-okr-research

zacharycmontoya commented May 13, 2025 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 bot commented May 13, 2025 •

edited

Loading

Uh oh!

andrewlock commented May 13, 2025 •

edited

Loading

Uh oh!

bouwkast left a comment

Uh oh!

Uh oh!

Uh oh!

andrewlock May 16, 2025

Uh oh!

zacharycmontoya May 16, 2025

Uh oh!

andrewlock commented May 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

zacharycmontoya commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Uh oh!

datadog-datadog-prod-us1 bot commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

Uh oh!

andrewlock commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

Uh oh!

bouwkast left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

andrewlock May 16, 2025

Choose a reason for hiding this comment

Uh oh!

zacharycmontoya May 16, 2025

Choose a reason for hiding this comment

Uh oh!

andrewlock commented May 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zacharycmontoya commented May 13, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented May 13, 2025 •

edited

Loading

andrewlock commented May 13, 2025 •

edited

Loading