Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

• This PR (7568)
• master

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

• Welch test with statistical test for significance of 5%
• Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7568) - mean (72ms)  : 71, 73
     .   : milestone, 72,
    master - mean (75ms)  : 74, 76
     .   : milestone, 75,

    section Baseline
    This PR (7568) - mean (68ms)  : 67, 70
     .   : milestone, 68,
    master - mean (71ms)  : 69, 73
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (7568) - mean (1,047ms)  : 1003, 1091
     .   : milestone, 1047,
    master - mean (1,069ms)  : 1007, 1131
     .   : milestone, 1069,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7568) - mean (107ms)  : 105, 108
     .   : milestone, 107,
    master - mean (112ms)  : 109, 114
     .   : milestone, 112,

    section Baseline
    This PR (7568) - mean (106ms)  : 103, 109
     .   : milestone, 106,
    master - mean (111ms)  : 108, 113
     .   : milestone, 111,

    section CallTarget+Inlining+NGEN
    This PR (7568) - mean (745ms)  : 721, 769
     .   : milestone, 745,
    master - mean (761ms)  : 735, 788
     .   : milestone, 761,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7568) - mean (95ms)  : 93, 96
     .   : milestone, 95,
    master - mean (103ms)  : 101, 105
     .   : milestone, 103,

    section Baseline
    This PR (7568) - mean (94ms)  : 92, 96
     .   : milestone, 94,
    master - mean (100ms)  : 96, 104
     .   : milestone, 100,

    section CallTarget+Inlining+NGEN
    This PR (7568) - mean (703ms)  : 667, 739
     .   : milestone, 703,
    master - mean (721ms)  : 681, 761
     .   : milestone, 721,

gantt
    title Execution time (ms) FakeDbCommand (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7568) - mean (94ms)  : 92, 95
     .   : milestone, 94,
    master - mean (98ms)  : 96, 101
     .   : milestone, 98,

    section Baseline
    This PR (7568) - mean (92ms)  : 90, 95
     .   : milestone, 92,
    master - mean (97ms)  : 94, 100
     .   : milestone, 97,

    section CallTarget+Inlining+NGEN
    This PR (7568) - mean (660ms)  : 644, 676
     .   : milestone, 660,
    master - mean (679ms)  : 664, 693
     .   : milestone, 679,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7568) - mean (197ms)  : 194, 200
     .   : milestone, 197,
    master - mean (197ms)  : 194, 200
     .   : milestone, 197,

    section Baseline
    This PR (7568) - mean (194ms)  : 190, 198
     .   : milestone, 194,
    master - mean (194ms)  : 190, 197
     .   : milestone, 194,

    section CallTarget+Inlining+NGEN
    This PR (7568) - mean (1,180ms)  : 1106, 1253
     .   : milestone, 1180,
    master - mean (1,166ms)  : 1111, 1220
     .   : milestone, 1166,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7568) - mean (277ms)  : 274, 281
     .   : milestone, 277,
    master - mean (278ms)  : 274, 281
     .   : milestone, 278,

    section Baseline
    This PR (7568) - mean (276ms)  : 272, 281
     .   : milestone, 276,
    master - mean (278ms)  : 273, 283
     .   : milestone, 278,

    section CallTarget+Inlining+NGEN
    This PR (7568) - mean (943ms)  : 907, 978
     .   : milestone, 943,
    master - mean (947ms)  : 898, 996
     .   : milestone, 947,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7568) - mean (271ms)  : 267, 274
     .   : milestone, 271,
    master - mean (271ms)  : 268, 274
     .   : milestone, 271,

    section Baseline
    This PR (7568) - mean (269ms)  : 266, 273
     .   : milestone, 269,
    master - mean (272ms)  : 266, 278
     .   : milestone, 272,

    section CallTarget+Inlining+NGEN
    This PR (7568) - mean (925ms)  : 869, 981
     .   : milestone, 925,
    master - mean (935ms)  : 873, 996
     .   : milestone, 935,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7568) - mean (271ms)  : 265, 276
     .   : milestone, 271,
    master - mean (271ms)  : 266, 276
     .   : milestone, 271,

    section Baseline
    This PR (7568) - mean (270ms)  : 266, 274
     .   : milestone, 270,
    master - mean (269ms)  : 265, 274
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (7568) - mean (851ms)  : 832, 870
     .   : milestone, 851,
    master - mean (861ms)  : 836, 886
     .   : milestone, 861,

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #7568 compared to master:

• 1 benchmarks are faster, with geometric mean 2.484
• 1 benchmarks are slower, with geometric mean 2.053
• 4 benchmarks have fewer allocations
• 5 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

• Mann–Whitney U test with statistical test for significance of 5%
• Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

TODO: Check for usage of DD_DOTNET_TRACER_HOME outside the managed loader. Not a blocker for this PR. For now this is a hidden/internal fallback and all docs and tooling still tell users to set DD_DOTNET_TRACER_HOME.

Edit: 🤖✨ found these usages:

1. Datadog.Trace\AppSec\Waf\Initialization\LibraryLocationHelper.cs:95 - Reads the env var to locate the WAF library
2. Datadog.Trace.Tools.dd_dotnet\Utils.cs:124 - Sets the env var when launching processes
3. Datadog.Trace.Tools.dd_dotnet\Checks\ProcessBasicCheck.cs:125,127,141 - Checks if the env var is set and valid
4. Datadog.Trace.Tools.Runner\Utils.cs:592 - Sets the env var when launching processes
5. Datadog.Trace.Coverage.collector\CoverageSettings.cs:46 - Reads the env var for coverage settings
6. Datadog.Trace.Coverage.collector\CoverageCollector.cs:139 - Error message referencing the env var
7. Datadog.Trace.BenchmarkDotNet\DatadogDiagnoser.cs:135,232 - Reads and sets the env var for benchmarks
8. Datadog.FleetInstaller\TracerValues.cs:28,48 - Sets the env var in environment variable dictionaries
9. Datadog.Tracer.Native\environment_variables.h - Native code constant definition

Edit: 🤖✨ made further analysis:

Summary of Impact When DD_DOTNET_TRACER_HOME is Not Set

1. AppSec WAF Library Location (LibraryLocationHelper.cs:95)

• Impact: Graceful fallback ✅
• Behavior: If not set, it skips this source and tries other methods to locate the WAF library:
  • First checks DD_INTERNAL_TRACE_NATIVE_ENGINE_PATH (set by native loader)
  • Then checks profiler path environment variables (CORECLR_PROFILER_PATH, etc.)
  • Finally checks AppDomain.CurrentDomain.BaseDirectory
• Result: AppSec can still work without it

2. dd_dotnet Tool (Utils.cs:124)

• Impact: Sets it regardless ✅
• Behavior: The tool computes tracerHome from either:
  • User-specified --tracer-home parameter
  • Or defaults to Path.Combine(AppContext.BaseDirectory, "..") (line 81)
• Result: Always sets the env var when launching processes; doesn't read the existing value

3. dd_dotnet Process Check (ProcessBasicCheck.cs:141)

• Impact: Validation fails ⚠️
• Behavior: Explicitly checks if the env var is set and displays warning: Environment variable not set: DD_DOTNET_TRACER_HOME
• Result: Diagnostic check reports failure (ok = false)

4. Trace Runner Tool (Utils.cs:592)

• Impact: Sets it regardless ✅
• Behavior: Similar to dd_dotnet, computes tracerHome from either:
  • User-specified tracerHome parameter
  • Or defaults to AppContext.BaseDirectory parent (line 505 shows error if not found)
• Result: Always sets the env var when launching processes

5. Coverage Collector (CoverageCollector.cs:139)

• Impact: Coverage disabled ❌
• Behavior: Logs error: Tracer home (DD_DOTNET_TRACER_HOME environment variable) is not defined or folder doesn't exist, coverage has been disabled.
• Result: Returns false and coverage collection is completely disabled

6. BenchmarkDotNet Integration (DatadogDiagnoser.cs:135,232)

• Impact: Graceful fallback ✅
• Behavior: Tries multiple locations in order:
  a. DD_DOTNET_TRACER_HOME env var
  b. {WorkingDirectory}/datadog
  c. {BenchmarkAssemblyLocation}/datadog
  d. (Optional) relative path from benchmark assembly
• Result: Can still find monitoring home from other locations; sets the env var when launching benchmark processes

7. Fleet Installer (TracerValues.cs:28,48)

• Impact: Sets it as part of configuration ✅
• Behavior: The TracerValues constructor requires tracerHomeDirectory parameter, which it then includes in the environment variable dictionaries
• Result: Always includes it in the env vars it configures for IIS and global installations

Conclusion

Components that REQUIRE it:

• ❌ Coverage Collector - completely disables coverage if not set

Components that VALIDATE it (but may not strictly require it):

• ⚠️ dd_dotnet Process Check - reports diagnostic failure

Components that DON'T need it (have fallbacks or set it themselves):

• ✅ AppSec WAF - has multiple fallback mechanisms
• ✅ dd_dotnet Tool - computes and sets it when launching processes
• ✅ Trace Runner Tool - computes and sets it when launching processes
• ✅ BenchmarkDotNet - has multiple fallback locations
• ✅ Fleet Installer - receives it as constructor parameter and sets it

The most critical dependency is the Coverage Collector (Datadog.Trace.Coverage.collector), which will completely disable coverage functionality without it.