Skip to content

Add smoke tests for Datadog.AzureFunctions NuGet package#8336

Merged
andrewlock merged 15 commits into
masterfrom
andrew/azure-devops-nuget-tests
Mar 24, 2026
Merged

Add smoke tests for Datadog.AzureFunctions NuGet package#8336
andrewlock merged 15 commits into
masterfrom
andrew/azure-devops-nuget-tests

Conversation

@andrewlock

@andrewlock andrewlock commented Mar 18, 2026

Copy link
Copy Markdown
Member

Summary of changes

Adds new Windows and Linux smoke tests for the Datadog.AzureFunctions NuGet package

Reason for change

We recently had an issue where the Datadog.AzureFunctions package was broken, but as we don't currently test the package itself, we didn't catch it. This adds tests to make sure we can actually install the package that we build, similar to how we test the Datadog.Trace.Bundle package today.

Implementation details

This was harder than I had hoped, and required a bit of refactoring to the Nuke smoke tests that we added in #8271, as well as enabling the new smoke tests.

For the refactoring, this PR:

  • Updates the "nuget" smoke test dockerfiles to allow providing a NUGET_PACKAGE variable, so we can reuse the dockerfiles for multiple nuget packages
  • Add IncludeDdDotnetScenario to the windows nuget scenario - today we always try the dd-dotnet case after the "env vars" case, but there's no dd-dotnet in Datadog.AzureFunctions.
  • Ensure we delete the logs from previous runs before starting the tests, this bit when I was testing locally as I was failing on errors from previous runs
  • Make the specifying of runtime environment variables to pass in scenario-specific. We bake a lot of env vars into the dockerfiles, but it means that if you need different env vars for different scenarios that use the same dockerfile, you can.

In terms of enabling the Azure Functions smoke tests:

  • Add two new stages, Windows and Linux, running tests with the Datadog.AzureFunctions NuGet package. Made them "extended" tests so they only run on main/hotfix branches, seeing as the package will rarely change.
  • Add a couple of exceptions to warnings caused by missing libdatadog and profiler. These are always logged today, and can't be avoided AFAICT. They may be candidates for looking into further, but they're benign, so this is the easiest approach.
  • Update the env vars we pass in:
    • Don't set LD_PRELOAD (because the file doesn't exist)
    • Disable ASM and profiling (the native libraries aren't included)
    • "Pretend" to be in AAS, to try to stop using libdatadog config
    • Set AWS_LAMBDA_FUNCTION_NAME to avoid sending config to the profiler (we can't pretend to be in Azure functions, because otherwise we disable the aspnetcore integration which we need😅)
    • Set a fake DD_API_KEY otherwise we're marked as "unsafe to trace" and disable tracing
  • Add a new "Azure Functions snapshot", seeing as we have a bunch of AAS tags added now, and are missing the ASM ones etc.

Test coverage

More now! 🎉

I did a test run in which I restore the previous "broken" NuGet, and it causes the smoke tests to fail (which is good):

 error: NU1102: Unable to find package Datadog.Trace.Annotations with version (>= 3.40.0)

I've done multiple runs showing it passes otherwise, but I'll do a final run before merging to make sure

Other details

Context:

@andrewlock andrewlock requested a review from a team as a code owner March 18, 2026 17:35
@andrewlock andrewlock added area:benchmarks Benchmarks, throughput tests, Crank, Bombardier, etc type:reliability area:serverless labels Mar 18, 2026
@github-actions github-actions Bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Mar 18, 2026
@pr-commenter

pr-commenter Bot commented Mar 18, 2026 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-03-24 09:37:44

Comparing candidate commit 646a9eb in PR branch andrew/azure-devops-nuget-tests with baseline commit 75d765c in branch master.

Found 8 performance improvements and 9 performance regressions! Performance is the same for 254 metrics, 17 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟥 execution_time [+88.561ms; +89.113ms] or [+80.030%; +80.529%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net6.0

  • 🟩 execution_time [-21.412ms; -16.810ms] or [-9.907%; -7.778%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1

  • 🟥 execution_time [+16.365ms; +22.757ms] or [+8.271%; +11.502%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1

  • 🟩 execution_time [-15.669ms; -10.764ms] or [-7.397%; -5.082%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net6.0

  • 🟩 execution_time [-21.972ms; -17.818ms] or [-10.232%; -8.298%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net6.0

  • 🟥 throughput [-407.037op/s; -378.522op/s] or [-5.760%; -5.357%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1

  • 🟩 execution_time [-22.629ms; -21.971ms] or [-11.110%; -10.787%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • 🟥 execution_time [+39.014ms; +40.626ms] or [+21.744%; +22.642%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟥 execution_time [+52.121ms; +56.493ms] or [+35.234%; +38.189%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net6.0

  • 🟩 execution_time [-250.251µs; -242.616µs] or [-15.045%; -14.586%]
  • 🟩 throughput [+102.900op/s; +106.216op/s] or [+17.116%; +17.668%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice netcoreapp3.1

  • 🟥 throughput [-195.278op/s; -144.343op/s] or [-36.922%; -27.291%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net472

  • 🟥 execution_time [+155.273µs; +165.874µs] or [+13.551%; +14.476%]
  • 🟥 throughput [-110.492op/s; -103.970op/s] or [-12.661%; -11.913%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync netcoreapp3.1

  • 🟥 throughput [-31830.272op/s; -24692.953op/s] or [-6.907%; -5.359%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1

  • 🟩 throughput [+14301903.576op/s; +15270943.986op/s] or [+6.324%; +6.752%]

scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin netcoreapp3.1

  • 🟩 execution_time [-17.916ms; -12.498ms] or [-8.491%; -5.923%]

@dd-trace-dotnet-ci-bot

dd-trace-dotnet-ci-bot Bot commented Mar 18, 2026 •
edited
Loading

Copy link
Copy Markdown

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (8336) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration69.28 ± (69.56 - 70.01) ms68.66 ± (68.72 - 69.01) ms-0.9%✅
.NET Framework 4.8 - Bailout
duration73.41 ± (73.37 - 73.69) ms73.01 ± (72.99 - 73.35) ms-0.6%✅
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1035.05 ± (1040.01 - 1049.10) ms1033.28 ± (1035.02 - 1041.29) ms-0.2%✅
.NET Core 3.1 - Baseline
process.internal_duration_ms21.90 ± (21.87 - 21.93) ms21.86 ± (21.84 - 21.88) ms-0.2%✅
process.time_to_main_ms79.43 ± (79.27 - 79.59) ms79.41 ± (79.24 - 79.58) ms-0.0%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.92 ± (10.92 - 10.92) MB10.91 ± (10.90 - 10.91) MB-0.1%✅
runtime.dotnet.threads.count12 ± (12 - 12)12 ± (12 - 12)+0.0%✅
.NET Core 3.1 - Bailout
process.internal_duration_ms21.82 ± (21.80 - 21.85) ms21.82 ± (21.80 - 21.84) ms-0.0%✅
process.time_to_main_ms80.65 ± (80.51 - 80.79) ms80.90 ± (80.77 - 81.03) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.95 ± (10.94 - 10.95) MB10.94 ± (10.93 - 10.94) MB-0.1%✅
runtime.dotnet.threads.count13 ± (13 - 13)13 ± (13 - 13)+0.0%✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms218.98 ± (217.89 - 220.07) ms218.88 ± (217.59 - 220.17) ms-0.0%✅
process.time_to_main_ms514.13 ± (513.20 - 515.06) ms515.08 ± (514.09 - 516.07) ms+0.2%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed48.26 ± (48.23 - 48.29) MB48.21 ± (48.18 - 48.24) MB-0.1%✅
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)-0.4%✅
.NET 6 - Baseline
process.internal_duration_ms20.56 ± (20.53 - 20.59) ms20.69 ± (20.66 - 20.72) ms+0.6%✅⬆️
process.time_to_main_ms68.76 ± (68.63 - 68.89) ms68.79 ± (68.65 - 68.92) ms+0.0%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.61 ± (10.61 - 10.62) MB10.64 ± (10.64 - 10.64) MB+0.2%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 6 - Bailout
process.internal_duration_ms20.42 ± (20.40 - 20.45) ms20.62 ± (20.60 - 20.65) ms+1.0%✅⬆️
process.time_to_main_ms69.33 ± (69.23 - 69.43) ms70.04 ± (69.94 - 70.14) ms+1.0%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.67 ± (10.66 - 10.67) MB10.75 ± (10.75 - 10.75) MB+0.8%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms206.13 ± (204.82 - 207.44) ms211.18 ± (209.59 - 212.78) ms+2.5%✅⬆️
process.time_to_main_ms515.04 ± (513.69 - 516.38) ms514.92 ± (513.89 - 515.95) ms-0.0%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed49.96 ± (49.91 - 50.01) MB50.09 ± (50.05 - 50.13) MB+0.3%✅⬆️
runtime.dotnet.threads.count29 ± (29 - 29)29 ± (29 - 29)+0.1%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms18.79 ± (18.77 - 18.82) ms18.78 ± (18.76 - 18.81) ms-0.1%✅
process.time_to_main_ms67.96 ± (67.85 - 68.07) ms67.99 ± (67.87 - 68.10) ms+0.0%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.68 ± (7.68 - 7.69) MB7.68 ± (7.67 - 7.68) MB-0.1%✅
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 8 - Bailout
process.internal_duration_ms18.76 ± (18.73 - 18.79) ms18.78 ± (18.75 - 18.80) ms+0.1%✅⬆️
process.time_to_main_ms69.22 ± (69.11 - 69.33) ms69.15 ± (69.03 - 69.26) ms-0.1%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.75 ± (7.74 - 7.76) MB7.74 ± (7.73 - 7.75) MB-0.1%✅
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms152.38 ± (151.75 - 153.02) ms153.19 ± (152.41 - 153.97) ms+0.5%✅⬆️
process.time_to_main_ms474.19 ± (473.41 - 474.98) ms475.16 ± (474.36 - 475.97) ms+0.2%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed36.87 ± (36.85 - 36.88) MB36.89 ± (36.87 - 36.92) MB+0.1%✅⬆️
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)-0.0%✅

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration192.68 ± (192.59 - 193.48) ms191.19 ± (191.22 - 191.94) ms-0.8%✅
.NET Framework 4.8 - Bailout
duration195.89 ± (195.70 - 196.19) ms195.56 ± (195.45 - 195.95) ms-0.2%✅
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1147.97 ± (1149.10 - 1157.48) ms1152.28 ± (1152.84 - 1161.66) ms+0.4%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms187.65 ± (187.26 - 188.03) ms187.36 ± (186.89 - 187.83) ms-0.2%✅
process.time_to_main_ms80.62 ± (80.42 - 80.83) ms80.41 ± (80.18 - 80.64) ms-0.3%✅
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.20 ± (16.18 - 16.23) MB16.09 ± (16.07 - 16.12) MB-0.7%✅
runtime.dotnet.threads.count20 ± (19 - 20)20 ± (19 - 20)+0.0%✅⬆️
.NET Core 3.1 - Bailout
process.internal_duration_ms187.12 ± (186.78 - 187.46) ms186.98 ± (186.54 - 187.42) ms-0.1%✅
process.time_to_main_ms81.95 ± (81.79 - 82.10) ms81.83 ± (81.62 - 82.05) ms-0.1%✅
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.09 ± (16.06 - 16.12) MB16.26 ± (16.22 - 16.30) MB+1.1%✅⬆️
runtime.dotnet.threads.count21 ± (20 - 21)21 ± (20 - 21)+0.1%✅⬆️
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms390.50 ± (389.13 - 391.86) ms389.42 ± (387.98 - 390.85) ms-0.3%✅
process.time_to_main_ms516.86 ± (515.87 - 517.85) ms516.85 ± (515.88 - 517.82) ms-0.0%✅
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed58.14 ± (57.93 - 58.35) MB57.93 ± (57.70 - 58.15) MB-0.4%✅
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)-0.1%✅
.NET 6 - Baseline
process.internal_duration_ms191.89 ± (191.44 - 192.34) ms192.43 ± (191.98 - 192.88) ms+0.3%✅⬆️
process.time_to_main_ms70.23 ± (70.01 - 70.44) ms70.23 ± (70.05 - 70.41) ms+0.0%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed16.43 ± (16.40 - 16.47) MB16.10 ± (15.97 - 16.23) MB-2.0%✅
runtime.dotnet.threads.count19 ± (19 - 19)18 ± (18 - 18)-3.7%✅
.NET 6 - Bailout
process.internal_duration_ms190.51 ± (190.23 - 190.79) ms190.57 ± (190.26 - 190.88) ms+0.0%✅⬆️
process.time_to_main_ms70.85 ± (70.72 - 70.98) ms70.79 ± (70.69 - 70.89) ms-0.1%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed16.15 ± (16.00 - 16.29) MB16.31 ± (16.19 - 16.43) MB+1.0%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)20 ± (19 - 20)+2.7%✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms412.92 ± (411.48 - 414.36) ms415.55 ± (414.14 - 416.97) ms+0.6%✅⬆️
process.time_to_main_ms520.04 ± (518.84 - 521.24) ms520.37 ± (519.02 - 521.72) ms+0.1%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed60.92 ± (60.89 - 60.96) MB60.86 ± (60.82 - 60.90) MB-0.1%✅
runtime.dotnet.threads.count31 ± (30 - 31)30 ± (30 - 31)-0.2%✅
.NET 8 - Baseline
process.internal_duration_ms189.29 ± (188.92 - 189.65) ms188.86 ± (188.55 - 189.17) ms-0.2%✅
process.time_to_main_ms69.54 ± (69.33 - 69.75) ms69.75 ± (69.54 - 69.97) ms+0.3%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.78 ± (11.76 - 11.81) MB11.81 ± (11.78 - 11.85) MB+0.2%✅⬆️
runtime.dotnet.threads.count18 ± (18 - 18)18 ± (18 - 18)-1.1%✅
.NET 8 - Bailout
process.internal_duration_ms188.50 ± (188.19 - 188.81) ms187.92 ± (187.64 - 188.20) ms-0.3%✅
process.time_to_main_ms70.55 ± (70.44 - 70.67) ms70.54 ± (70.45 - 70.63) ms-0.0%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.85 ± (11.82 - 11.88) MB11.86 ± (11.83 - 11.88) MB+0.1%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 19)+0.6%✅⬆️
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms339.28 ± (338.15 - 340.42) ms338.50 ± (337.26 - 339.73) ms-0.2%✅
process.time_to_main_ms477.06 ± (475.98 - 478.13) ms477.59 ± (476.61 - 478.57) ms+0.1%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed48.76 ± (48.72 - 48.80) MB48.79 ± (48.76 - 48.82) MB+0.1%✅⬆️
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)-0.2%✅
Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts
FakeDbCommand (.NET Framework 4.8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8336) - mean (69ms)  : 67, 71
    master - mean (70ms)  : 66, 73

    section Bailout
    This PR (8336) - mean (73ms)  : 70, 76
    master - mean (74ms)  : 72, 75

    section CallTarget+Inlining+NGEN
    This PR (8336) - mean (1,038ms)  : 994, 1083
    master - mean (1,045ms)  : 979, 1110
Loading
FakeDbCommand (.NET Core 3.1) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8336) - mean (107ms)  : 104, 110
    master - mean (107ms)  : 104, 110

    section Bailout
    This PR (8336) - mean (109ms)  : 107, 110
    master - mean (108ms)  : 106, 110

    section CallTarget+Inlining+NGEN
    This PR (8336) - mean (771ms)  : 753, 790
    master - mean (768ms)  : 749, 788
Loading
FakeDbCommand (.NET 6) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8336) - mean (95ms)  : 92, 98
    master - mean (95ms)  : 92, 97

    section Bailout
    This PR (8336) - mean (96ms)  : 94, 97
    master - mean (95ms)  : 93, 96

    section CallTarget+Inlining+NGEN
    This PR (8336) - mean (758ms)  : 741, 776
    master - mean (753ms)  : 735, 771
Loading
FakeDbCommand (.NET 8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8336) - mean (93ms)  : 91, 96
    master - mean (93ms)  : 90, 97

    section Bailout
    This PR (8336) - mean (94ms)  : 93, 96
    master - mean (95ms)  : 92, 97

    section CallTarget+Inlining+NGEN
    This PR (8336) - mean (661ms)  : 638, 683
    master - mean (658ms)  : 632, 684
Loading
HttpMessageHandler (.NET Framework 4.8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8336) - mean (192ms)  : 188, 195
    master - mean (193ms)  : 189, 197

    section Bailout
    This PR (8336) - mean (196ms)  : 193, 198
    master - mean (196ms)  : 194, 198

    section CallTarget+Inlining+NGEN
    This PR (8336) - mean (1,157ms)  : 1094, 1221
    master - mean (1,153ms)  : 1092, 1215
Loading
HttpMessageHandler (.NET Core 3.1) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8336) - mean (277ms)  : 269, 284
    master - mean (277ms)  : 272, 281

    section Bailout
    This PR (8336) - mean (277ms)  : 271, 283
    master - mean (277ms)  : 272, 283

    section CallTarget+Inlining+NGEN
    This PR (8336) - mean (937ms)  : 913, 960
    master - mean (938ms)  : 909, 967
Loading
HttpMessageHandler (.NET 6) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8336) - mean (271ms)  : 265, 276
    master - mean (270ms)  : 264, 277

    section Bailout
    This PR (8336) - mean (269ms)  : 265, 273
    master - mean (269ms)  : 266, 273

    section CallTarget+Inlining+NGEN
    This PR (8336) - mean (964ms)  : 940, 988
    master - mean (962ms)  : 943, 980
Loading
HttpMessageHandler (.NET 8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8336) - mean (268ms)  : 263, 273
    master - mean (269ms)  : 263, 274

    section Bailout
    This PR (8336) - mean (268ms)  : 264, 271
    master - mean (268ms)  : 265, 272

    section CallTarget+Inlining+NGEN
    This PR (8336) - mean (846ms)  : 820, 873
    master - mean (847ms)  : 828, 867
Loading

NachoEchevarria
NachoEchevarria reviewed Mar 19, 2026
View reviewed changes
Comment thread tracer/build/_build/SmokeTests/SmokeTestScenario.cs
_ => base.SnapshotFile,
};

public override Dictionary<string, string> GetEnvironment(bool isCrashTest)

@NachoEchevarria NachoEchevarria Mar 19, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is the same as the previous one. Would it make sense to move it to a helper?

@andrewlock andrewlock Mar 19, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's actually not the same as the other one 😅 The linux one has an extra env var, AWS_LAMBDA_FUNCTION_NAME, because it requires more hackery 😅 Even if we resolve them, I think it makes sense to keep them separate and just accept a small amount of duplication personally

NachoEchevarria
NachoEchevarria reviewed Mar 19, 2026
View reviewed changes
Comment thread tracer/build/_build/docker/smoke.windows.nuget.dockerfile
NachoEchevarria
NachoEchevarria approved these changes Mar 19, 2026
View reviewed changes

@NachoEchevarria NachoEchevarria left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

andrewlock
andrewlock commented Mar 19, 2026
View reviewed changes
Comment thread tracer/build/_build/docker/smoke.nuget.dockerfile Outdated
lucaspimentel
lucaspimentel reviewed Mar 23, 2026
View reviewed changes
Comment thread tracer/build/_build/docker/smoke.nuget.dockerfile Outdated
@andrewlock @lucaspimentel
Update tracer/build/_build/docker/smoke.nuget.dockerfile
646a9eb 
Co-authored-by: Lucas Pimentel <lucas.pimentel@datadoghq.com>
@andrewlock andrewlock enabled auto-merge (squash) March 24, 2026 08:56
@andrewlock andrewlock merged commit 1bb5b79 into master Mar 24, 2026
137 of 138 checks passed
@andrewlock andrewlock deleted the andrew/azure-devops-nuget-tests branch March 24, 2026 12:03
@github-actions github-actions Bot added this to the vNext-v3 milestone Mar 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lucaspimentel lucaspimentel lucaspimentel approved these changes

@NachoEchevarria NachoEchevarria NachoEchevarria approved these changes

Assignees

No one assigned

Labels

area:benchmarks Benchmarks, throughput tests, Crank, Bombardier, etc area:builds project files, build scripts, pipelines, versioning, releases, packages area:serverless type:reliability

Projects

None yet

Milestone

3.41.0

Development

Successfully merging this pull request may close these issues.

3 participants

@andrewlock @lucaspimentel @NachoEchevarria