[Dynamic Instrumentation] Added native hot standby mode to support Remote Enablement and Stable Config by GreenMatan · Pull Request #7441 · DataDog/dd-trace-dotnet

GreenMatan · 2025-08-27T10:08:48Z

Summary of changes

Added hot standby mode to Dynamic Instrumentation and Exception Replay products in the native side of the tracer to support Remote Enablement and Stable Config.

For DI/ER products to function correctly, the native side, among other things, need to inject a new field into each and every StateMachine type that is generated for async methods. Our injected field is used during the execution of DI/ER instrumentation to capture continuation of async flows.
The only point in time where it's possible to change type's layout is in ModuleLoadFinished event, and this event is fired only once per module.
Until Stable Configuration we only injected that field when DI/ER env vars were enable, but now we can not rely on these env var anymore as they're no longer the source of truth. With Stable Config we can not know if DI/ER will be enabled or not at startup. To accommodate that, the native side now acts as if DI/ER is enabled, just in case it will.
This is true not only for Stable Config, but also for Remote Enablement.

There are two new env vars:

DD_DYNAMIC_INSTRUMENTATION_MANAGED_ACTIVATION_ENABLED
DD_EXCEPTION_REPLAY_MANAGED_ACTIVATION_ENABLED

These env vars, as described in Stable Config Standby mode doc, are used to deactivate hot standby for ER/DI.

A big contribution to support remote enablement for DI/ER involved changing our injection strategy of async methods.
In DI/ER instrumentation, on ModuleLoadFinished, we inject a field of type AsyncDebuggerState into all StateMachine types. This field is defined in Datadog.Trace. When the managed loader fails/bails out in loading Datadog.Trace, it results in TypeLoadException as the runtime tried to implicitly load Datadog.Trace to no avail.
In this PR, the field type is changed to be System.Object to avoid that cumbersome.

Implementation

The field that is injected and used in async methods to determine continuation is changed to be System.Object. To support that, the instrumentation of async methods for log probes has been changed - instead of accepting AsyncDebuggerState as the state, it's now accepting System.Object.
The entrypoints now goes through a new shim class AsyncMethodDebuggerInvokerV2, and from there to the underlying AsyncMethodDebuggerInvoker after materializing the state from System.Object -> AsyncDebuggerState.

In span probes on async methods the same concept has been applied, but, instead of a new class shim, new signatures were added that accept the state as System.Object and in similar fashion materializing to AsyncDebuggerState before transferring control the the previous functions.

Test

Relying on existing test suite of debugger & tracer to cover the instrumentation changes

Reason for change

Fully support Remote Enablement and Stable Config.

Resolves DEBUG-4421

dd-trace-dotnet-ci-bot · 2025-08-27T10:47:31Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7441) - mean (72ms)  : 71, 74
     .   : milestone, 72,
    master - mean (72ms)  : 71, 73
     .   : milestone, 72,

    section Baseline
    This PR (7441) - mean (69ms)  : 63, 75
     .   : milestone, 69,
    master - mean (70ms)  : 61, 79
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (7441) - mean (997ms)  : 973, 1021
     .   : milestone, 997,
    master - mean (1,003ms)  : 978, 1027
     .   : milestone, 1003,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7441) - mean (107ms)  : 106, 108
     .   : milestone, 107,
    master - mean (106ms)  : 105, 107
     .   : milestone, 106,

    section Baseline
    This PR (7441) - mean (106ms)  : 104, 108
     .   : milestone, 106,
    master - mean (105ms)  : 103, 108
     .   : milestone, 105,

    section CallTarget+Inlining+NGEN
    This PR (7441) - mean (705ms)  : 690, 721
     .   : milestone, 705,
    master - mean (709ms)  : 681, 737
     .   : milestone, 709,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7441) - mean (94ms)  : 93, 95
     .   : milestone, 94,
    master - mean (94ms)  : 93, 96
     .   : milestone, 94,

    section Baseline
    This PR (7441) - mean (93ms)  : 92, 95
     .   : milestone, 93,
    master - mean (93ms)  : 91, 96
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (7441) - mean (661ms)  : 641, 682
     .   : milestone, 661,
    master - mean (665ms)  : 643, 688
     .   : milestone, 665,

gantt
    title Execution time (ms) FakeDbCommand (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7441) - mean (93ms)  : 92, 94
     .   : milestone, 93,
    master - mean (93ms)  : 92, 94
     .   : milestone, 93,

    section Baseline
    This PR (7441) - mean (92ms)  : 90, 94
     .   : milestone, 92,
    master - mean (92ms)  : 90, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (7441) - mean (594ms)  : 582, 606
     .   : milestone, 594,
    master - mean (600ms)  : 582, 617
     .   : milestone, 600,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7441) - mean (196ms)  : 193, 199
     .   : milestone, 196,
    master - mean (208ms)  : 203, 212
     .   : milestone, 208,

    section Baseline
    This PR (7441) - mean (193ms)  : 186, 200
     .   : milestone, 193,
    master - mean (204ms)  : 198, 210
     .   : milestone, 204,

    section CallTarget+Inlining+NGEN
    This PR (7441) - mean (1,112ms)  : 1072, 1152
     .   : milestone, 1112,
    master - mean (1,154ms)  : 1127, 1182
     .   : milestone, 1154,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7441) - mean (277ms)  : 273, 281
     .   : milestone, 277,
    master - mean (296ms)  : 289, 302
     .   : milestone, 296,

    section Baseline
    This PR (7441) - mean (278ms)  : 272, 283
     .   : milestone, 278,
    master - mean (295ms)  : 285, 305
     .   : milestone, 295,

    section CallTarget+Inlining+NGEN
    This PR (7441) - mean (897ms)  : 867, 927
     .   : milestone, 897,
    master - mean (935ms)  : 900, 971
     .   : milestone, 935,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7441) - mean (270ms)  : 266, 274
     .   : milestone, 270,
    master - mean (288ms)  : 282, 295
     .   : milestone, 288,

    section Baseline
    This PR (7441) - mean (270ms)  : 266, 275
     .   : milestone, 270,
    master - mean (288ms)  : 282, 294
     .   : milestone, 288,

    section CallTarget+Inlining+NGEN
    This PR (7441) - mean (877ms)  : 844, 910
     .   : milestone, 877,
    master - mean (920ms)  : 890, 951
     .   : milestone, 920,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7441) - mean (268ms)  : 264, 273
     .   : milestone, 268,
    master - mean (286ms)  : 281, 292
     .   : milestone, 286,

    section Baseline
    This PR (7441) - mean (270ms)  : 262, 277
     .   : milestone, 270,
    master - mean (286ms)  : 279, 293
     .   : milestone, 286,

    section CallTarget+Inlining+NGEN
    This PR (7441) - mean (785ms)  : 765, 805
     .   : milestone, 785,
    master - mean (847ms)  : 772, 922
     .   : milestone, 847,

pr-commenter · 2025-08-27T11:08:30Z

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #7441 compared to master:

1 benchmarks are faster, with geometric mean 2.458
8 benchmarks have fewer allocations
8 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7441

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net472	6.04 KB	6 KB	-37 B	-0.61%
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑netcoreapp3.1	5.71 KB	5.64 KB	-66 B	-1.16%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	10.5μs	56.3ns	319ns	0	0	0	5.49 KB
master	`StartStopWithChild`	netcoreapp3.1	14.2μs	45ns	174ns	0	0	0	5.71 KB
master	`StartStopWithChild`	net472	21.7μs	67.8ns	254ns	1.02	0.338	0.113	6.04 KB
#7441	`StartStopWithChild`	net6.0	10.5μs	57.4ns	354ns	0	0	0	5.5 KB
#7441	`StartStopWithChild`	netcoreapp3.1	13.5μs	65.2ns	276ns	0	0	0	5.64 KB
#7441	`StartStopWithChild`	net472	21.3μs	112ns	523ns	0.88	0.33	0	6 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7441

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces‑net472	3.35 KB	3.31 KB	-46 B	-1.37%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	922μs	218ns	843ns	2.71 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	1.02ms	207ns	803ns	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	1.21ms	2.26μs	8.76μs	3.35 KB
#7441	`WriteAndFlushEnrichedTraces`	net6.0	929μs	116ns	418ns	2.71 KB
#7441	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	1.02ms	68.9ns	248ns	2.7 KB
#7441	`WriteAndFlushEnrichedTraces`	net472	1.2ms	114ns	441ns	3.31 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`AllCycleSimpleBody`	net6.0	334μs	610ns	2.36μs	0	175.6 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	499μs	822ns	3.18μs	0	177.77 KB
master	`AllCycleSimpleBody`	net472	440μs	145ns	560ns	30.2	199.81 KB
master	`AllCycleMoreComplexBody`	net6.0	337μs	1.36μs	5.1μs	0	179.1 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	501μs	771ns	2.89μs	0	181.18 KB
master	`AllCycleMoreComplexBody`	net472	463μs	268ns	1.04μs	30.1	203.32 KB
master	`ObjectExtractorSimpleBody`	net6.0	329ns	1.6ns	6.61ns	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	397ns	1.06ns	4.09ns	0	272 B
master	`ObjectExtractorSimpleBody`	net472	300ns	0.0507ns	0.19ns	0.0439	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	6.37μs	1.92ns	7.18ns	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	7.68μs	33.5ns	130ns	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	6.72μs	2.92ns	11.3ns	0.571	3.8 KB
#7441	`AllCycleSimpleBody`	net6.0	332μs	1.45μs	5.21μs	0	175.6 KB
#7441	`AllCycleSimpleBody`	netcoreapp3.1	477μs	1.7μs	6.58μs	0	177.77 KB
#7441	`AllCycleSimpleBody`	net472	439μs	160ns	620ns	30.2	199.81 KB
#7441	`AllCycleMoreComplexBody`	net6.0	338μs	1.63μs	6.53μs	0	179.1 KB
#7441	`AllCycleMoreComplexBody`	netcoreapp3.1	514μs	400ns	1.55μs	0	181.18 KB
#7441	`AllCycleMoreComplexBody`	net472	452μs	169ns	654ns	31.2	203.32 KB
#7441	`ObjectExtractorSimpleBody`	net6.0	318ns	1.82ns	13.1ns	0	280 B
#7441	`ObjectExtractorSimpleBody`	netcoreapp3.1	405ns	2.02ns	8.33ns	0	272 B
#7441	`ObjectExtractorSimpleBody`	net472	302ns	0.0274ns	0.103ns	0.0441	281 B
#7441	`ObjectExtractorMoreComplexBody`	net6.0	6.35μs	22.8ns	88.4ns	0	3.78 KB
#7441	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	7.73μs	38.5ns	154ns	0	3.69 KB
#7441	`ObjectExtractorMoreComplexBody`	net472	6.72μs	6.07ns	23.5ns	0.571	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EncodeArgs`	net6.0	76.5μs	180ns	698ns	0	32.4 KB
master	`EncodeArgs`	netcoreapp3.1	97μs	238ns	921ns	0	32.4 KB
master	`EncodeArgs`	net472	108μs	12.4ns	48ns	4.86	32.51 KB
master	`EncodeLegacyArgs`	net6.0	146μs	50ns	173ns	0	2.15 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	197μs	372ns	1.44μs	0	2.14 KB
master	`EncodeLegacyArgs`	net472	263μs	22.6ns	87.5ns	0	2.16 KB
#7441	`EncodeArgs`	net6.0	76.5μs	225ns	841ns	0	32.4 KB
#7441	`EncodeArgs`	netcoreapp3.1	95.4μs	15.6ns	58.2ns	0	32.4 KB
#7441	`EncodeArgs`	net472	111μs	8.45ns	31.6ns	5.01	32.5 KB
#7441	`EncodeLegacyArgs`	net6.0	144μs	83.3ns	323ns	0	2.14 KB
#7441	`EncodeLegacyArgs`	netcoreapp3.1	203μs	171ns	594ns	0	2.14 KB
#7441	`EncodeLegacyArgs`	net472	265μs	142ns	549ns	0	2.16 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #7441

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack‑netcoreapp3.1	2.458	732,916.82	298,128.27

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`RunWafRealisticBenchmark`	net6.0	392μs	130ns	505ns	4.56 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	416μs	117ns	404ns	4.48 KB
master	`RunWafRealisticBenchmark`	net472	429μs	79ns	306ns	4.66 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	287μs	29.7ns	103ns	2.24 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	730μs	3.51μs	14.5μs	2.22 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	308μs	25.8ns	100ns	2.29 KB
#7441	`RunWafRealisticBenchmark`	net6.0	395μs	64.8ns	234ns	4.56 KB
#7441	`RunWafRealisticBenchmark`	netcoreapp3.1	412μs	187ns	647ns	4.48 KB
#7441	`RunWafRealisticBenchmark`	net472	428μs	39.8ns	154ns	4.66 KB
#7441	`RunWafRealisticBenchmarkWithAttack`	net6.0	286μs	32.4ns	117ns	2.24 KB
#7441	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	298μs	155ns	601ns	2.22 KB
#7441	`RunWafRealisticBenchmarkWithAttack`	net472	312μs	38.9ns	145ns	2.29 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`SendRequest`	net6.0	60.9μs	37.6ns	136ns	14.52 KB
master	`SendRequest`	netcoreapp3.1	71.5μs	66ns	238ns	17.42 KB
master	`SendRequest`	net472	0.00282ns	0.00156ns	0.00606ns	0 b
#7441	`SendRequest`	net6.0	60.9μs	52.1ns	202ns	14.52 KB
#7441	`SendRequest`	netcoreapp3.1	71.3μs	60.1ns	225ns	17.42 KB
#7441	`SendRequest`	net472	0.021ns	0.00175ns	0.00679ns	0 b

Benchmarks.Trace.CharSliceBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7441

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net472	0 b	73 B	73 B	∞
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net472	0 b	47 B	47 B	∞
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net6.0	4 B	7 B	3 B	75.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net6.0	3 B	4 B	1 B	33.33%

Fewer allocations 🎉 in #7441

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑netcoreapp3.1	1 B	0 b	-1 B	-100.00%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`OriginalCharSlice`	net6.0	2.03ms	1.48μs	5.54μs	0	640.01 KB
master	`OriginalCharSlice`	netcoreapp3.1	2.1ms	7.31μs	28.3μs	0	640 KB
master	`OriginalCharSlice`	net472	2.67ms	276ns	1.07μs	100	641.95 KB
master	`OptimizedCharSlice`	net6.0	1.41ms	554ns	2.07μs	0	4 B
master	`OptimizedCharSlice`	netcoreapp3.1	1.74ms	500ns	1.94μs	0	1 B
master	`OptimizedCharSlice`	net472	1.95ms	442ns	1.71μs	0	0 b
master	`OptimizedCharSliceWithPool`	net6.0	800μs	120ns	465ns	0	3 B
master	`OptimizedCharSliceWithPool`	netcoreapp3.1	813μs	106ns	395ns	0	1 B
master	`OptimizedCharSliceWithPool`	net472	1.13ms	91ns	353ns	0	0 b
#7441	`OriginalCharSlice`	net6.0	1.89ms	4.42μs	17.1μs	0	640.01 KB
#7441	`OriginalCharSlice`	netcoreapp3.1	2.14ms	8.01μs	30μs	0	640 KB
#7441	`OriginalCharSlice`	net472	2.66ms	197ns	764ns	100	641.95 KB
#7441	`OptimizedCharSlice`	net6.0	1.42ms	272ns	1.05μs	0	7 B
#7441	`OptimizedCharSlice`	netcoreapp3.1	1.69ms	721ns	2.79μs	0	1 B
#7441	`OptimizedCharSlice`	net472	1.97ms	680ns	2.63μs	0	73 B
#7441	`OptimizedCharSliceWithPool`	net6.0	807μs	432ns	1.61μs	0	4 B
#7441	`OptimizedCharSliceWithPool`	netcoreapp3.1	821μs	71.2ns	276ns	0	0 b
#7441	`OptimizedCharSliceWithPool`	net472	1.14ms	76.2ns	295ns	0	47 B

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7441

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net472	55.68 KB	56.4 KB	718 B	1.29%
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0	41.68 KB	41.93 KB	251 B	0.60%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	703μs	3.22μs	12.9μs	0	0	41.68 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	678μs	3.91μs	32.9μs	0	0	41.94 KB
master	`WriteAndFlushEnrichedTraces`	net472	910μs	2.25μs	8.43μs	4.46	0	55.68 KB
#7441	`WriteAndFlushEnrichedTraces`	net6.0	677μs	3.89μs	29.4μs	0	0	41.93 KB
#7441	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	682μs	3.95μs	36μs	0	0	41.87 KB
#7441	`WriteAndFlushEnrichedTraces`	net472	985μs	4.79μs	19.8μs	8.93	4.46	56.4 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.88μs	8.89ns	37.7ns	0	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	2.55μs	7.93ns	30.7ns	0	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.78μs	3.08ns	11.9ns	0.154	0.014	987 B
#7441	`ExecuteNonQuery`	net6.0	1.92μs	8.76ns	33.9ns	0	0	1.02 KB
#7441	`ExecuteNonQuery`	netcoreapp3.1	2.49μs	7.78ns	30.1ns	0	0	1.02 KB
#7441	`ExecuteNonQuery`	net472	2.79μs	3.69ns	14.3ns	0.154	0.014	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.79μs	6.33ns	24.5ns	0	1.03 KB
master	`CallElasticsearch`	netcoreapp3.1	2.23μs	8.52ns	33ns	0	1.03 KB
master	`CallElasticsearch`	net472	3.47μs	1.31ns	4.74ns	0.159	1.04 KB
master	`CallElasticsearchAsync`	net6.0	1.77μs	8.8ns	36.3ns	0	1.01 KB
master	`CallElasticsearchAsync`	netcoreapp3.1	2.4μs	8.77ns	34ns	0	1.08 KB
master	`CallElasticsearchAsync`	net472	3.69μs	5.86ns	22.7ns	0.166	1.1 KB
#7441	`CallElasticsearch`	net6.0	1.83μs	1.88ns	7.27ns	0	1.03 KB
#7441	`CallElasticsearch`	netcoreapp3.1	2.25μs	6.82ns	26.4ns	0	1.03 KB
#7441	`CallElasticsearch`	net472	3.51μs	3.03ns	11.7ns	0.16	1.04 KB
#7441	`CallElasticsearchAsync`	net6.0	1.87μs	2.61ns	9.78ns	0	1.01 KB
#7441	`CallElasticsearchAsync`	netcoreapp3.1	2.44μs	11.8ns	45.8ns	0	1.08 KB
#7441	`CallElasticsearchAsync`	net472	3.68μs	3.51ns	13.6ns	0.166	1.1 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.89μs	6.72ns	26ns	0	952 B
master	`ExecuteAsync`	netcoreapp3.1	2.23μs	8.48ns	32.9ns	0	952 B
master	`ExecuteAsync`	net472	2.5μs	2.14ns	8.02ns	0.138	915 B
#7441	`ExecuteAsync`	net6.0	1.94μs	1.19ns	4.45ns	0	952 B
#7441	`ExecuteAsync`	netcoreapp3.1	2.38μs	4.19ns	16.2ns	0	952 B
#7441	`ExecuteAsync`	net472	2.54μs	2.21ns	8.56ns	0.139	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	6.95μs	21.2ns	79.2ns	0	2.36 KB
master	`SendAsync`	netcoreapp3.1	8.55μs	19.4ns	72.5ns	0	2.9 KB
master	`SendAsync`	net472	12.5μs	16.8ns	65.2ns	0.495	3.18 KB
#7441	`SendAsync`	net6.0	7.04μs	22.6ns	84.7ns	0	2.36 KB
#7441	`SendAsync`	netcoreapp3.1	8.52μs	22.7ns	88ns	0	2.9 KB
#7441	`SendAsync`	net472	12.6μs	11.1ns	43.1ns	0.503	3.18 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7441

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472	57.34 KB	65.54 KB	8.19 KB	14.29%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net472	278.53 KB	286.72 KB	8.19 KB	2.94%

Fewer allocations 🎉 in #7441

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net6.0	43.78 KB	43.52 KB	-256 B	-0.58%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0	277.54 KB	275.8 KB	-1.74 KB	-0.63%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	278.16 KB	255.38 KB	-22.78 KB	-8.19%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑netcoreapp3.1	86.24 KB	42.76 KB	-43.48 KB	-50.42%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	48.6μs	268ns	1.65μs	43.78 KB
master	`StringConcatBenchmark`	netcoreapp3.1	49.7μs	289ns	2.65μs	86.24 KB
master	`StringConcatBenchmark`	net472	57.4μs	235ns	878ns	57.34 KB
master	`StringConcatAspectBenchmark`	net6.0	471μs	1.94μs	6.72μs	277.54 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	548μs	1.83μs	6.86μs	278.16 KB
master	`StringConcatAspectBenchmark`	net472	400μs	2.28μs	17.1μs	278.53 KB
#7441	`StringConcatBenchmark`	net6.0	44.5μs	256ns	2μs	43.52 KB
#7441	`StringConcatBenchmark`	netcoreapp3.1	50.7μs	561ns	5.5μs	42.76 KB
#7441	`StringConcatBenchmark`	net472	58μs	205ns	766ns	65.54 KB
#7441	`StringConcatAspectBenchmark`	net6.0	457μs	1.52μs	5.47μs	275.8 KB
#7441	`StringConcatAspectBenchmark`	netcoreapp3.1	507μs	2.31μs	9.24μs	255.38 KB
#7441	`StringConcatAspectBenchmark`	net472	404μs	2μs	10μs	286.72 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.61μs	12.3ns	49.3ns	0	1.7 KB
master	`EnrichedLog`	netcoreapp3.1	3.38μs	11.7ns	42.2ns	0	1.7 KB
master	`EnrichedLog`	net472	3.88μs	3.5ns	13.1ns	0.252	1.64 KB
#7441	`EnrichedLog`	net6.0	2.53μs	13.7ns	77.5ns	0	1.7 KB
#7441	`EnrichedLog`	netcoreapp3.1	3.5μs	1.14ns	4.11ns	0	1.7 KB
#7441	`EnrichedLog`	net472	4.02μs	2.72ns	10.6ns	0.26	1.64 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`EnrichedLog`	net6.0	122μs	45.3ns	170ns	4.31 KB
master	`EnrichedLog`	netcoreapp3.1	127μs	377ns	1.41μs	4.31 KB
master	`EnrichedLog`	net472	166μs	98.6ns	382ns	4.51 KB
#7441	`EnrichedLog`	net6.0	122μs	45.4ns	164ns	4.31 KB
#7441	`EnrichedLog`	netcoreapp3.1	128μs	196ns	734ns	4.31 KB
#7441	`EnrichedLog`	net472	167μs	26.1ns	101ns	4.52 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	4.9μs	19.5ns	67.7ns	0	2.26 KB
master	`EnrichedLog`	netcoreapp3.1	6.5μs	20.3ns	78.8ns	0	2.26 KB
master	`EnrichedLog`	net472	7.56μs	6.99ns	26.1ns	0.302	2.08 KB
#7441	`EnrichedLog`	net6.0	4.97μs	18.1ns	65.2ns	0	2.26 KB
#7441	`EnrichedLog`	netcoreapp3.1	6.63μs	6.5ns	25.2ns	0	2.26 KB
#7441	`EnrichedLog`	net472	7.57μs	13.1ns	50.7ns	0.303	2.08 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	2.02μs	10.6ns	44.8ns	0	1.2 KB
master	`SendReceive`	netcoreapp3.1	2.57μs	1.44ns	5.38ns	0	1.2 KB
master	`SendReceive`	net472	3.21μs	2.5ns	9.7ns	0.177	1.2 KB
#7441	`SendReceive`	net6.0	2.01μs	8.13ns	31.5ns	0	1.2 KB
#7441	`SendReceive`	netcoreapp3.1	2.7μs	13ns	52.2ns	0	1.2 KB
#7441	`SendReceive`	net472	3.23μs	11.5ns	44.7ns	0.181	1.2 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	4.38μs	4.13ns	15.4ns	0	1.58 KB
master	`EnrichedLog`	netcoreapp3.1	5.47μs	10.1ns	39.3ns	0	1.63 KB
master	`EnrichedLog`	net472	6.63μs	7.13ns	27.6ns	0.296	2.03 KB
#7441	`EnrichedLog`	net6.0	4.17μs	6.2ns	24ns	0	1.58 KB
#7441	`EnrichedLog`	netcoreapp3.1	5.42μs	14.3ns	55.5ns	0	1.63 KB
#7441	`EnrichedLog`	net472	6.6μs	8.41ns	32.6ns	0.295	2.03 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	753ns	3.44ns	13.8ns	0	576 B
master	`StartFinishSpan`	netcoreapp3.1	990ns	4.88ns	20.7ns	0	576 B
master	`StartFinishSpan`	net472	906ns	0.761ns	2.95ns	0.0903	578 B
master	`StartFinishScope`	net6.0	883ns	0.187ns	0.724ns	0	696 B
master	`StartFinishScope`	netcoreapp3.1	1.18μs	6.38ns	30.6ns	0	696 B
master	`StartFinishScope`	net472	1.1μs	0.537ns	2.01ns	0.0996	658 B
#7441	`StartFinishSpan`	net6.0	749ns	0.2ns	0.747ns	0	576 B
#7441	`StartFinishSpan`	netcoreapp3.1	936ns	4.93ns	23.6ns	0	576 B
#7441	`StartFinishSpan`	net472	918ns	0.75ns	2.9ns	0.0914	578 B
#7441	`StartFinishScope`	net6.0	882ns	4.82ns	27.3ns	0	696 B
#7441	`StartFinishScope`	netcoreapp3.1	1.24μs	1.34ns	5.19ns	0	696 B
#7441	`StartFinishScope`	net472	1.09μs	0.492ns	1.91ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	1.06μs	5.05ns	20.8ns	0	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.36μs	4.01ns	15.5ns	0	696 B
master	`RunOnMethodBegin`	net472	1.4μs	0.391ns	1.51ns	0.0979	658 B
#7441	`RunOnMethodBegin`	net6.0	1.04μs	0.547ns	2.04ns	0	696 B
#7441	`RunOnMethodBegin`	netcoreapp3.1	1.41μs	7.1ns	33.3ns	0	696 B
#7441	`RunOnMethodBegin`	net472	1.41μs	2.69ns	10.4ns	0.0979	658 B

tracer/src/Datadog.Tracer.Native/debugger_probes_instrumentation_requester.cpp

anna-git

thank you! I left one small NIT comment

tracer/src/Datadog.Tracer.Native/debugger_environment_variables.h

tracer/src/Datadog.Tracer.Native/debugger_environment_variables_util.cpp

…adog.Trace

…yncMethodDebuggerInvokerV2` does not exist (#7513) ## Summary of changes In #7441, the injected field of async methods has been changed from `AsyncDebuggerState` to `System.Object`. In this PR, we make sure instrumentation is not being applied if the loaded `Datadog.Trace` does not contain the new signatures that accept `System.Object` for log/span probes. ## Reason for change & Implementation Avoid instrumentation of async methods if the loaded `Datadog.Trace` is an old version that does not contain the new signatures introduced in #7441. The only place where we can change the layout of a type is in `ModuleLoadFinished`, this is why injection of our field is done there. This callback is fired by the runtime when a new module is being loaded. Modules of interest (including application's main module) may load before our managed loader had a chance to resolve and load our managed `Datadog.Trace` assembly into the application. The fact that at injection time we don't know if and which version of `Datadog.Trace` is going to be loaded, we cannot guarantee that the new signatures will be there. Once `Datadog.Trace` actually loads, we try to lookup for the existence of the type `AsyncMethodDebuggerInvokerV2` as an anchor to know if the new signatures are there. If the type does not exist, we skip instrumentation on async methods (as there are no methods that accept the newly injected `System.Object`), log appropriate message and emit probe status with a message that invites the customer to upgrade `Datadog.Trace`. In the case of bailing out our async instrumentation, the injected `System.Object` field will effectively be a NOP.

## Summary of changes This PR adds remote enablement support for debugger products. depends on #7441 ## Reason for change Currently, the only option to enable DI (for example) is by setting an environment variable. We want to give customers the ability to enable products on demand. ## Implementation details Gets RC events with new configuration and update the products based on defined rules. ## Test coverage All existing tests New system tests (DataDog/system-tests#4991) DebuggerManagerTests DebuggerManagerDynamicTests --------- Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>

This comment has been minimized.

Sign in to view

GreenMatan force-pushed the matang/handsoff-config-di branch from c68d1f2 to c42cbea Compare August 27, 2025 11:56

GreenMatan marked this pull request as ready for review August 27, 2025 11:57

GreenMatan requested a review from a team as a code owner August 27, 2025 11:57

GreenMatan requested a review from anna-git August 27, 2025 12:02

GreenMatan changed the title ~~[HandsOff Config] Decouple native code from DI/ER env vars~~ [Stable Configuration] Decouple native code from DI/ER env vars Aug 27, 2025

anna-git reviewed Aug 28, 2025

View reviewed changes

tracer/src/Datadog.Tracer.Native/debugger_probes_instrumentation_requester.cpp Show resolved Hide resolved

GreenMatan force-pushed the matang/handsoff-config-di branch 3 times, most recently from 6492b8b to 04e7415 Compare August 28, 2025 11:37

tylfin approved these changes Aug 28, 2025

View reviewed changes

anna-git approved these changes Aug 29, 2025

View reviewed changes

tracer/src/Datadog.Tracer.Native/debugger_environment_variables.h Outdated Show resolved Hide resolved

tracer/src/Datadog.Tracer.Native/debugger_environment_variables_util.cpp Outdated Show resolved Hide resolved

anna-git reviewed Aug 29, 2025

View reviewed changes

tracer/src/Datadog.Tracer.Native/debugger_environment_variables_util.cpp Outdated Show resolved Hide resolved

GreenMatan changed the title ~~[Stable Configuration] Decouple native code from DI/ER env vars~~ [Stable Configuration] Keep DI/ER in hot standby Aug 30, 2025

GreenMatan force-pushed the matang/handsoff-config-di branch 4 times, most recently from 982e215 to d353039 Compare September 1, 2025 20:28

dudikeleti mentioned this pull request Sep 2, 2025

[Debugger Default-On] DEBUG-3322 Debugger in-product enablement #7366

Merged

GreenMatan force-pushed the matang/handsoff-config-di branch 9 times, most recently from a15e7d1 to 228a93c Compare September 11, 2025 15:05

GreenMatan requested a review from a team as a code owner September 11, 2025 15:05

GreenMatan force-pushed the matang/handsoff-config-di branch 2 times, most recently from 642548f to a6fbdb0 Compare September 11, 2025 15:26

GreenMatan added 2 commits September 12, 2025 21:15

decoupled the native side from DI/ER env vars to support HandsOff Config

8666810

TO REVERT: ADDED LOGS

3be43b0

GreenMatan force-pushed the matang/handsoff-config-di branch from a6fbdb0 to b697c38 Compare September 12, 2025 18:15

GreenMatan changed the title ~~[Stable Configuration] Keep DI/ER in hot standby~~ [Dynamic Instrumentation] Native hot standby mode to support Remote Enablement and Stable Config Sep 12, 2025

GreenMatan changed the title ~~[Dynamic Instrumentation] Native hot standby mode to support Remote Enablement and Stable Config~~ [Dynamic Instrumentation] Added native hot standby mode to support Remote Enablement and Stable Config Sep 12, 2025

GreenMatan force-pushed the matang/handsoff-config-di branch from b697c38 to c8e7a40 Compare September 12, 2025 21:03

github-actions bot added the area:debugger label Sep 12, 2025

got rid of injecting fields into statemachine that are defined in Dat…

a241260

…adog.Trace

GreenMatan force-pushed the matang/handsoff-config-di branch from c8e7a40 to a241260 Compare September 12, 2025 23:13

GreenMatan merged commit 50ee1f9 into master Sep 13, 2025
156 checks passed

GreenMatan deleted the matang/handsoff-config-di branch September 13, 2025 10:15

github-actions bot added this to the vNext-v3 milestone Sep 13, 2025

GreenMatan mentioned this pull request Sep 13, 2025

[Dynamic Instrumentation] Avoid async instrumentation if the type AsyncMethodDebuggerInvokerV2 does not exist #7513

Merged

Conversation

GreenMatan commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Implementation

Test

Reason for change

Uh oh!

dd-trace-dotnet-ci-bot bot commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

Uh oh!

This comment has been minimized.

pr-commenter bot commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmark details

Fewer allocations 🎉 in #7441

Raw results

Fewer allocations 🎉 in #7441

Raw results

Raw results

Raw results

Faster 🎉 in #7441

Raw results

Raw results

More allocations ⚠️ in #7441

Fewer allocations 🎉 in #7441

Raw results

More allocations ⚠️ in #7441

Raw results

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #7441

Fewer allocations 🎉 in #7441

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Uh oh!

Uh oh!

anna-git left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GreenMatan commented Aug 27, 2025 •

edited

Loading

dd-trace-dotnet-ci-bot bot commented Aug 27, 2025 •

edited

Loading

pr-commenter bot commented Aug 27, 2025 •

edited

Loading