Use P/Invoke to call `lstat` instead of shelling out to `stat` to retrieve the inode by andrewlock · Pull Request #7453 · DataDog/dd-trace-dotnet

andrewlock · 2025-08-29T14:06:49Z

Summary of changes

Adds a P/Invoke path for retrieving the inode on linux and only later falls back to using stat -c %i <path>

Reason for change

Calling stat will fail if stat isn't available (as in chiseled containers, for example). It also results in additional spans being created when we're running in a version-mismatch scenario(because the Process invocation is traced)

Implementation details

Based my initial PoC on P/Invoking into System.Native, which is a shim shipped with recent .NET that handles a bunch of marshalling issues etc and returns this type. That's how recent .NET versions handle various bits like this.

However, System.Native only gives the correct inode value in .NET 7+ (from my local testing) and isn't available in early .NET Core. I then tried the "raw" route, trying to P/Invoke directly into libc. I also investigated making the syscall directly, but libc papers over differences in old kernels etc, so it seemed the easiest route. However, older versions of libc don't export the lstat or fstatat symbols as public symbols, and instead they're just macros.

Finally settled on a similar approach to System.Native, i.e. we P/Invoke into Datadog.Trace.Native which then makes the lstat call.

Test coverage

Should mostly be covered by existing tests (and smoke tests). I added an additional unit test to make sure the stat -c %i call and the P/Invoke give the same value, otherwise we should be covered.

I also rebased on top of this branch which is where the error appeared - and did a full-installer run, and it fixed all the issues, so we should be good 🤞 And we still have the existing fallback if not

Note that I explicitly disabled the process spans in the version-conflict tests, as whether we get the spans or not is dependent on the underlying host system, which is a bit gross.

Other details

~~AI assisted in various places, including generating the structs.~~ None of that worked in the end. Great job AI.

I made the failure an error as it shouldn't fail in normal execution, and we'd like to report if it does, however there are some cases where the P/Invoke calls aren't re-written so we know this won't work. In case of failure, we explicitly check for that, and log the error as debug instead.

To get everything to compile easily I split the ContainerMetadata implementation for .NET FX as it's not supported in that scenario anyway.

Also, to be able to unit test the P/Invoke work, I copied the native libs into the test directories.

Finally, it's worth us looking into this method to allow P/Invokes to happen in the native runner, given we know where the native libs are, they're just not re-written. Something to think about in a separate PR I think

datadog-datadog-prod-us1 · 2025-08-29T14:58:09Z

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 21c92be | Docs | Was this helpful? Give us feedback!}

pr-commenter · 2025-08-29T15:25:32Z

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #7453 compared to master:

1 benchmarks are faster, with geometric mean 2.087
2 benchmarks are slower, with geometric mean 1.856
3 benchmarks have fewer allocations
6 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7453

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net472	6.09 KB	5.98 KB	-108 B	-1.77%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	10.5μs	57.5ns	350ns	0	0	0	5.5 KB
master	`StartStopWithChild`	netcoreapp3.1	13.2μs	64.6ns	266ns	0	0	0	5.7 KB
master	`StartStopWithChild`	net472	22.4μs	124ns	732ns	0.945	0.42	0	6.09 KB
#7453	`StartStopWithChild`	net6.0	10.6μs	55.5ns	272ns	0	0	0	5.49 KB
#7453	`StartStopWithChild`	netcoreapp3.1	13.5μs	69.1ns	317ns	0	0	0	5.69 KB
#7453	`StartStopWithChild`	net472	22.1μs	126ns	969ns	0.96	0.32	0.107	5.98 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	936μs	63.9ns	247ns	2.71 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	1.02ms	176ns	634ns	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	1.2ms	304ns	1.1μs	3.31 KB
#7453	`WriteAndFlushEnrichedTraces`	net6.0	940μs	126ns	487ns	2.7 KB
#7453	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	1.04ms	213ns	769ns	2.7 KB
#7453	`WriteAndFlushEnrichedTraces`	net472	1.21ms	1.2μs	4.65μs	3.31 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`AllCycleSimpleBody`	net6.0	322μs	1.32μs	5.1μs	0	172.07 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	462μs	972ns	3.77μs	0	174.18 KB
master	`AllCycleSimpleBody`	net472	438μs	315ns	1.22μs	30.2	194.25 KB
master	`AllCycleMoreComplexBody`	net6.0	332μs	1.56μs	6.05μs	0	175.58 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	507μs	513ns	1.99μs	0	177.6 KB
master	`AllCycleMoreComplexBody`	net472	437μs	174ns	674ns	30.2	197.76 KB
master	`ObjectExtractorSimpleBody`	net6.0	317ns	0.108ns	0.42ns	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	402ns	2.18ns	11.3ns	0	272 B
master	`ObjectExtractorSimpleBody`	net472	300ns	0.0488ns	0.189ns	0.0439	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	6.22μs	23.1ns	89.3ns	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	7.86μs	30.4ns	118ns	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	6.66μs	2.03ns	7.86ns	0.599	3.8 KB
#7453	`AllCycleSimpleBody`	net6.0	326μs	821ns	3.18μs	0	172.07 KB
#7453	`AllCycleSimpleBody`	netcoreapp3.1	457μs	1.16μs	4.49μs	0	174.18 KB
#7453	`AllCycleSimpleBody`	net472	432μs	191ns	716ns	30.2	194.25 KB
#7453	`AllCycleMoreComplexBody`	net6.0	329μs	1.13μs	4.22μs	0	175.58 KB
#7453	`AllCycleMoreComplexBody`	netcoreapp3.1	503μs	1.17μs	4.38μs	0	177.6 KB
#7453	`AllCycleMoreComplexBody`	net472	434μs	150ns	541ns	30.2	197.76 KB
#7453	`ObjectExtractorSimpleBody`	net6.0	322ns	0.214ns	0.772ns	0	280 B
#7453	`ObjectExtractorSimpleBody`	netcoreapp3.1	409ns	0.648ns	2.51ns	0	272 B
#7453	`ObjectExtractorSimpleBody`	net472	300ns	0.04ns	0.155ns	0.0437	281 B
#7453	`ObjectExtractorMoreComplexBody`	net6.0	6.21μs	31ns	135ns	0	3.78 KB
#7453	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	7.75μs	34.2ns	133ns	0	3.69 KB
#7453	`ObjectExtractorMoreComplexBody`	net472	6.74μs	1.9ns	7.12ns	0.571	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EncodeArgs`	net6.0	76.8μs	220ns	823ns	0	32.4 KB
master	`EncodeArgs`	netcoreapp3.1	98.5μs	245ns	948ns	0	32.4 KB
master	`EncodeArgs`	net472	109μs	48.6ns	188ns	4.92	32.51 KB
master	`EncodeLegacyArgs`	net6.0	141μs	30.7ns	119ns	0	2.15 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	199μs	221ns	854ns	0	2.14 KB
master	`EncodeLegacyArgs`	net472	262μs	40.4ns	156ns	0	2.16 KB
#7453	`EncodeArgs`	net6.0	77.6μs	55.5ns	208ns	0	32.4 KB
#7453	`EncodeArgs`	netcoreapp3.1	98.5μs	301ns	1.17μs	0	32.4 KB
#7453	`EncodeArgs`	net472	109μs	11.1ns	43ns	4.9	32.51 KB
#7453	`EncodeLegacyArgs`	net6.0	143μs	93.4ns	349ns	0	2.15 KB
#7453	`EncodeLegacyArgs`	netcoreapp3.1	198μs	289ns	1.12μs	0	2.14 KB
#7453	`EncodeLegacyArgs`	net472	262μs	46.1ns	179ns	0	2.16 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #7453

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark‑netcoreapp3.1	2.087	861,427.05	412,820.10

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`RunWafRealisticBenchmark`	net6.0	396μs	122ns	458ns	4.56 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	792μs	14.9μs	148μs	4.48 KB
master	`RunWafRealisticBenchmark`	net472	434μs	48.4ns	188ns	4.66 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	292μs	50.3ns	195ns	2.24 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	724μs	3.14μs	11.3μs	2.22 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	313μs	70.9ns	275ns	2.3 KB
#7453	`RunWafRealisticBenchmark`	net6.0	397μs	69ns	258ns	4.56 KB
#7453	`RunWafRealisticBenchmark`	netcoreapp3.1	413μs	259ns	1μs	4.48 KB
#7453	`RunWafRealisticBenchmark`	net472	430μs	37.5ns	140ns	4.66 KB
#7453	`RunWafRealisticBenchmarkWithAttack`	net6.0	289μs	93.8ns	363ns	2.24 KB
#7453	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	674μs	14μs	140μs	2.22 KB
#7453	`RunWafRealisticBenchmarkWithAttack`	net472	313μs	27.6ns	107ns	2.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`SendRequest`	net6.0	61.3μs	41.5ns	161ns	14.52 KB
master	`SendRequest`	netcoreapp3.1	71.3μs	93.8ns	325ns	17.42 KB
master	`SendRequest`	net472	0.0117ns	0.00328ns	0.0127ns	0 b
#7453	`SendRequest`	net6.0	61.4μs	46ns	172ns	14.52 KB
#7453	`SendRequest`	netcoreapp3.1	70.8μs	96.8ns	349ns	17.42 KB
#7453	`SendRequest`	net472	0.0221ns	0.00327ns	0.0127ns	0 b

Benchmarks.Trace.CharSliceBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7453

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net6.0	2 B	5 B	3 B	150.00%

Fewer allocations 🎉 in #7453

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑netcoreapp3.1	1 B	0 b	-1 B	-100.00%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`OriginalCharSlice`	net6.0	1.97ms	644ns	2.41μs	0	640.01 KB
master	`OriginalCharSlice`	netcoreapp3.1	2.13ms	11.2μs	57μs	0	640 KB
master	`OriginalCharSlice`	net472	2.73ms	174ns	650ns	100	641.95 KB
master	`OptimizedCharSlice`	net6.0	1.36ms	284ns	1.1μs	0	7 B
master	`OptimizedCharSlice`	netcoreapp3.1	1.72ms	295ns	1.14μs	0	1 B
master	`OptimizedCharSlice`	net472	1.95ms	444ns	1.72μs	0	0 b
master	`OptimizedCharSliceWithPool`	net6.0	916μs	50.4ns	195ns	0	2 B
master	`OptimizedCharSliceWithPool`	netcoreapp3.1	827μs	72.6ns	262ns	0	1 B
master	`OptimizedCharSliceWithPool`	net472	1.16ms	87.6ns	316ns	0	0 b
#7453	`OriginalCharSlice`	net6.0	2.02ms	747ns	2.79μs	0	640 KB
#7453	`OriginalCharSlice`	netcoreapp3.1	2.11ms	3.78μs	14.1μs	0	640 KB
#7453	`OriginalCharSlice`	net472	2.68ms	459ns	1.78μs	100	641.95 KB
#7453	`OptimizedCharSlice`	net6.0	1.43ms	451ns	1.75μs	0	7 B
#7453	`OptimizedCharSlice`	netcoreapp3.1	1.8ms	207ns	801ns	0	1 B
#7453	`OptimizedCharSlice`	net472	2.01ms	499ns	1.93μs	0	0 b
#7453	`OptimizedCharSliceWithPool`	net6.0	835μs	46.4ns	180ns	0	5 B
#7453	`OptimizedCharSliceWithPool`	netcoreapp3.1	833μs	68.7ns	266ns	0	0 b
#7453	`OptimizedCharSliceWithPool`	net472	1.13ms	53.7ns	201ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	694μs	571ns	2.06μs	0	41.65 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	677μs	2.11μs	7.29μs	0	41.73 KB
master	`WriteAndFlushEnrichedTraces`	net472	905μs	4.57μs	20.4μs	8.33	55.98 KB
#7453	`WriteAndFlushEnrichedTraces`	net6.0	651μs	729ns	2.82μs	0	41.67 KB
#7453	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	627μs	1.61μs	5.8μs	0	41.85 KB
#7453	`WriteAndFlushEnrichedTraces`	net472	953μs	2.76μs	10.7μs	4.46	56.17 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.94μs	5.81ns	22.5ns	0	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	2.59μs	11.6ns	44.9ns	0	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.83μs	3.08ns	11.9ns	0.155	0.0141	987 B
#7453	`ExecuteNonQuery`	net6.0	1.94μs	7.9ns	30.6ns	0	0	1.02 KB
#7453	`ExecuteNonQuery`	netcoreapp3.1	2.48μs	3.76ns	14.6ns	0	0	1.02 KB
#7453	`ExecuteNonQuery`	net472	2.69μs	3.46ns	13.4ns	0.15	0.0136	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.71μs	8.15ns	33.6ns	0	1.03 KB
master	`CallElasticsearch`	netcoreapp3.1	2.35μs	6.04ns	23.4ns	0	1.03 KB
master	`CallElasticsearch`	net472	3.61μs	2.6ns	10.1ns	0.162	1.04 KB
master	`CallElasticsearchAsync`	net6.0	1.85μs	2.16ns	8.35ns	0	1.01 KB
master	`CallElasticsearchAsync`	netcoreapp3.1	2.38μs	11.8ns	51.3ns	0	1.08 KB
master	`CallElasticsearchAsync`	net472	3.74μs	3.68ns	14.2ns	0.168	1.1 KB
#7453	`CallElasticsearch`	net6.0	1.71μs	1.61ns	6.22ns	0	1.03 KB
#7453	`CallElasticsearch`	netcoreapp3.1	2.27μs	10.9ns	43.5ns	0	1.03 KB
#7453	`CallElasticsearch`	net472	3.59μs	0.83ns	2.99ns	0.162	1.04 KB
#7453	`CallElasticsearchAsync`	net6.0	1.87μs	0.531ns	1.99ns	0	1.01 KB
#7453	`CallElasticsearchAsync`	netcoreapp3.1	2.37μs	11.4ns	44.2ns	0	1.08 KB
#7453	`CallElasticsearchAsync`	net472	3.88μs	4.5ns	17.4ns	0.173	1.1 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.87μs	2.26ns	8.47ns	0	952 B
master	`ExecuteAsync`	netcoreapp3.1	2.33μs	11.5ns	47.2ns	0	952 B
master	`ExecuteAsync`	net472	2.58μs	4.02ns	15.6ns	0.14	915 B
#7453	`ExecuteAsync`	net6.0	1.81μs	6.43ns	24.9ns	0	952 B
#7453	`ExecuteAsync`	netcoreapp3.1	2.38μs	6.9ns	25.8ns	0	952 B
#7453	`ExecuteAsync`	net472	2.46μs	3.76ns	14.5ns	0.136	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	7.08μs	31.8ns	119ns	0	2.36 KB
master	`SendAsync`	netcoreapp3.1	8.35μs	26.9ns	104ns	0	2.9 KB
master	`SendAsync`	net472	12.2μs	8.46ns	31.6ns	0.486	3.18 KB
#7453	`SendAsync`	net6.0	7.09μs	22.3ns	86.3ns	0	2.36 KB
#7453	`SendAsync`	netcoreapp3.1	8.56μs	19.3ns	72.1ns	0	2.9 KB
#7453	`SendAsync`	net472	12.2μs	6.76ns	26.2ns	0.485	3.18 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Slower ⚠️ More allocations ⚠️

Slower ⚠️ in #7453

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0	2.764	452,850.00	1,251,500.00
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	1.247	417,850.00	520,950.00

More allocations ⚠️ in #7453

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0	258.64 KB	319.46 KB	60.82 KB	23.51%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472	57.34 KB	65.54 KB	8.19 KB	14.29%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net472	277.66 KB	294.91 KB	17.25 KB	6.21%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net6.0	44.33 KB	46.48 KB	2.15 KB	4.85%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑netcoreapp3.1	42.64 KB	43.28 KB	640 B	1.50%

Fewer allocations 🎉 in #7453

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	276.1 KB	273.93 KB	-2.17 KB	-0.79%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	49μs	387ns	3.67μs	44.33 KB
master	`StringConcatBenchmark`	netcoreapp3.1	49.7μs	236ns	1.9μs	42.64 KB
master	`StringConcatBenchmark`	net472	57.9μs	130ns	449ns	57.34 KB
master	`StringConcatAspectBenchmark`	net6.0	457μs	2.16μs	9.4μs	258.64 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	441μs	4.74μs	46.2μs	276.1 KB
master	`StringConcatAspectBenchmark`	net472	407μs	2.25μs	17μs	277.66 KB
#7453	`StringConcatBenchmark`	net6.0	47.9μs	241ns	1.68μs	46.48 KB
#7453	`StringConcatBenchmark`	netcoreapp3.1	49.3μs	256ns	1.81μs	43.28 KB
#7453	`StringConcatBenchmark`	net472	57.3μs	218ns	846ns	65.54 KB
#7453	`StringConcatAspectBenchmark`	net6.0	1.25ms	2.43μs	8.41μs	319.46 KB
#7453	`StringConcatAspectBenchmark`	netcoreapp3.1	520μs	2.62μs	12.3μs	273.93 KB
#7453	`StringConcatAspectBenchmark`	net472	405μs	2.04μs	9.12μs	294.91 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.52μs	13.6ns	70.5ns	0	1.7 KB
master	`EnrichedLog`	netcoreapp3.1	3.47μs	12.3ns	47.8ns	0	1.7 KB
master	`EnrichedLog`	net472	3.91μs	4.06ns	15.7ns	0.254	1.64 KB
#7453	`EnrichedLog`	net6.0	2.52μs	1.3ns	4.68ns	0	1.7 KB
#7453	`EnrichedLog`	netcoreapp3.1	3.37μs	16.7ns	68.8ns	0	1.7 KB
#7453	`EnrichedLog`	net472	4.03μs	3.63ns	14.1ns	0.24	1.64 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`EnrichedLog`	net6.0	123μs	31.3ns	113ns	4.31 KB
master	`EnrichedLog`	netcoreapp3.1	128μs	205ns	769ns	4.31 KB
master	`EnrichedLog`	net472	168μs	40.6ns	157ns	4.52 KB
#7453	`EnrichedLog`	net6.0	123μs	94.3ns	365ns	4.31 KB
#7453	`EnrichedLog`	netcoreapp3.1	128μs	112ns	418ns	4.31 KB
#7453	`EnrichedLog`	net472	168μs	149ns	576ns	4.52 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	4.96μs	16ns	57.8ns	0	2.26 KB
master	`EnrichedLog`	netcoreapp3.1	6.64μs	15.1ns	58.6ns	0	2.26 KB
master	`EnrichedLog`	net472	7.64μs	4.79ns	17.9ns	0.306	2.08 KB
#7453	`EnrichedLog`	net6.0	5.29μs	18.7ns	72.2ns	0	2.26 KB
#7453	`EnrichedLog`	netcoreapp3.1	6.71μs	10.7ns	41.6ns	0	2.26 KB
#7453	`EnrichedLog`	net472	7.42μs	5.8ns	22.5ns	0.296	2.08 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.96μs	8.88ns	35.5ns	0	1.2 KB
master	`SendReceive`	netcoreapp3.1	2.65μs	8.52ns	30.7ns	0	1.2 KB
master	`SendReceive`	net472	3.11μs	2.93ns	11ns	0.185	1.2 KB
#7453	`SendReceive`	net6.0	1.98μs	0.792ns	3.07ns	0	1.2 KB
#7453	`SendReceive`	netcoreapp3.1	2.74μs	9.09ns	35.2ns	0	1.2 KB
#7453	`SendReceive`	net472	3μs	2.24ns	8.69ns	0.18	1.2 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	4.15μs	7.6ns	29.4ns	0	1.58 KB
master	`EnrichedLog`	netcoreapp3.1	5.61μs	12ns	44.9ns	0	1.63 KB
master	`EnrichedLog`	net472	6.63μs	6.9ns	26.7ns	0.298	2.03 KB
#7453	`EnrichedLog`	net6.0	4.07μs	5.76ns	22.3ns	0	1.58 KB
#7453	`EnrichedLog`	netcoreapp3.1	5.43μs	9.98ns	38.7ns	0	1.63 KB
#7453	`EnrichedLog`	net472	6.39μs	5.74ns	20.7ns	0.318	2.03 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	733ns	3.49ns	13.5ns	0	576 B
master	`StartFinishSpan`	netcoreapp3.1	945ns	5ns	22.4ns	0	576 B
master	`StartFinishSpan`	net472	896ns	0.431ns	1.61ns	0.0904	578 B
master	`StartFinishScope`	net6.0	884ns	4.87ns	30.4ns	0	696 B
master	`StartFinishScope`	netcoreapp3.1	1.17μs	5.23ns	20.3ns	0	696 B
master	`StartFinishScope`	net472	1.1μs	0.952ns	3.56ns	0.0991	658 B
#7453	`StartFinishSpan`	net6.0	746ns	0.201ns	0.778ns	0	576 B
#7453	`StartFinishSpan`	netcoreapp3.1	978ns	0.743ns	2.78ns	0	576 B
#7453	`StartFinishSpan`	net472	920ns	0.82ns	3.07ns	0.0877	578 B
#7453	`StartFinishScope`	net6.0	878ns	4.59ns	21.5ns	0	696 B
#7453	`StartFinishScope`	netcoreapp3.1	1.17μs	0.633ns	2.37ns	0	696 B
#7453	`StartFinishScope`	net472	1.08μs	0.149ns	0.578ns	0.103	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	1.03μs	5.21ns	25ns	0	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.44μs	0.341ns	1.18ns	0	696 B
master	`RunOnMethodBegin`	net472	1.36μs	1.39ns	5.39ns	0.102	658 B
#7453	`RunOnMethodBegin`	net6.0	1.01μs	5.02ns	23ns	0	696 B
#7453	`RunOnMethodBegin`	netcoreapp3.1	1.41μs	6.85ns	28.2ns	0	696 B
#7453	`RunOnMethodBegin`	net472	1.39μs	0.808ns	3.13ns	0.0978	658 B

dd-trace-dotnet-ci-bot · 2025-08-29T15:33:35Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7453) - mean (73ms)  : 72, 75
     .   : milestone, 73,
    master - mean (73ms)  : 72, 75
     .   : milestone, 73,

    section Baseline
    This PR (7453) - mean (71ms)  : 65, 77
     .   : milestone, 71,
    master - mean (70ms)  : 65, 76
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (7453) - mean (994ms)  : 971, 1017
     .   : milestone, 994,
    master - mean (995ms)  : 968, 1022
     .   : milestone, 995,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7453) - mean (109ms)  : 107, 111
     .   : milestone, 109,
    master - mean (109ms)  : 108, 110
     .   : milestone, 109,

    section Baseline
    This PR (7453) - mean (108ms)  : 105, 110
     .   : milestone, 108,
    master - mean (108ms)  : 105, 111
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (7453) - mean (707ms)  : 685, 729
     .   : milestone, 707,
    master - mean (705ms)  : 684, 726
     .   : milestone, 705,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7453) - mean (95ms)  : 94, 97
     .   : milestone, 95,
    master - mean (96ms)  : 95, 98
     .   : milestone, 96,

    section Baseline
    This PR (7453) - mean (95ms)  : 93, 97
     .   : milestone, 95,
    master - mean (96ms)  : 93, 98
     .   : milestone, 96,

    section CallTarget+Inlining+NGEN
    This PR (7453) - mean (656ms)  : 636, 677
     .   : milestone, 656,
    master - mean (661ms)  : 644, 679
     .   : milestone, 661,

gantt
    title Execution time (ms) FakeDbCommand (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7453) - mean (95ms)  : 93, 97
     .   : milestone, 95,
    master - mean (96ms)  : 94, 97
     .   : milestone, 96,

    section Baseline
    This PR (7453) - mean (94ms)  : 92, 96
     .   : milestone, 94,
    master - mean (95ms)  : 92, 97
     .   : milestone, 95,

    section CallTarget+Inlining+NGEN
    This PR (7453) - mean (594ms)  : 581, 608
     .   : milestone, 594,
    master - mean (599ms)  : 586, 612
     .   : milestone, 599,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7453) - mean (193ms)  : 191, 196
     .   : milestone, 193,
    master - mean (194ms)  : 191, 197
     .   : milestone, 194,

    section Baseline
    This PR (7453) - mean (190ms)  : 186, 194
     .   : milestone, 190,
    master - mean (191ms)  : 186, 196
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (7453) - mean (1,098ms)  : 1066, 1130
     .   : milestone, 1098,
    master - mean (1,102ms)  : 1066, 1138
     .   : milestone, 1102,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7453) - mean (275ms)  : 270, 279
     .   : milestone, 275,
    master - mean (278ms)  : 273, 282
     .   : milestone, 278,

    section Baseline
    This PR (7453) - mean (274ms)  : 270, 279
     .   : milestone, 274,
    master - mean (276ms)  : 271, 281
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (7453) - mean (886ms)  : 853, 919
     .   : milestone, 886,
    master - mean (891ms)  : 867, 916
     .   : milestone, 891,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7453) - mean (268ms)  : 263, 273
     .   : milestone, 268,
    master - mean (268ms)  : 265, 272
     .   : milestone, 268,

    section Baseline
    This PR (7453) - mean (267ms)  : 262, 272
     .   : milestone, 267,
    master - mean (268ms)  : 264, 272
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (7453) - mean (876ms)  : 840, 912
     .   : milestone, 876,
    master - mean (873ms)  : 846, 900
     .   : milestone, 873,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7453) - mean (268ms)  : 264, 271
     .   : milestone, 268,
    master - mean (268ms)  : 263, 272
     .   : milestone, 268,

    section Baseline
    This PR (7453) - mean (267ms)  : 262, 271
     .   : milestone, 267,
    master - mean (269ms)  : 265, 272
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (7453) - mean (786ms)  : 762, 810
     .   : milestone, 786,
    master - mean (790ms)  : 773, 807
     .   : milestone, 790,

zacharycmontoya · 2025-08-29T17:16:08Z

tracer/src/Datadog.Trace/PlatformHelpers/ContainerMetadata.cs

+            public static bool GetInode(string path, out long inode)
+            {
+                var arch = RuntimeInformation.ProcessArchitecture;
+                // using AT_FDCWD and AT_SYMLINK_NOFOLLOW for lstat-like behaviour


Was there a particular reason you didn't use lstat if that is the behavior you're trying to achieve?

Yeah, it was because lstat isn't actually exported as a public symbol in some early versions of glibc, and is rather just a macro, so it's not possible to P/Invoke it directly. Unfortunately, it seems that's also true of fstatat, so that doesn't work either 😅 I'm trying a different approach in which we just use the native tracer for interop instead... 🤞

tracer/src/Datadog.Trace/PlatformHelpers/ContainerMetadata.cs

zacharycmontoya

Overall LGTM, but I have a couple of small questions

tracer/build/_build/Build.Steps.cs

…some older versions

…tation

## Summary of changes - Adds a workaround for the version-conflict issue that occurs on app startup on Linux - Allow doing call target modification of version conflict dll - Fix a bug in CallTarget instrumentation where we get the assembly reference wrong ## Reason for change As part of app startup, on _some_ linux distros, we shell out to `stat` to build the container tags/entity ID. We don't want to trace this `Process.Start()` call, so in #5280 we added a flag to skip instrumenting these calls. However, this fix relies on a `[ThreadStatic]` variable, and in version-conflict scenarios (2.x.x manual, 3.x.x automatic) we end up still instrumenting this call, which causes recursion in `Tracer` initialization and [errors](https://app.datadoghq.com/error-tracking?query=service%3Ainstrumentation-telemetry-data%20source%3Adotnet%20%40tracer_version%3A3.29.0.0%20-%40error.is_crash%3Atrue&et-side=activity&order=total_count&refresh_mode=sliding&source=all&sp=%5B%7B%22p%22%3A%7B%22issueId%22%3A%22cbce5fc2-3adf-11f0-a4be-da7ad0900002%22%7D%2C%22i%22%3A%22error-tracking-issue%22%7D%5D&view=spans&from_ts=1761134402500&to_ts=1762344002500&live=true). > Note that since #7453 we don't do a process start at all, but that doesn't help in this situation, because it's the 2.x.x library that's doing the `Process.Start()` ## Implementation details - Use standard call target instrumentation on the 2.x.x version of `Datadog.Trace` (i.e. version conflict only) - Hook the `ProcessHelpers.StartWithDoNotTrace()` method, and set the 3.x.x `_doNotTrace` variable for the duration of the method call - Tweak the Rejit handler so that we _do_ rejit/call target the Datadog.Trace 2.x.x module (but not the 3.x.x module) - Fix a bug in the module builder which was incorrectly injecting a reference to the 2.x.x assembly instead of the 3.x.x assembly ## Test coverage We were already working around this issue in our VersionConflict tests, so I removed the workaround, confirmed that the test failed, then made the fix, and confirmed the tests pass again. ## Other details This will only help for customers using a manual version of 2.49.0+ (when we introduced the `StartWithDoNotTrace()` call). I think that's good enough support. --------- Co-authored-by: Lucas Pimentel <lucas.pimentel@datadoghq.com>

andrewlock added the area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) label Aug 29, 2025

andrewlock changed the title ~~Use PInvoke to call the libc lstat instead of shelling out to stat to retrieve the inode~~ Use P/Invoke to call the libc lstat instead of shelling out to stat to retrieve the inode Aug 29, 2025

andrewlock changed the title ~~Use P/Invoke to call the libc lstat instead of shelling out to stat to retrieve the inode~~ Use P/Invoke to call the libc fstatat instead of shelling out to stat to retrieve the inode Aug 29, 2025

andrewlock marked this pull request as ready for review August 29, 2025 16:00

andrewlock requested review from a team as code owners August 29, 2025 16:00

zacharycmontoya reviewed Aug 29, 2025

View reviewed changes

tracer/src/Datadog.Trace/PlatformHelpers/ContainerMetadata.cs Outdated Show resolved Hide resolved

zacharycmontoya approved these changes Aug 29, 2025

View reviewed changes

lucaspimentel approved these changes Aug 29, 2025

View reviewed changes

andrewlock force-pushed the andrew/use-pinvoke branch 2 times, most recently from 05f4183 to 9ec69fb Compare September 2, 2025 12:24

andrewlock commented Sep 2, 2025

View reviewed changes

tracer/build/_build/Build.Steps.cs Outdated Show resolved Hide resolved

andrewlock commented Sep 2, 2025

View reviewed changes

tracer/build/_build/Build.Steps.cs Outdated Show resolved Hide resolved

andrewlock added 7 commits September 3, 2025 09:43

Add a PInvoke-based alternative for retrieving the inode

5e43e19

ContainerMetadata doesn't work on .NET FX today

0ede90f

Switch to fstatat because lstat isn’t exported as a public symbol in …

cdcf92a

…some older versions

Try a different approach that hopefully works on all platforms

7194ac0

Don't log an error in "known" failure scenarios i.e. manual-instrumen…

06e18b6

…tation

Copy the native tracer into the test folder too

82cd225

Disable process integration to avoid rogue stat spans

21c92be

andrewlock force-pushed the andrew/use-pinvoke branch from 30a3114 to 21c92be Compare September 3, 2025 08:50

andrewlock changed the title ~~Use P/Invoke to call the libc fstatat instead of shelling out to stat to retrieve the inode~~ Use P/Invoke to call the libc lstat instead of shelling out to stat to retrieve the inode Sep 3, 2025

andrewlock changed the title ~~Use P/Invoke to call the libc lstat instead of shelling out to stat to retrieve the inode~~ Use P/Invoke to call lstat instead of shelling out to stat to retrieve the inode Sep 3, 2025

andrewlock merged commit 6b483d7 into master Sep 3, 2025
153 checks passed

andrewlock deleted the andrew/use-pinvoke branch September 3, 2025 12:06

github-actions bot added this to the vNext-v3 milestone Sep 3, 2025

andrewlock mentioned this pull request Nov 6, 2025

Add fix for persistent Process.Start version-conflict issue #7789

Merged

Conversation

andrewlock commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Uh oh!

datadog-datadog-prod-us1 bot commented Aug 29, 2025 • edited by datadog-official bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter bot commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmark details

Fewer allocations 🎉 in #7453

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #7453

Raw results

Raw results

More allocations ⚠️ in #7453

Fewer allocations 🎉 in #7453

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #7453

More allocations ⚠️ in #7453

Fewer allocations 🎉 in #7453

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Uh oh!

dd-trace-dotnet-ci-bot bot commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

Uh oh!

zacharycmontoya Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

andrewlock Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zacharycmontoya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andrewlock commented Aug 29, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Aug 29, 2025 •

edited by datadog-official bot

Loading

pr-commenter bot commented Aug 29, 2025 •

edited

Loading

dd-trace-dotnet-ci-bot bot commented Aug 29, 2025 •

edited

Loading