[Tracing] Support remote config for `DD_TRACE_ENABLED` by lucaspimentel · Pull Request #5181 · DataDog/dd-trace-dotnet

lucaspimentel · 2024-02-12T22:29:35Z

Summary of changes

Allow users to remotely disable tracing through the Datadog UI (aka "remote disablement").

Reason for change

New feature across all tracing libraries to improve onboarding experience.

Implementation details

Map the RCM setting to existing DD_TRACE_ENABLE local config, which all (probably all) instrumentations already check via TracerSettings.TraceEnabled.

Test coverage

Tested manually.
Added DD_TRACE_ENABLE to existing unit and integration tests.
Covered by existing shared tests:

Other details

AIT-9648

datadog-ddstaging · 2024-02-12T22:44:18Z

Datadog Report

Branch report: lpimentel/remote-disablement
Commit report: dd2c589
Test service: dd-trace-dotnet

✅ 0 Failed, 331430 Passed, 1573 Skipped, 41m 28.79s Wall Time

andrewlock · 2024-02-12T22:52:47Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5181) - mean (76ms)  : 66, 86
     .   : milestone, 76,
    master - mean (74ms)  : 65, 83
     .   : milestone, 74,

    section CallTarget+Inlining+NGEN
    This PR (5181) - mean (990ms)  : 969, 1010
     .   : milestone, 990,
    master - mean (985ms)  : 961, 1008
     .   : milestone, 985,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5181) - mean (112ms)  : 108, 116
     .   : milestone, 112,
    master - mean (118ms)  : 96, 140
     .   : milestone, 118,

    section CallTarget+Inlining+NGEN
    This PR (5181) - mean (727ms)  : 702, 752
     .   : milestone, 727,
    master - mean (713ms)  : 691, 735
     .   : milestone, 713,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5181) - mean (97ms)  : 91, 102
     .   : milestone, 97,
    master - mean (95ms)  : 91, 98
     .   : milestone, 95,

    section CallTarget+Inlining+NGEN
    This PR (5181) - mean (672ms)  : 645, 699
     .   : milestone, 672,
    master - mean (671ms)  : 647, 694
     .   : milestone, 671,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5181) - mean (188ms)  : 186, 191
     .   : milestone, 188,
    master - mean (188ms)  : 182, 193
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (5181) - mean (1,067ms)  : 1045, 1090
     .   : milestone, 1067,
    master - mean (1,063ms)  : 1038, 1089
     .   : milestone, 1063,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5181) - mean (272ms)  : 268, 276
     .   : milestone, 272,
    master - mean (269ms)  : 262, 276
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (5181) - mean (870ms)  : 852, 889
     .   : milestone, 870,
    master - mean (861ms)  : 839, 884
     .   : milestone, 861,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5181) - mean (260ms)  : 257, 264
     .   : milestone, 260,
    master - mean (258ms)  : 250, 265
     .   : milestone, 258,

    section CallTarget+Inlining+NGEN
    This PR (5181) - mean (852ms)  : 820, 884
     .   : milestone, 852,
    master - mean (855ms)  : 834, 876
     .   : milestone, 855,

andrewlock · 2024-02-15T19:40:49Z

Benchmarks Report 🐌

Benchmarks for #5181 compared to master:

1 benchmarks are slower, with geometric mean 1.192
1 benchmarks have fewer allocations
2 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.67μs	48.2ns	316ns	0.0269	0.00896	0	7.5 KB
master	`StartStopWithChild`	netcoreapp3.1	10.6μs	56.4ns	309ns	0.0316	0.0158	0	7.59 KB
master	`StartStopWithChild`	net472	17μs	55.7ns	216ns	1.33	0.337	0.101	7.96 KB
#5181	`StartStopWithChild`	net6.0	8.59μs	45.8ns	238ns	0.0368	0.0163	0	7.49 KB
#5181	`StartStopWithChild`	netcoreapp3.1	10.7μs	58.1ns	329ns	0.0215	0.0107	0	7.58 KB
#5181	`StartStopWithChild`	net472	16.8μs	64.4ns	250ns	1.35	0.358	0.114	7.95 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	459μs	325ns	1.26μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	619μs	420ns	1.57μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	814μs	208ns	777ns	0.408	3.3 KB
#5181	`WriteAndFlushEnrichedTraces`	net6.0	482μs	154ns	597ns	0	2.7 KB
#5181	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	597μs	199ns	745ns	0	2.7 KB
#5181	`WriteAndFlushEnrichedTraces`	net472	807μs	335ns	1.25μs	0.398	3.3 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5181

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody‑netcoreapp3.1	1.192	3,855.03	4,597.05

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	40.4μs	46ns	178ns	0.0202	0	2.36 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	44.3μs	80.6ns	312ns	0.022	0	2.34 KB
master	`AllCycleSimpleBody`	net472	46.4μs	15.7ns	60.9ns	0.37	0	2.41 KB
master	`AllCycleMoreComplexBody`	net6.0	214μs	143ns	497ns	0.107	0	9.84 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	226μs	571ns	2.21μs	0.112	0	9.73 KB
master	`AllCycleMoreComplexBody`	net472	241μs	155ns	602ns	1.56	0	9.91 KB
master	`ObjectExtractorSimpleBody`	net6.0	141ns	0.191ns	0.739ns	0.00392	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	198ns	0.139ns	0.519ns	0.00369	0	272 B
master	`ObjectExtractorSimpleBody`	net472	172ns	0.201ns	0.779ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	3μs	1.25ns	4.51ns	0.0524	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.85μs	1.97ns	7.61ns	0.05	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	3.83μs	2.13ns	7.98ns	0.603	0.00574	3.8 KB
#5181	`AllCycleSimpleBody`	net6.0	40.8μs	11.6ns	43.5ns	0.0202	0	2.36 KB
#5181	`AllCycleSimpleBody`	netcoreapp3.1	43.6μs	43.5ns	157ns	0.0218	0	2.34 KB
#5181	`AllCycleSimpleBody`	net472	47.2μs	17.5ns	63.2ns	0.376	0	2.41 KB
#5181	`AllCycleMoreComplexBody`	net6.0	216μs	139ns	519ns	0.107	0	9.84 KB
#5181	`AllCycleMoreComplexBody`	netcoreapp3.1	227μs	421ns	1.63μs	0.113	0	9.73 KB
#5181	`AllCycleMoreComplexBody`	net472	241μs	151ns	586ns	1.56	0	9.91 KB
#5181	`ObjectExtractorSimpleBody`	net6.0	139ns	0.101ns	0.389ns	0.00393	0	280 B
#5181	`ObjectExtractorSimpleBody`	netcoreapp3.1	201ns	0.222ns	0.861ns	0.00374	0	272 B
#5181	`ObjectExtractorSimpleBody`	net472	173ns	0.103ns	0.398ns	0.0446	0	281 B
#5181	`ObjectExtractorMoreComplexBody`	net6.0	2.98μs	1.21ns	4.19ns	0.0535	0	3.78 KB
#5181	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.6μs	2.17ns	8.12ns	0.0506	0	3.69 KB
#5181	`ObjectExtractorMoreComplexBody`	net472	3.84μs	2.91ns	11.3ns	0.601	0.00575	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EncodeArgs`	net6.0	52.7μs	62.6ns	242ns	0.426	0	30.94 KB
master	`EncodeArgs`	netcoreapp3.1	70.6μs	68.3ns	246ns	0.42	0	31.47 KB
master	`EncodeArgs`	net472	85.8μs	131ns	507ns	5.12	0.0853	32.27 KB
master	`EncodeLegacyArgs`	net6.0	130μs	343ns	1.24μs	0.45	0	33.89 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	157μs	658ns	2.55μs	0.387	0	34.09 KB
master	`EncodeLegacyArgs`	net472	213μs	432ns	1.67μs	5.52	0.424	34.99 KB
#5181	`EncodeArgs`	net6.0	52.6μs	95.8ns	371ns	0.421	0	30.94 KB
#5181	`EncodeArgs`	netcoreapp3.1	68.9μs	103ns	384ns	0.411	0	31.47 KB
#5181	`EncodeArgs`	net472	85.7μs	106ns	411ns	5.12	0.0853	32.27 KB
#5181	`EncodeLegacyArgs`	net6.0	127μs	667ns	3.46μs	0.438	0.0626	33.89 KB
#5181	`EncodeLegacyArgs`	netcoreapp3.1	155μs	522ns	1.95μs	0.453	0	34.09 KB
#5181	`EncodeLegacyArgs`	net472	213μs	379ns	1.37μs	5.56	0.419	34.99 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunWafRealisticBenchmark`	net6.0	193μs	183ns	683ns	0.0972	6.51 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	205μs	363ns	1.4μs	0	6.49 KB
master	`RunWafRealisticBenchmark`	net472	224μs	156ns	604ns	1	6.59 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	125μs	80.9ns	313ns	0.0623	4.15 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	136μs	283ns	1.09μs	0	4.14 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	147μs	53.6ns	208ns	0.66	4.19 KB
#5181	`RunWafRealisticBenchmark`	net6.0	192μs	188ns	727ns	0.0953	6.51 KB
#5181	`RunWafRealisticBenchmark`	netcoreapp3.1	203μs	125ns	432ns	0	6.49 KB
#5181	`RunWafRealisticBenchmark`	net472	226μs	93.1ns	361ns	1.02	6.59 KB
#5181	`RunWafRealisticBenchmarkWithAttack`	net6.0	126μs	74.5ns	279ns	0.0627	4.15 KB
#5181	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	134μs	127ns	476ns	0	4.15 KB
#5181	`RunWafRealisticBenchmarkWithAttack`	net472	147μs	47.4ns	184ns	0.659	4.19 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	175μs	229ns	886ns	0.259	18.26 KB
master	`SendRequest`	netcoreapp3.1	194μs	299ns	1.16μs	0.193	20.42 KB
master	`SendRequest`	net472	0.00037ns	0.000168ns	0.000627ns	0	0 b
#5181	`SendRequest`	net6.0	172μs	227ns	880ns	0.172	18.26 KB
#5181	`SendRequest`	netcoreapp3.1	194μs	226ns	875ns	0.193	20.42 KB
#5181	`SendRequest`	net472	0.000264ns	0.000118ns	0.00044ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	534μs	1.28μs	4.95μs	0.541	0	0	41.67 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	659μs	1.15μs	4.46μs	0.327	0	0	41.58 KB
master	`WriteAndFlushEnrichedTraces`	net472	842μs	3.94μs	15.3μs	8.22	2.47	0.411	53.24 KB
#5181	`WriteAndFlushEnrichedTraces`	net6.0	551μs	1.13μs	4.36μs	0.541	0	0	41.65 KB
#5181	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	665μs	1.03μs	3.99μs	0.34	0	0	41.77 KB
#5181	`WriteAndFlushEnrichedTraces`	net472	865μs	4.23μs	17.4μs	8.19	2.59	0.431	53.22 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.12μs	0.443ns	1.72ns	0.0107	776 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.48μs	0.572ns	2.22ns	0.0106	776 B
master	`ExecuteNonQuery`	net472	1.73μs	0.667ns	2.5ns	0.117	738 B
#5181	`ExecuteNonQuery`	net6.0	1.07μs	0.739ns	2.86ns	0.011	776 B
#5181	`ExecuteNonQuery`	netcoreapp3.1	1.49μs	0.743ns	2.78ns	0.0104	776 B
#5181	`ExecuteNonQuery`	net472	1.76μs	3.21ns	12.4ns	0.117	738 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.24μs	0.591ns	2.05ns	0.0136	944 B
master	`CallElasticsearch`	netcoreapp3.1	1.5μs	3.34ns	12.9ns	0.0129	944 B
master	`CallElasticsearch`	net472	2.46μs	0.645ns	2.5ns	0.152	963 B
master	`CallElasticsearchAsync`	net6.0	1.29μs	0.605ns	2.26ns	0.0129	920 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.66μs	2.81ns	10.9ns	0.0134	992 B
master	`CallElasticsearchAsync`	net472	2.7μs	1.58ns	6.12ns	0.161	1.02 KB
#5181	`CallElasticsearch`	net6.0	1.15μs	0.832ns	3.22ns	0.0131	944 B
#5181	`CallElasticsearch`	netcoreapp3.1	1.46μs	0.911ns	3.53ns	0.0125	944 B
#5181	`CallElasticsearch`	net472	2.47μs	1ns	3.89ns	0.152	963 B
#5181	`CallElasticsearchAsync`	net6.0	1.27μs	0.47ns	1.82ns	0.0127	920 B
#5181	`CallElasticsearchAsync`	netcoreapp3.1	1.62μs	1.13ns	4.06ns	0.0128	992 B
#5181	`CallElasticsearchAsync`	net472	2.63μs	1.42ns	5.48ns	0.162	1.02 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.27μs	0.688ns	2.57ns	0.0127	920 B
master	`ExecuteAsync`	netcoreapp3.1	1.58μs	0.947ns	3.67ns	0.0119	920 B
master	`ExecuteAsync`	net472	1.94μs	0.783ns	3.03ns	0.139	883 B
#5181	`ExecuteAsync`	net6.0	1.29μs	0.763ns	2.96ns	0.0129	920 B
#5181	`ExecuteAsync`	netcoreapp3.1	1.63μs	0.487ns	1.82ns	0.0122	920 B
#5181	`ExecuteAsync`	net472	1.82μs	0.85ns	3.29ns	0.14	883 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.14μs	2.21ns	7.97ns	0.0291	2.1 KB
master	`SendAsync`	netcoreapp3.1	5.03μs	1.83ns	6.85ns	0.0353	2.64 KB
master	`SendAsync`	net472	7.72μs	5.55ns	21.5ns	0.523	3.31 KB
#5181	`SendAsync`	net6.0	4.07μs	0.824ns	3.08ns	0.0283	2.1 KB
#5181	`SendAsync`	netcoreapp3.1	4.95μs	2.61ns	9.75ns	0.0346	2.64 KB
#5181	`SendAsync`	net472	7.68μs	3.49ns	13.5ns	0.526	3.31 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5181

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0	204.17 KB	218.26 KB	14.09 KB	6.90%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472	57.34 KB	60.72 KB	3.38 KB	5.89%

Fewer allocations 🎉 in #5181

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net472	229.38 KB	221.18 KB	-8.19 KB	-3.57%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	58.5μs	742ns	7.38μs	43.44 KB
master	`StringConcatBenchmark`	netcoreapp3.1	52.6μs	264ns	1.24μs	42.64 KB
master	`StringConcatBenchmark`	net472	37.5μs	96.4ns	361ns	57.34 KB
master	`StringConcatAspectBenchmark`	net6.0	274μs	1.32μs	6.71μs	204.17 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	268μs	3.48μs	33.7μs	202.87 KB
master	`StringConcatAspectBenchmark`	net472	217μs	1.5μs	13.9μs	229.38 KB
#5181	`StringConcatBenchmark`	net6.0	64.2μs	778ns	7.78μs	43.44 KB
#5181	`StringConcatBenchmark`	netcoreapp3.1	53.4μs	162ns	585ns	42.64 KB
#5181	`StringConcatBenchmark`	net472	37μs	167ns	852ns	60.72 KB
#5181	`StringConcatAspectBenchmark`	net6.0	293μs	2.86μs	26.8μs	218.26 KB
#5181	`StringConcatAspectBenchmark`	netcoreapp3.1	284μs	1.2μs	5.11μs	202.56 KB
#5181	`StringConcatAspectBenchmark`	net472	225μs	1.34μs	12.1μs	221.18 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.53μs	0.753ns	2.82ns	0.0222	1.58 KB
master	`EnrichedLog`	netcoreapp3.1	2.3μs	0.975ns	3.65ns	0.0208	1.58 KB
master	`EnrichedLog`	net472	2.49μs	1.09ns	4.23ns	0.239	1.51 KB
#5181	`EnrichedLog`	net6.0	1.42μs	0.622ns	2.41ns	0.022	1.58 KB
#5181	`EnrichedLog`	netcoreapp3.1	2.24μs	8.79ns	31.7ns	0.0205	1.58 KB
#5181	`EnrichedLog`	net472	2.61μs	0.782ns	2.71ns	0.239	1.51 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	116μs	71.3ns	257ns	0	0	4.22 KB
master	`EnrichedLog`	netcoreapp3.1	119μs	81.8ns	317ns	0.059	0	4.22 KB
master	`EnrichedLog`	net472	147μs	93.9ns	364ns	0.66	0.22	4.4 KB
#5181	`EnrichedLog`	net6.0	112μs	49.2ns	190ns	0.0562	0	4.22 KB
#5181	`EnrichedLog`	netcoreapp3.1	119μs	111ns	428ns	0.0592	0	4.22 KB
#5181	`EnrichedLog`	net472	147μs	76.1ns	295ns	0.657	0.219	4.4 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.02μs	1.35ns	5.23ns	0.0303	2.14 KB
master	`EnrichedLog`	netcoreapp3.1	4.19μs	2.25ns	8.72ns	0.0273	2.14 KB
master	`EnrichedLog`	net472	4.84μs	3.7ns	14.3ns	0.31	1.95 KB
#5181	`EnrichedLog`	net6.0	3.08μs	1.26ns	4.86ns	0.0293	2.14 KB
#5181	`EnrichedLog`	netcoreapp3.1	4.1μs	1.26ns	4.86ns	0.0287	2.14 KB
#5181	`EnrichedLog`	net472	4.95μs	1.01ns	3.92ns	0.309	1.95 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.37μs	0.929ns	3.6ns	0.0151	1.11 KB
master	`SendReceive`	netcoreapp3.1	1.77μs	1.74ns	6.75ns	0.0151	1.11 KB
master	`SendReceive`	net472	2.11μs	0.942ns	3.52ns	0.178	1.12 KB
#5181	`SendReceive`	net6.0	1.41μs	0.651ns	2.43ns	0.0156	1.11 KB
#5181	`SendReceive`	netcoreapp3.1	1.81μs	1.71ns	6.62ns	0.0144	1.11 KB
#5181	`SendReceive`	net472	2.19μs	2.07ns	7.75ns	0.178	1.12 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.75μs	0.771ns	2.99ns	0.0206	1.54 KB
master	`EnrichedLog`	netcoreapp3.1	3.82μs	1.08ns	4.2ns	0.0212	1.58 KB
master	`EnrichedLog`	net472	4.22μs	1.1ns	4.13ns	0.313	1.97 KB
#5181	`EnrichedLog`	net6.0	2.73μs	0.747ns	2.89ns	0.0218	1.54 KB
#5181	`EnrichedLog`	netcoreapp3.1	4.04μs	1.57ns	6.07ns	0.0202	1.58 KB
#5181	`EnrichedLog`	net472	4.29μs	2.11ns	8.15ns	0.314	1.97 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	453ns	0.386ns	1.39ns	0.00753	544 B
master	`StartFinishSpan`	netcoreapp3.1	756ns	1.26ns	4.89ns	0.00722	544 B
master	`StartFinishSpan`	net472	773ns	1.83ns	7.1ns	0.0867	546 B
master	`StartFinishScope`	net6.0	516ns	0.63ns	2.44ns	0.00924	664 B
master	`StartFinishScope`	netcoreapp3.1	824ns	1.26ns	4.89ns	0.00898	664 B
master	`StartFinishScope`	net472	932ns	2.43ns	9.4ns	0.0992	626 B
#5181	`StartFinishSpan`	net6.0	497ns	0.396ns	1.53ns	0.00762	544 B
#5181	`StartFinishSpan`	netcoreapp3.1	739ns	1.56ns	6.05ns	0.00716	544 B
#5181	`StartFinishSpan`	net472	773ns	1.41ns	5.45ns	0.0865	546 B
#5181	`StartFinishScope`	net6.0	513ns	0.567ns	2.2ns	0.00923	664 B
#5181	`StartFinishScope`	netcoreapp3.1	805ns	1.64ns	6.33ns	0.00884	664 B
#5181	`StartFinishScope`	net472	981ns	2.3ns	8.91ns	0.0992	626 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	643ns	1.19ns	4.62ns	0.00937	664 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.02μs	1.83ns	7.08ns	0.00907	664 B
master	`RunOnMethodBegin`	net472	1.1μs	2.11ns	8.17ns	0.0991	626 B
#5181	`RunOnMethodBegin`	net6.0	652ns	1.17ns	4.53ns	0.0094	664 B
#5181	`RunOnMethodBegin`	netcoreapp3.1	967ns	1.55ns	6.02ns	0.00899	664 B
#5181	`RunOnMethodBegin`	net472	1.1μs	1.89ns	7.34ns	0.0992	626 B

andrewlock · 2024-02-15T19:46:45Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5181) (11.048M)   : 0, 11047798
    master (11.301M)   : 0, 11301171
    benchmarks/2.9.0 (10.912M)   : 0, 10912209

    section Automatic
    This PR (5181) (7.777M)   : 0, 7776791
    master (7.786M)   : 0, 7786387
    benchmarks/2.9.0 (8.150M)   : 0, 8149943

    section Trace stats
    This PR (5181) (8.105M)   : 0, 8104722
    master (8.162M)   : 0, 8161989

    section Manual
    This PR (5181) (9.806M)   : 0, 9806451
    master (9.926M)   : 0, 9926297

    section Manual + Automatic
    This PR (5181) (7.387M)   : 0, 7387348
    master (7.412M)   : 0, 7411648

    section Version Conflict
    This PR (5181) (6.662M)   : 0, 6662201
    master (6.730M)   : 0, 6730224

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5181) (9.643M)   : 0, 9642958
    master (9.529M)   : 0, 9529282
    benchmarks/2.9.0 (9.446M)   : 0, 9446328

    section Automatic
    This PR (5181) (6.540M)   : 0, 6540338
    master (6.506M)   : 0, 6505611

    section Trace stats
    This PR (5181) (7.035M)   : 0, 7034753
    master (6.940M)   : 0, 6940301

    section Manual
    This PR (5181) (8.169M)   : 0, 8169375
    master (8.383M)   : 0, 8383218

    section Manual + Automatic
    This PR (5181) (6.326M)   : 0, 6326250
    master (6.263M)   : 0, 6262794

    section Version Conflict
    This PR (5181) (5.639M)   : 0, 5639261
    master (5.698M)   : 0, 5698114

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5181) (10.325M)   : 0, 10324671
    master (10.221M)   : 0, 10221490
    benchmarks/2.9.0 (10.227M)   : 0, 10226734

    section Automatic
    This PR (5181) (7.292M)   : 0, 7292438
    master (7.341M)   : 0, 7340809
    benchmarks/2.9.0 (7.735M)   : 0, 7734705

    section Trace stats
    This PR (5181) (7.620M)   : 0, 7619800
    master (7.697M)   : 0, 7697428

    section Manual
    This PR (5181) (8.989M)   : 0, 8989165
    master (9.139M)   : 0, 9138836

    section Manual + Automatic
    This PR (5181) (6.991M)   : 0, 6991056
    master (7.059M)   : 0, 7058587

    section Version Conflict
    This PR (5181) (6.306M)   : 0, 6306460
    master (6.504M)   : 0, 6503694

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.461M)   : 0, 7461396
    benchmarks/2.9.0 (7.826M)   : 0, 7826159

    section No attack
    master (1.854M)   : 0, 1853890
    benchmarks/2.9.0 (3.248M)   : 0, 3247998

    section Attack
    master (1.450M)   : 0, 1449724
    benchmarks/2.9.0 (2.477M)   : 0, 2477217

    section Blocking
    master (3.161M)   : 0, 3160542

    section IAST default
    master (6.464M)   : 0, 6464188

    section IAST full
    master (5.600M)   : 0, 5600403

    section Base vuln
    master (0.918M)   : 0, 917723

    section IAST vuln
    master (0.859M)   : 0, 859223

tracer/src/Datadog.Trace/RemoteConfigurationManagement/RcmCapabilitiesIndices.cs

tracer/test/Datadog.Trace.Tests/Configuration/DynamicConfigurationTests.cs

zacharycmontoya

Before approving I want to make sure that we have some end-to-end test, I left a comment inquiring about the system-tests conformance

andrewlock

The PR LGTM, but is it sufficient? I don't think we ever test this in integration tests so there's no guarantee that this works for all integrations (just "it should")?

Similarly, if the customer has custom instrumentation, I don't think this will drop the traces? Should we have something in AgentWriter that checks the current trace settings and drops the trace if it shouldn't have been created in the first place? That acts as a back-stop incase anything gets through?

lucaspimentel · 2024-02-16T15:17:30Z

if the customer has custom instrumentation, I don't think this will drop the traces? Should we have something in AgentWriter that checks the current trace settings and drops the trace if it shouldn't have been created in the first place?

We extended DD_TRACE_ENABLED to include manual instrumentation in #796 (v1.19). Currently, this check happens here:

dd-trace-dotnet/tracer/src/Datadog.Trace/Tracer.cs

Lines 387 to 393 in 0ebc75b

    
           void IDatadogTracer.Write(ArraySegment<Span> trace) 
        
           { 
        
               if (Settings.TraceEnabledInternal || Settings.AzureAppServiceMetadata?.CustomTracingEnabled is true) 
        
               { 
        
                   TracerManager.WriteTrace(trace); 
        
               } 
        
           }

lucaspimentel · 2024-02-29T14:38:59Z

tracer/src/Datadog.Trace/RemoteConfigurationManagement/RcmCapabilitiesIndices.cs


+        public static readonly BigInteger AsmExclusionData = Create(18);
+
+        public static readonly BigInteger ApmTracingTracingEnabled = Create(19);


..TracingTracing.. looks weird, but we're following the convention {ProductName}{SettingName}. See also APM_TRACING_TRACING_ENABLED.

It's nothing compared to AutoInstrumentation.ManualInstrumentation 😅

andrewlock

LGTM 👍

Would still be nice to know we definitely don't ship anything if this is disabled. e.g. for future integrations 🤔

andrewlock · 2024-03-05T15:49:02Z

tracer/src/Datadog.Trace/RemoteConfigurationManagement/RcmCapabilitiesIndices.cs


+        public static readonly BigInteger AsmExclusionData = Create(18);
+
+        public static readonly BigInteger ApmTracingTracingEnabled = Create(19);


It's nothing compared to AutoInstrumentation.ManualInstrumentation 😅

lucaspimentel · 2024-03-05T16:23:04Z

Would still be nice to know we definitely don't ship anything if this is disabled. e.g. for future integrations

We know we don't send any spans (even manual ones) to the agent if DD_TRACE_ENABLED=false, because we check this setting in the "span writer".

What I would like to know is if all integrations check this setting before they even create and populate spans, to avoid doing that extra work only to have the span thrown away by the span writer. I'm working on adding more tests for this on another branch (we'll see, it's tricky to generalize for all integrations), but this is a general DD_TRACE_ENABLED concern, not specific to remote configuration. Should we let that block this PR?

zacharycmontoya

LGTM!

commit 832de4b Author: Flavien Darche <11708575+e-n-0@users.noreply.github.com> Date: Tue Mar 12 20:24:21 2024 +0000 [ASM][IAST] Configure maximum IAST Ranges (#5292) * Add configuration key * Use a RangeList in some case to not exceed the max number * Revert some code + implem correct merge * Fix + Add unit and integration tests * Usual macos fix for snapshot * Fix snapshots hashs * Update snapshots (remove other tests as they can't apply different env var values in same run) * Apply comment * Re-integrate integration tests with multiple processes (new fixture) * Add test case for setting MaxRangeCount to zero commit 83f6ab1 Author: Tony Redondo <tony.redondo@datadoghq.com> Date: Tue Mar 12 21:20:39 2024 +0100 [CI Visibility] - Enable snapshot testing of current testing framework implementations (#5226) commit 233695a Author: Daniel Romano <108014683+daniel-romano-DD@users.noreply.github.com> Date: Tue Mar 12 17:06:06 2024 +0100 [IAST] Vulnerability and Evidence truncation (#5302) * Initial implementation * Updated test bundle * Fix test compilation error * Fix snapshot (from rebase) * Fix typo in config value. Updated tests * Fix typo * Refactor converters initialization commit ea31cf5 Author: Anna <anna.yafi@datadoghq.com> Date: Tue Mar 12 16:39:09 2024 +0100 Deactivate benchmark for legacy encoder (#5299) commit d0d713a Author: NachoEchevarria <53266532+NachoEchevarria@users.noreply.github.com> Date: Tue Mar 12 09:25:27 2024 +0100 Set big regex timeouts for tests (#5297) commit d5388d6 Author: Lucas Pimentel <lucas.pimentel@datadoghq.com> Date: Mon Mar 11 15:20:58 2024 -0400 [Tracing] Support configuring `DD_TRACE_ENABLED` remotely (#5181) * add support for remote TraceEnabled setting * fix unrelated typo * add ApmTracingEnabled capability 19 * add missing RCM capability 18 * add mapping * add unit test * add comments to unit test * rename property to match RCM constant * include config in integration tests * fix test json * rewrite tests to use raw values instead of strings commit 2b95f46 Author: Flavien Darche <11708575+e-n-0@users.noreply.github.com> Date: Mon Mar 11 17:47:55 2024 +0100 [ASM][IAST] Support manual JSON deserialisation (Newtonsoft.Json) (#5238) * Add Newtonsoft.Json (non -working yet) * Refactor the tainting proces + add tests * Add the JToken Parse aspect + test * Rename Aspects class + Duck orignal method call * Add integration test * Fix nullability * Fix compilation issue for netfx * Change JSON formatting in ParseTests * Fix a test json format * Refactor NewtonsoftJsonAspects to static constructor commit 0d511f9 Author: Igor Kravchenko <21974069+kr-igor@users.noreply.github.com> Date: Mon Mar 11 09:35:23 2024 -0500 [DSM] - Fixes for IbmMq instrumentation (#5271) * Use byte properties instead of strings * Fixed nullability files * Added some debug info * Fixed lint issues * Added a bit more logs * Using slow byte->sbyte conversion * Added noop headers adapter * Fixed nullability files * Added more logs * Cleaned up debug logs * Removed symlink * Update tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/IbmMq/IbmMqHeadersAdapter.cs Removed debug code Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> * Update tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/IbmMq/IbmMqHeadersAdapter.cs Using Unsafe.As instead of BlockCopy Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> * Update tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/IbmMq/IbmMqHeadersAdapter.cs Use Unsafe.As instead of BlockCopy Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> * Addressed some of the comments * Removed context propagation options --------- Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com> commit 5684a72 Author: Zach Montoya <zach.montoya@datadoghq.com> Date: Fri Mar 8 20:56:30 2024 -0800 [Tracing] Update instrumentation point for DD_TRACE_DELAY_WCF_INSTRUMENTATION_ENABLED=true (#5206) Updates the instrumentation point for `DD_TRACE_DELAY_WCF_INSTRUMENTATION_ENABLED=true` so that now a server span is created immediately before IDispatchMessageInspector implementations are run, so application code can access the root span from inside a IDispatchMessageInspector.AfterReceiveRequest callback. This PR also does some cleanup to remove unused Wcf files and it makes the entire Wcf instrumentation use nullable reference types. commit ca1bb6e Author: Andrew Lock <andrew.lock@datadoghq.com> Date: Fri Mar 8 17:43:57 2024 +0000 Fix errors identified from telemetry (#5279) * Try to avoid MongoDb exception We're seeing exceptions like this: ``` System.FieldAccessException at REDACTED at Datadog.Trace.ClrProfiler.AutoInstrumentation.MongoDb.MongoDbIntegration.CreateScope[TConnection](Object wireProtocol, TConnection connection) at REDACTED at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken) ``` and the only explanation I can think of is a duck-chaining issue, so stopped doing duck chaining and being explicit instead * Add local functions to try to isolate problems * Fix ArgumentNullException in AWS SQS integration

github-actions bot added the area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) label Feb 12, 2024

lucaspimentel force-pushed the lpimentel/remote-disablement branch 2 times, most recently from dad4f05 to b046157 Compare February 15, 2024 18:06

lucaspimentel marked this pull request as ready for review February 15, 2024 22:15

lucaspimentel requested a review from a team as a code owner February 15, 2024 22:15

lucaspimentel force-pushed the lpimentel/remote-disablement branch from 8c75cf7 to fd67d49 Compare February 15, 2024 22:17

zacharycmontoya reviewed Feb 15, 2024

View reviewed changes

tracer/src/Datadog.Trace/RemoteConfigurationManagement/RcmCapabilitiesIndices.cs Show resolved Hide resolved

lucaspimentel requested review from a team February 15, 2024 22:28

lucaspimentel changed the title ~~Support remote-config trace_enabled configuration~~ Support remotely configuring DD_TRACE_ENABLED Feb 15, 2024

lucaspimentel added area:remote-config type:new-feature labels Feb 15, 2024

zacharycmontoya reviewed Feb 15, 2024

View reviewed changes

tracer/test/Datadog.Trace.Tests/Configuration/DynamicConfigurationTests.cs Show resolved Hide resolved

zacharycmontoya reviewed Feb 15, 2024

View reviewed changes

lucaspimentel added the status:do-not-merge Work is done. Can review, but do not merge yet. label Feb 15, 2024

lucaspimentel changed the title ~~Support remotely configuring DD_TRACE_ENABLED~~ [DO NOT MERGE] Support remotely configuring DD_TRACE_ENABLED Feb 15, 2024

andrewlock reviewed Feb 16, 2024

View reviewed changes

lucaspimentel changed the title ~~[DO NOT MERGE] Support remotely configuring DD_TRACE_ENABLED~~ [DO NOT MERGE] Support configuring DD_TRACE_ENABLED remotely Feb 16, 2024

lucaspimentel force-pushed the lpimentel/remote-disablement branch 3 times, most recently from ab01a73 to 0d57251 Compare February 28, 2024 21:43

lucaspimentel changed the title ~~[DO NOT MERGE] Support configuring DD_TRACE_ENABLED remotely~~ [Tracing] Support configuring DD_TRACE_ENABLED remotely Feb 28, 2024

lucaspimentel removed the status:do-not-merge Work is done. Can review, but do not merge yet. label Feb 28, 2024

lucaspimentel force-pushed the lpimentel/remote-disablement branch from f012934 to 4229d0b Compare February 29, 2024 14:35

lucaspimentel commented Feb 29, 2024

View reviewed changes

lucaspimentel force-pushed the lpimentel/remote-disablement branch from 294309d to 3d2843c Compare February 29, 2024 15:53

andrewlock reviewed Mar 5, 2024

View reviewed changes

lucaspimentel force-pushed the lpimentel/remote-disablement branch from 3d2843c to 8980775 Compare March 5, 2024 16:15

andrewlock approved these changes Mar 5, 2024

View reviewed changes

lucaspimentel requested a review from a team March 6, 2024 14:04

lucaspimentel force-pushed the lpimentel/remote-disablement branch 3 times, most recently from 68af04d to dc62b4f Compare March 7, 2024 20:27

lucaspimentel added 11 commits March 8, 2024 11:49

add support for remote TraceEnabled setting

0a2df8d

fix unrelated typo

2faa499

add ApmTracingEnabled capability 19

b5f0475

add missing RCM capability 18

b1d0a62

add mapping

6b83d7b

add unit test

87c0af7

add comments to unit test

4187337

rename property to match RCM constant

dbe6f37

include config in integration tests

3b24c35

fix test json

02cd6a3

rewrite tests to use raw values instead of strings

dd2c589

lucaspimentel force-pushed the lpimentel/remote-disablement branch from dc62b4f to dd2c589 Compare March 8, 2024 16:49

zacharycmontoya approved these changes Mar 11, 2024

View reviewed changes

lucaspimentel merged commit d5388d6 into master Mar 11, 2024

lucaspimentel deleted the lpimentel/remote-disablement branch March 11, 2024 19:20

github-actions bot added this to the vNext milestone Mar 11, 2024

lucaspimentel changed the title ~~[Tracing] Support configuring DD_TRACE_ENABLED remotely~~ [Tracing] Support remote config for DD_TRACE_ENABLED Apr 16, 2024


		public static readonly BigInteger AsmExclusionData = Create(18);

		public static readonly BigInteger ApmTracingTracingEnabled = Create(19);

Conversation

lucaspimentel commented Feb 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Uh oh!

datadog-ddstaging bot commented Feb 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

Uh oh!

andrewlock commented Feb 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

Uh oh!

andrewlock commented Feb 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Slower ⚠️ in #5181

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #5181

Fewer allocations 🎉 in #5181

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Uh oh!

andrewlock commented Feb 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Throughput/Crank Report:zap:

Uh oh!

Uh oh!

Uh oh!

zacharycmontoya left a comment

Choose a reason for hiding this comment

Uh oh!

andrewlock left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucaspimentel commented Feb 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lucaspimentel Feb 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewlock Mar 5, 2024

Choose a reason for hiding this comment

Uh oh!

andrewlock left a comment

Choose a reason for hiding this comment

Uh oh!

andrewlock Mar 5, 2024

Choose a reason for hiding this comment

Uh oh!

lucaspimentel commented Mar 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zacharycmontoya left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

lucaspimentel commented Feb 12, 2024 •

edited

Loading

datadog-ddstaging bot commented Feb 12, 2024 •

edited

Loading

andrewlock commented Feb 12, 2024 •

edited

Loading

andrewlock commented Feb 15, 2024 •

edited

Loading

andrewlock commented Feb 15, 2024 •

edited

Loading

andrewlock left a comment •

edited

Loading

lucaspimentel commented Feb 16, 2024 •

edited

Loading

lucaspimentel Feb 29, 2024 •

edited

Loading

lucaspimentel commented Mar 5, 2024 •

edited

Loading