[Tracer] Increasing GraphQL ObfuscationQueryStringRegexTimeout To Prevent Flakes by link04 · Pull Request #5255 · DataDog/dd-trace-dotnet

link04 · 2024-02-29T22:12:15Z

Summary of changes

Updating when the ObfuscationQueryStringRegexTimeout setting should be change on the existing GraphQL tests and the value for it.

Also added an e in for the logged ObfuscationQueryStringRegexTimeout JSON key.

Reason for change

After the latest updates to the regex obfuscator we started getting multiple flaky GraphQL tests(for multiple runtimes).

datadog-ddstaging · 2024-02-29T22:27:21Z

Datadog Report

Branch report: maximo/flaky-graphql-querystring-increase-timeout
Commit report: dc874d1
Test service: dd-trace-dotnet

✅ 0 Failed, 339905 Passed, 1587 Skipped, 40m 35.31s Wall Time

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/GraphQLTests.cs

andrewlock · 2024-02-29T22:35:09Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5255) - mean (74ms)  : 65, 83
     .   : milestone, 74,
    master - mean (73ms)  : 65, 81
     .   : milestone, 73,

    section CallTarget+Inlining+NGEN
    This PR (5255) - mean (987ms)  : 961, 1012
     .   : milestone, 987,
    master - mean (987ms)  : 967, 1007
     .   : milestone, 987,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5255) - mean (111ms)  : 107, 115
     .   : milestone, 111,
    master - mean (110ms)  : 106, 114
     .   : milestone, 110,

    section CallTarget+Inlining+NGEN
    This PR (5255) - mean (716ms)  : 695, 738
     .   : milestone, 716,
    master - mean (713ms)  : 691, 735
     .   : milestone, 713,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5255) - mean (95ms)  : 91, 99
     .   : milestone, 95,
    master - mean (95ms)  : 91, 98
     .   : milestone, 95,

    section CallTarget+Inlining+NGEN
    This PR (5255) - mean (674ms)  : 646, 701
     .   : milestone, 674,
    master - mean (672ms)  : 651, 693
     .   : milestone, 672,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5255) - mean (188ms)  : 185, 190
     .   : milestone, 188,
    master - mean (187ms)  : 185, 190
     .   : milestone, 187,

    section CallTarget+Inlining+NGEN
    This PR (5255) - mean (1,061ms)  : 1033, 1089
     .   : milestone, 1061,
    master - mean (1,066ms)  : 1043, 1088
     .   : milestone, 1066,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5255) - mean (270ms)  : 265, 275
     .   : milestone, 270,
    master - mean (270ms)  : 267, 273
     .   : milestone, 270,

    section CallTarget+Inlining+NGEN
    This PR (5255) - mean (862ms)  : 837, 888
     .   : milestone, 862,
    master - mean (865ms)  : 833, 898
     .   : milestone, 865,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5255) - mean (260ms)  : 256, 264
     .   : milestone, 260,
    master - mean (259ms)  : 256, 263
     .   : milestone, 259,

    section CallTarget+Inlining+NGEN
    This PR (5255) - mean (854ms)  : 832, 877
     .   : milestone, 854,
    master - mean (850ms)  : 829, 870
     .   : milestone, 850,

zacharycmontoya

Thanks!

andrewlock · 2024-03-01T00:05:17Z

Benchmarks Report 🐌

Benchmarks for #5255 compared to master:

7 benchmarks are faster, with geometric mean 1.378
2 benchmarks are slower, with geometric mean 1.217
8 benchmarks have fewer allocations
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.69μs	49.3ns	338ns	0.0219	0.00874	0	7.49 KB
master	`StartStopWithChild`	netcoreapp3.1	10.6μs	57.9ns	332ns	0.0367	0.0157	0	7.59 KB
master	`StartStopWithChild`	net472	17.1μs	65.4ns	253ns	1.33	0.338	0.11	7.95 KB
#5255	`StartStopWithChild`	net6.0	8.79μs	48.7ns	323ns	0.0216	0.00863	0	7.49 KB
#5255	`StartStopWithChild`	netcoreapp3.1	10.6μs	57.3ns	329ns	0.02	0.00999	0	7.59 KB
#5255	`StartStopWithChild`	net472	17.1μs	55.1ns	214ns	1.33	0.346	0.101	7.96 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	457μs	196ns	732ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	624μs	332ns	1.29μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	803μs	236ns	914ns	0.401	3.3 KB
#5255	`WriteAndFlushEnrichedTraces`	net6.0	441μs	171ns	640ns	0	2.7 KB
#5255	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	607μs	74.8ns	259ns	0	2.7 KB
#5255	`WriteAndFlushEnrichedTraces`	net472	797μs	598ns	2.31μs	0.398	3.3 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5255

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net472	1.422	66,382.20	46,688.46

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	41.6μs	76ns	294ns	0.0206	0	2.36 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	43.5μs	87.3ns	302ns	0.0218	0	2.34 KB
master	`AllCycleSimpleBody`	net472	66.4μs	15.5ns	59.9ns	0.366	0	2.41 KB
master	`AllCycleMoreComplexBody`	net6.0	216μs	240ns	930ns	0.109	0	9.84 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	227μs	208ns	807ns	0.113	0	9.73 KB
master	`AllCycleMoreComplexBody`	net472	241μs	122ns	472ns	1.56	0	9.91 KB
master	`ObjectExtractorSimpleBody`	net6.0	139ns	0.0984ns	0.368ns	0.00391	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	208ns	0.104ns	0.403ns	0.00377	0	272 B
master	`ObjectExtractorSimpleBody`	net472	171ns	0.0727ns	0.281ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	3.02μs	1.31ns	5.09ns	0.0528	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.03μs	2.02ns	7.82ns	0.0504	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	3.82μs	2.49ns	8.99ns	0.602	0.00573	3.8 KB
#5255	`AllCycleSimpleBody`	net6.0	40.8μs	24.6ns	95.3ns	0.0205	0	2.36 KB
#5255	`AllCycleSimpleBody`	netcoreapp3.1	44.1μs	17.5ns	67.8ns	0.0221	0	2.34 KB
#5255	`AllCycleSimpleBody`	net472	46.7μs	10.9ns	40.7ns	0.38	0	2.41 KB
#5255	`AllCycleMoreComplexBody`	net6.0	214μs	63.8ns	239ns	0.107	0	9.84 KB
#5255	`AllCycleMoreComplexBody`	netcoreapp3.1	228μs	101ns	393ns	0.114	0	9.73 KB
#5255	`AllCycleMoreComplexBody`	net472	239μs	60.4ns	234ns	1.55	0	9.91 KB
#5255	`ObjectExtractorSimpleBody`	net6.0	143ns	0.17ns	0.636ns	0.00396	0	280 B
#5255	`ObjectExtractorSimpleBody`	netcoreapp3.1	204ns	0.216ns	0.837ns	0.00375	0	272 B
#5255	`ObjectExtractorSimpleBody`	net472	173ns	0.0932ns	0.361ns	0.0446	0	281 B
#5255	`ObjectExtractorMoreComplexBody`	net6.0	2.98μs	1.1ns	4.25ns	0.0535	0	3.78 KB
#5255	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.07μs	1.55ns	5.79ns	0.0509	0	3.69 KB
#5255	`ObjectExtractorMoreComplexBody`	net472	4.01μs	1.62ns	6.07ns	0.602	0.00571	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Faster 🎉 Fewer allocations 🎉

Faster 🎉 in #5255

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs‑net6.0	1.578	50,778.73	32,187.92
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs‑netcoreapp3.1	1.516	68,649.93	45,274.13
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs‑net472	1.425	84,737.49	59,478.26
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs‑net6.0	1.326	128,119.13	96,647.43
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs‑netcoreapp3.1	1.230	155,396.38	126,304.99
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs‑net472	1.195	217,433.91	181,917.48

Fewer allocations 🎉 in #5255

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs‑net6.0	30.94 KB	27.94 KB	-3 KB	-9.69%
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs‑net6.0	33.89 KB	30.34 KB	-3.55 KB	-10.47%
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs‑netcoreapp3.1	34.09 KB	30.34 KB	-3.74 KB	-10.98%
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs‑netcoreapp3.1	31.47 KB	27.94 KB	-3.53 KB	-11.20%
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs‑net472	34.99 KB	30.44 KB	-4.55 KB	-13.00%
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs‑net472	32.27 KB	28.03 KB	-4.23 KB	-13.12%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EncodeArgs`	net6.0	50.9μs	51.7ns	200ns	0.426	0	30.94 KB
master	`EncodeArgs`	netcoreapp3.1	68.7μs	86.4ns	323ns	0.407	0	31.47 KB
master	`EncodeArgs`	net472	84.8μs	67.7ns	262ns	5.12	0.0846	32.27 KB
master	`EncodeLegacyArgs`	net6.0	128μs	146ns	547ns	0.443	0	33.89 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	155μs	374ns	1.45μs	0.468	0	34.09 KB
master	`EncodeLegacyArgs`	net472	217μs	571ns	2.21μs	5.46	0.42	34.99 KB
#5255	`EncodeArgs`	net6.0	32.2μs	19.8ns	76.8ns	0.387	0	27.94 KB
#5255	`EncodeArgs`	netcoreapp3.1	45.3μs	31.5ns	118ns	0.382	0	27.94 KB
#5255	`EncodeArgs`	net472	59.5μs	76.6ns	297ns	4.44	0.0592	28.03 KB
#5255	`EncodeLegacyArgs`	net6.0	97.2μs	447ns	1.61μs	0.4	0.05	30.34 KB
#5255	`EncodeLegacyArgs`	netcoreapp3.1	126μs	618ns	2.47μs	0.386	0	30.34 KB
#5255	`EncodeLegacyArgs`	net472	182μs	188ns	727ns	4.79	0.361	30.44 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunWafRealisticBenchmark`	net6.0	188μs	46.1ns	172ns	0.0943	6.51 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	204μs	184ns	714ns	0	6.49 KB
master	`RunWafRealisticBenchmark`	net472	224μs	125ns	485ns	1	6.59 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	126μs	78.4ns	293ns	0	4.15 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	134μs	148ns	573ns	0	4.14 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	147μs	50.8ns	190ns	0.657	4.19 KB
#5255	`RunWafRealisticBenchmark`	net6.0	188μs	200ns	773ns	0.095	6.51 KB
#5255	`RunWafRealisticBenchmark`	netcoreapp3.1	207μs	282ns	1.09μs	0	6.49 KB
#5255	`RunWafRealisticBenchmark`	net472	224μs	146ns	566ns	1.01	6.59 KB
#5255	`RunWafRealisticBenchmarkWithAttack`	net6.0	125μs	44.5ns	172ns	0.063	4.15 KB
#5255	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	134μs	304ns	1.18μs	0	4.14 KB
#5255	`RunWafRealisticBenchmarkWithAttack`	net472	147μs	66.2ns	248ns	0.661	4.19 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	172μs	143ns	554ns	0.171	18.26 KB
master	`SendRequest`	netcoreapp3.1	191μs	198ns	769ns	0.191	20.42 KB
master	`SendRequest`	net472	0.000896ns	0.00028ns	0.00101ns	0	0 b
#5255	`SendRequest`	net6.0	172μs	153ns	594ns	0.171	18.26 KB
#5255	`SendRequest`	netcoreapp3.1	195μs	227ns	879ns	0.193	20.42 KB
#5255	`SendRequest`	net472	0.032ns	0.00696ns	0.027ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5255

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	41.59 KB	41.93 KB	343 B	0.82%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	549μs	2.1μs	8.12μs	0.563	0	0	41.47 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	653μs	514ns	1.92μs	0.329	0	0	41.59 KB
master	`WriteAndFlushEnrichedTraces`	net472	837μs	3.96μs	16.8μs	8.25	2.6	0.434	53.21 KB
#5255	`WriteAndFlushEnrichedTraces`	net6.0	544μs	1.22μs	4.73μs	0.548	0	0	41.32 KB
#5255	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	661μs	1.19μs	4.61μs	0.326	0	0	41.93 KB
#5255	`WriteAndFlushEnrichedTraces`	net472	860μs	3.51μs	13.6μs	8.08	2.55	0.425	53.25 KB

Benchmarks.Trace.DbCommandBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5255

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery‑net6.0	1.310	1,018.63	1,333.94

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.02μs	0.816ns	3.16ns	0.0107	776 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.42μs	0.572ns	2.06ns	0.0106	776 B
master	`ExecuteNonQuery`	net472	1.79μs	0.713ns	2.76ns	0.117	738 B
#5255	`ExecuteNonQuery`	net6.0	1.33μs	0.642ns	2.4ns	0.0107	776 B
#5255	`ExecuteNonQuery`	netcoreapp3.1	1.51μs	0.583ns	2.26ns	0.0106	776 B
#5255	`ExecuteNonQuery`	net472	1.68μs	4.41ns	17.1ns	0.117	738 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.31μs	0.685ns	2.56ns	0.0135	944 B
master	`CallElasticsearch`	netcoreapp3.1	1.52μs	1.07ns	3.99ns	0.0128	944 B
master	`CallElasticsearch`	net472	2.5μs	0.715ns	2.68ns	0.153	963 B
master	`CallElasticsearchAsync`	net6.0	1.39μs	0.628ns	2.43ns	0.0132	920 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.65μs	0.514ns	1.92ns	0.0132	992 B
master	`CallElasticsearchAsync`	net472	2.56μs	0.691ns	2.68ns	0.161	1.02 KB
#5255	`CallElasticsearch`	net6.0	1.23μs	1.43ns	5.34ns	0.0135	944 B
#5255	`CallElasticsearch`	netcoreapp3.1	1.52μs	0.56ns	1.94ns	0.0122	944 B
#5255	`CallElasticsearch`	net472	2.39μs	1.29ns	5.01ns	0.153	963 B
#5255	`CallElasticsearchAsync`	net6.0	1.29μs	0.725ns	2.71ns	0.0125	920 B
#5255	`CallElasticsearchAsync`	netcoreapp3.1	1.55μs	0.549ns	2.05ns	0.0134	992 B
#5255	`CallElasticsearchAsync`	net472	2.55μs	1.09ns	4.21ns	0.161	1.02 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.42μs	3.01ns	10.9ns	0.0128	920 B
master	`ExecuteAsync`	netcoreapp3.1	1.69μs	0.868ns	3.36ns	0.0127	920 B
master	`ExecuteAsync`	net472	1.84μs	0.648ns	2.51ns	0.14	883 B
#5255	`ExecuteAsync`	net6.0	1.29μs	1.52ns	5.9ns	0.0129	920 B
#5255	`ExecuteAsync`	netcoreapp3.1	1.61μs	0.685ns	2.56ns	0.012	920 B
#5255	`ExecuteAsync`	net472	1.81μs	1.22ns	4.73ns	0.14	883 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.14μs	2.11ns	7.61ns	0.0289	2.1 KB
master	`SendAsync`	netcoreapp3.1	4.99μs	3.29ns	12.8ns	0.0349	2.64 KB
master	`SendAsync`	net472	7.58μs	2.82ns	10.9ns	0.525	3.31 KB
#5255	`SendAsync`	net6.0	4.13μs	1.95ns	7.28ns	0.0289	2.1 KB
#5255	`SendAsync`	netcoreapp3.1	5μs	2.99ns	11.6ns	0.035	2.64 KB
#5255	`SendAsync`	net472	7.66μs	5.84ns	22.6ns	0.524	3.31 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #5255

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472	61.06 KB	57.9 KB	-3.17 KB	-5.19%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	216.77 KB	202.11 KB	-14.66 KB	-6.76%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	58.8μs	795ns	7.95μs	43.44 KB
master	`StringConcatBenchmark`	netcoreapp3.1	52.6μs	240ns	930ns	42.64 KB
master	`StringConcatBenchmark`	net472	37.3μs	120ns	416ns	61.06 KB
master	`StringConcatAspectBenchmark`	net6.0	264μs	1.25μs	7.05μs	203.54 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	306μs	1.49μs	11.2μs	216.77 KB
master	`StringConcatAspectBenchmark`	net472	227μs	1.19μs	6.16μs	229.38 KB
#5255	`StringConcatBenchmark`	net6.0	50.9μs	257ns	1.18μs	43.44 KB
#5255	`StringConcatBenchmark`	netcoreapp3.1	51.4μs	233ns	839ns	42.64 KB
#5255	`StringConcatBenchmark`	net472	37.6μs	95.8ns	345ns	57.9 KB
#5255	`StringConcatAspectBenchmark`	net6.0	244μs	3.14μs	30μs	202.91 KB
#5255	`StringConcatAspectBenchmark`	netcoreapp3.1	277μs	1.42μs	6.52μs	202.11 KB
#5255	`StringConcatAspectBenchmark`	net472	231μs	1.23μs	7.98μs	229.38 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.66μs	0.352ns	1.27ns	0.022	1.58 KB
master	`EnrichedLog`	netcoreapp3.1	2.18μs	0.728ns	2.72ns	0.0209	1.58 KB
master	`EnrichedLog`	net472	2.72μs	1.17ns	4.36ns	0.238	1.51 KB
#5255	`EnrichedLog`	net6.0	1.5μs	1.2ns	4.66ns	0.0222	1.58 KB
#5255	`EnrichedLog`	netcoreapp3.1	2.16μs	0.882ns	3.3ns	0.0205	1.58 KB
#5255	`EnrichedLog`	net472	2.52μs	2.42ns	9.36ns	0.239	1.51 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	114μs	227ns	879ns	0.0566	0	4.22 KB
master	`EnrichedLog`	netcoreapp3.1	118μs	89.8ns	336ns	0.0589	0	4.22 KB
master	`EnrichedLog`	net472	148μs	85.6ns	331ns	0.663	0.221	4.4 KB
#5255	`EnrichedLog`	net6.0	115μs	188ns	705ns	0.0579	0	4.22 KB
#5255	`EnrichedLog`	netcoreapp3.1	118μs	135ns	524ns	0	0	4.22 KB
#5255	`EnrichedLog`	net472	148μs	82.9ns	310ns	0.666	0.222	4.4 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.13μs	0.909ns	3.52ns	0.0298	2.14 KB
master	`EnrichedLog`	netcoreapp3.1	4.12μs	2.11ns	8.18ns	0.0289	2.14 KB
master	`EnrichedLog`	net472	4.88μs	1.39ns	5.19ns	0.31	1.95 KB
#5255	`EnrichedLog`	net6.0	2.9μs	1.87ns	7.23ns	0.0305	2.14 KB
#5255	`EnrichedLog`	netcoreapp3.1	4.24μs	2.08ns	8.05ns	0.0272	2.14 KB
#5255	`EnrichedLog`	net472	4.78μs	2.58ns	9.28ns	0.309	1.95 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.34μs	0.73ns	2.83ns	0.0153	1.11 KB
master	`SendReceive`	netcoreapp3.1	1.69μs	0.787ns	2.84ns	0.0151	1.11 KB
master	`SendReceive`	net472	2.16μs	4.56ns	17ns	0.178	1.12 KB
#5255	`SendReceive`	net6.0	1.25μs	0.732ns	2.84ns	0.0156	1.11 KB
#5255	`SendReceive`	netcoreapp3.1	1.66μs	0.789ns	3.05ns	0.0149	1.11 KB
#5255	`SendReceive`	net472	2μs	2.03ns	7.33ns	0.178	1.12 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.9μs	1.24ns	4.82ns	0.0216	1.54 KB
master	`EnrichedLog`	netcoreapp3.1	4.07μs	1.03ns	3.85ns	0.0205	1.58 KB
master	`EnrichedLog`	net472	4.2μs	1.26ns	4.9ns	0.313	1.97 KB
#5255	`EnrichedLog`	net6.0	2.63μs	0.669ns	2.59ns	0.021	1.54 KB
#5255	`EnrichedLog`	netcoreapp3.1	3.9μs	1.37ns	5.3ns	0.0216	1.58 KB
#5255	`EnrichedLog`	net472	4.34μs	1.68ns	6.3ns	0.313	1.97 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5255

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1	1.132	652.93	738.97

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	463ns	0.748ns	2.9ns	0.00771	544 B
master	`StartFinishSpan`	netcoreapp3.1	652ns	0.896ns	3.23ns	0.00744	544 B
master	`StartFinishSpan`	net472	782ns	2.01ns	7.77ns	0.0865	546 B
master	`StartFinishScope`	net6.0	584ns	0.937ns	3.63ns	0.00937	664 B
master	`StartFinishScope`	netcoreapp3.1	853ns	1.35ns	5.23ns	0.00919	664 B
master	`StartFinishScope`	net472	928ns	1.6ns	6.21ns	0.099	626 B
#5255	`StartFinishSpan`	net6.0	449ns	0.841ns	3.26ns	0.00748	544 B
#5255	`StartFinishSpan`	netcoreapp3.1	739ns	0.985ns	3.82ns	0.00739	544 B
#5255	`StartFinishSpan`	net472	740ns	0.814ns	3.15ns	0.0866	546 B
#5255	`StartFinishScope`	net6.0	630ns	1.13ns	4.36ns	0.00923	664 B
#5255	`StartFinishScope`	netcoreapp3.1	940ns	0.756ns	2.83ns	0.00877	664 B
#5255	`StartFinishScope`	net472	993ns	1.38ns	5.33ns	0.0994	626 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	702ns	1.27ns	4.9ns	0.0091	664 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.02μs	1.77ns	6.84ns	0.00904	664 B
master	`RunOnMethodBegin`	net472	1.11μs	1.8ns	6.98ns	0.0992	626 B
#5255	`RunOnMethodBegin`	net6.0	631ns	0.857ns	3.32ns	0.00948	664 B
#5255	`RunOnMethodBegin`	netcoreapp3.1	954ns	2.12ns	8.21ns	0.00897	664 B
#5255	`RunOnMethodBegin`	net472	1.14μs	1.72ns	6.43ns	0.0993	626 B

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/GraphQLTests.cs

andrewlock · 2024-03-01T05:01:47Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5255) (11.057M)   : 0, 11057062
    master (11.049M)   : 0, 11048964
    benchmarks/2.9.0 (11.124M)   : 0, 11124125

    section Automatic
    This PR (5255) (7.558M)   : 0, 7558361
    master (7.647M)   : 0, 7647044
    benchmarks/2.9.0 (8.268M)   : 0, 8268275

    section Trace stats
    This PR (5255) (7.892M)   : 0, 7891947
    master (7.930M)   : 0, 7929875

    section Manual
    This PR (5255) (9.506M)   : 0, 9506264
    master (9.692M)   : 0, 9691903

    section Manual + Automatic
    This PR (5255) (7.116M)   : 0, 7115928
    master (7.121M)   : 0, 7120766

    section Version Conflict
    This PR (5255) (6.374M)   : 0, 6374074
    master (6.404M)   : 0, 6403827

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5255) (9.489M)   : 0, 9489180
    master (9.351M)   : 0, 9351384
    benchmarks/2.9.0 (9.557M)   : 0, 9556831

    section Automatic
    This PR (5255) (6.688M)   : 0, 6688286
    master (6.456M)   : 0, 6455727

    section Trace stats
    This PR (5255) (6.780M)   : 0, 6779995
    master (6.580M)   : 0, 6580411

    section Manual
    This PR (5255) (8.250M)   : 0, 8249531
    master (8.037M)   : 0, 8037032

    section Manual + Automatic
    This PR (5255) (6.340M)   : 0, 6340133
    master (6.270M)   : 0, 6269626

    section Version Conflict
    This PR (5255) (5.703M)   : 0, 5703392
    master (5.595M)   : 0, 5595041

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5255) (9.514M)   : 0, 9514092
    master (10.406M)   : 0, 10406304
    benchmarks/2.9.0 (9.910M)   : 0, 9909593

    section Automatic
    This PR (5255) (7.010M)   : 0, 7010114
    master (7.296M)   : 0, 7295789
    benchmarks/2.9.0 (7.464M)   : 0, 7464259

    section Trace stats
    This PR (5255) (7.309M)   : 0, 7309364
    master (7.634M)   : 0, 7634119

    section Manual
    This PR (5255) (8.602M)   : 0, 8601941
    master (9.026M)   : 0, 9026110

    section Manual + Automatic
    This PR (5255) (6.699M)   : crit ,0, 6699475
    master (7.113M)   : 0, 7113148

    section Version Conflict
    This PR (5255) (6.072M)   : 0, 6072269
    master (6.379M)   : 0, 6378587

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.518M)   : 0, 7517670
    benchmarks/2.9.0 (7.908M)   : 0, 7908237

    section No attack
    master (1.831M)   : 0, 1831221
    benchmarks/2.9.0 (3.212M)   : 0, 3211779

    section Attack
    master (1.442M)   : 0, 1442009
    benchmarks/2.9.0 (2.469M)   : 0, 2469017

    section Blocking
    master (3.159M)   : 0, 3158780

    section IAST default
    master (6.504M)   : 0, 6503728

    section IAST full
    master (5.620M)   : 0, 5619566

    section Base vuln
    master (0.900M)   : 0, 899927

    section IAST vuln
    master (0.875M)   : 0, 875067

andrewlock

Thanks - I wonder if we should set this somewhere "centrally" so we don't introduce flake if we use querystrings in the future?

andrewlock · 2024-03-01T09:14:36Z

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/GraphQLTests.cs

-#endif
+
+            // After updating the regex implementation multiple runtimes started hitting the a timeout, updating here to prevent flakes on master
+            SetEnvironmentVariable(Configuration.ConfigurationKeys.ObfuscationQueryStringRegexTimeout, "5_000");


I wonder if we should set this for all test, not just GraphQL. ASM tests in particular probably use querystrings, as do some aspnetcore test IIRC. But this is a good start!

…Tests.cs Co-authored-by: Lucas Pimentel <lucas.pimentel@datadoghq.com>

link04 added 2 commits February 29, 2024 16:51

Changing small log typo

50cce85

Changing timeout to wider scope and range

04ae014

link04 requested a review from a team as a code owner February 29, 2024 22:12

bouwkast reviewed Feb 29, 2024

View reviewed changes

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/GraphQLTests.cs Outdated Show resolved Hide resolved

bouwkast approved these changes Feb 29, 2024

View reviewed changes

Removing redundant if and typo in example log

5535bd6

zacharycmontoya approved these changes Feb 29, 2024

View reviewed changes

lucaspimentel reviewed Mar 1, 2024

View reviewed changes

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/GraphQLTests.cs Outdated Show resolved Hide resolved

lucaspimentel reviewed Mar 1, 2024

View reviewed changes

tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/GraphQLTests.cs Outdated Show resolved Hide resolved

andrewlock approved these changes Mar 1, 2024

View reviewed changes

link04 and others added 3 commits March 1, 2024 10:09

Updating value so is taken into account

2fc97ea

Update tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/GraphQL…

d95f212

…Tests.cs Co-authored-by: Lucas Pimentel <lucas.pimentel@datadoghq.com>

Setting ObfuscationQueryStringRegexTimeout for HotChocolateTests

dc874d1

link04 merged commit 160ca3b into master Mar 3, 2024

link04 deleted the maximo/flaky-graphql-querystring-increase-timeout branch March 3, 2024 03:29

github-actions bot added this to the vNext milestone Mar 3, 2024

link04 mentioned this pull request Mar 3, 2024

[Tracer] Fixing Missing Query String for CleanUri_HttpUrlTag Tests #5258

Merged

kevingosse mentioned this pull request Mar 5, 2024

Warm up the query string obfuscator regex #5266

Merged

Conversation

link04 commented Feb 29, 2024

Summary of changes

Reason for change

Uh oh!

datadog-ddstaging bot commented Feb 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Datadog Report

Uh oh!

Uh oh!

andrewlock commented Feb 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

Uh oh!

zacharycmontoya left a comment

Choose a reason for hiding this comment

Uh oh!

andrewlock commented Mar 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Faster 🎉 in #5255

Raw results

Faster 🎉 in #5255

Fewer allocations 🎉 in #5255

Raw results

Raw results

Raw results

More allocations ⚠️ in #5255

Raw results

Slower ⚠️ in #5255

Raw results

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #5255

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5255

Raw results

Raw results

Uh oh!

Uh oh!

Uh oh!

andrewlock commented Mar 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Throughput/Crank Report:zap:

Uh oh!

andrewlock left a comment

Choose a reason for hiding this comment

Uh oh!

andrewlock Mar 1, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

datadog-ddstaging bot commented Feb 29, 2024 •

edited

Loading

andrewlock commented Feb 29, 2024 •

edited

Loading

andrewlock commented Mar 1, 2024 •

edited

Loading

andrewlock commented Mar 1, 2024 •

edited

Loading