[Test Optimization] ci: add run id + config keys#8071
Merged
tonyredondo merged 2 commits intomasterfrom Feb 3, 2026
Merged
Conversation
This was referenced Jan 15, 2026
BenchmarksBenchmark execution time: 2026-02-02 15:36:28 Comparing candidate commit f0e013a in PR branch Found 6 performance improvements and 5 performance regressions! Performance is the same for 165 metrics, 16 unstable metrics. scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net6.0
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net6.0
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net6.0
scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1
scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1
scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net6.0
scenario:Benchmarks.Trace.NLogBenchmark.EnrichedLog netcoreapp3.1
scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1
scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin netcoreapp3.1
|
This comment has been minimized.
This comment has been minimized.
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8071) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8071) - mean (69ms) : 67, 70
master - mean (69ms) : 67, 70
section Bailout
This PR (8071) - mean (73ms) : 71, 74
master - mean (72ms) : 71, 74
section CallTarget+Inlining+NGEN
This PR (8071) - mean (1,014ms) : 958, 1069
master - mean (1,018ms) : 933, 1104
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8071) - mean (106ms) : 103, 110
master - mean (106ms) : 103, 109
section Bailout
This PR (8071) - mean (107ms) : 106, 108
master - mean (107ms) : 105, 108
section CallTarget+Inlining+NGEN
This PR (8071) - mean (746ms) : 690, 802
master - mean (748ms) : 703, 794
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8071) - mean (94ms) : 92, 96
master - mean (94ms) : 92, 96
section Bailout
This PR (8071) - mean (95ms) : 94, 96
master - mean (94ms) : 93, 95
section CallTarget+Inlining+NGEN
This PR (8071) - mean (726ms) : 691, 761
master - mean (716ms) : 680, 752
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8071) - mean (93ms) : 90, 95
master - mean (92ms) : 89, 95
section Bailout
This PR (8071) - mean (94ms) : 92, 95
master - mean (93ms) : 92, 94
section CallTarget+Inlining+NGEN
This PR (8071) - mean (637ms) : 624, 650
master - mean (635ms) : 622, 648
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8071) - mean (197ms) : 189, 204
master - mean (194ms) : 190, 197
section Bailout
This PR (8071) - mean (202ms) : 197, 206
master - mean (197ms) : 195, 200
section CallTarget+Inlining+NGEN
This PR (8071) - mean (1,169ms) : 1091, 1247
master - mean (1,144ms) : 1061, 1227
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8071) - mean (283ms) : 275, 291
master - mean (278ms) : 272, 283
section Bailout
This PR (8071) - mean (281ms) : 274, 289
master - mean (280ms) : 271, 288
section CallTarget+Inlining+NGEN
This PR (8071) - mean (945ms) : 907, 983
master - mean (936ms) : 901, 972
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8071) - mean (276ms) : 268, 284
master - mean (270ms) : 265, 275
section Bailout
This PR (8071) - mean (274ms) : 267, 281
master - mean (271ms) : 266, 276
section CallTarget+Inlining+NGEN
This PR (8071) - mean (943ms) : 890, 997
master - mean (924ms) : 879, 969
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8071) - mean (272ms) : 266, 279
master - mean (270ms) : 265, 275
section Bailout
This PR (8071) - mean (275ms) : 265, 285
master - mean (270ms) : 266, 275
section CallTarget+Inlining+NGEN
This PR (8071) - mean (841ms) : 824, 857
master - mean (836ms) : 815, 857
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ad6c32d to
f0ccdd2
Compare
a92777c to
d0595bf
Compare
7a2fadb to
c58aa40
Compare
d0595bf to
7216d32
Compare
tonyredondo
commented
Jan 16, 2026
Comment on lines
+469
to
+498
| try | ||
| { | ||
| var cacheFolder = Path.Combine(baseDirectory ?? CIValues.WorkspacePath ?? Environment.CurrentDirectory, ".dd", runId); | ||
| Log.Debug("TestOptimization: Creating cache folder: {Folder}", cacheFolder); | ||
| if (!Directory.Exists(cacheFolder)) | ||
| { | ||
| Directory.CreateDirectory(cacheFolder); | ||
| } | ||
|
|
||
| LifetimeManager.Instance.AddShutdownTask(_ => | ||
| { | ||
| try | ||
| { | ||
| Log.Debug("TestOptimization: Removing cache folder: {Folder}", cacheFolder); | ||
| if (Directory.Exists(cacheFolder)) | ||
| { | ||
| Directory.Delete(cacheFolder, true); | ||
| } | ||
| } | ||
| catch (Exception exInner) | ||
| { | ||
| Log.Warning(exInner, "TestOptimization: Error deleting cache folder."); | ||
| } | ||
| }); | ||
| } | ||
| catch (Exception ex) | ||
| { | ||
| Log.Warning(ex, "TestOptimization: Error creating cache folder."); | ||
| } | ||
| } |
Member
Author
There was a problem hiding this comment.
If we are the one creating the RunId for children processes then we are the owners of the temporary folder for the cache of the run.
tonyredondo
commented
Jan 16, 2026
Comment on lines
+72
to
+73
| protected static bool IsMacOS() | ||
| => string.Equals(FrameworkDescription.Instance.OSPlatform, OSPlatformName.MacOS, StringComparison.OrdinalIgnoreCase); |
Member
Author
There was a problem hiding this comment.
when running on Rider in the mac, all tests deadlock because the UDPClient.Close call when we have statsd. So to be able to run and debug the tests, we are identifying the macOs platform and use this boolean when creating the mock agent.
This doesn't affect actual test because we don't do integration tests on mac.
Contributor
There was a problem hiding this comment.
maybe this could be added as comment where IsMacOS is used
tonyredondo
commented
Jan 16, 2026
Comment on lines
+420
to
+421
| runId = Guid.NewGuid().ToString("n"); | ||
| SetEnvironmentVariable(ConfigurationKeys.CIVisibility.TestOptimizationRunId, runId); |
Member
Author
There was a problem hiding this comment.
We need to create the RunId manually for each test iteration to avoid using the current one created by the TestOptimizationClient. because the RunId is being used to create a cache folder we need to invalidate that cache in each run.
tonyredondo
commented
Jan 16, 2026
| using var logsIntake = new MockLogsIntakeForCiVisibility(); | ||
| EnableDirectLogSubmission(logsIntake.Port, nameof(IntegrationId.XUnit), nameof(XUnitTests)); | ||
| using var agent = EnvironmentHelper.GetMockAgent(useStatsD: true); | ||
| using var agent = EnvironmentHelper.GetMockAgent(useStatsD: !IsMacOS()); |
Member
Author
There was a problem hiding this comment.
and here is the IsMacOS() usage
tonyredondo
commented
Jan 16, 2026
Comment on lines
+435
to
+439
| var thread = new Thread(DoWork) | ||
| { | ||
| IsBackground = true, | ||
| Name = "ConcurrentRunner:" + i | ||
| }; |
Member
Author
There was a problem hiding this comment.
Just adding a name to those ConcurrentRunner threads to improve debugging in the IDE.
anmarchenko
approved these changes
Jan 16, 2026
4a70c7a to
b3116fd
Compare
4a649b1 to
3f3f143
Compare
b3116fd to
643a4d8
Compare
643a4d8 to
36cf4ce
Compare
tonyredondo
added a commit
that referenced
this pull request
Jan 30, 2026
## Summary of changes - Add FixedSizeArrayPool and logger overloads to reduce per-log allocations. - Add RefStopwatch/StopwatchHelpers/CodeDuration and apply timing to TraceClock/TracerManagerFactory/Instrumentation and CI processors. - Lower init logs to Debug for CI protocol writer/sender and processors. ## Reason for change Reduce allocations and improve timing diagnostics in hot paths. ## Implementation details - New utility helpers in `Datadog.Trace.Util` for allocation-free timing and pooling. - Logging changes to avoid params array allocations in common log paths. - Timing/log-level tweaks in CI protocol writer/sender and trace processors. ## Test coverage CI passes then all changes are good. ## Other details - Stacked PRs (current marked): - **CURRENT**: PR1 #8070 - PR2 #8071 - PR3 #8072 - PR4 #8073
Base automatically changed from
tony/testoptimization-performance-work-pr1
to
master
January 30, 2026 15:33
36cf4ce to
1e2d8dd
Compare
anna-git
reviewed
Feb 2, 2026
Comment on lines
+72
to
+73
| protected static bool IsMacOS() | ||
| => string.Equals(FrameworkDescription.Instance.OSPlatform, OSPlatformName.MacOS, StringComparison.OrdinalIgnoreCase); |
Contributor
There was a problem hiding this comment.
maybe this could be added as comment where IsMacOS is used
tracer/test/Datadog.Trace.Tests/Telemetry/ConfigurationTests.cs
Outdated
Show resolved
Hide resolved
tracer/test/Datadog.Trace.Tests/Telemetry/ConfigurationTests.cs
Outdated
Show resolved
Hide resolved
anna-git
approved these changes
Feb 2, 2026
tonyredondo
added a commit
that referenced
this pull request
Feb 3, 2026
## Summary of changes - Add git command caching and telemetry improvements for CI visibility. - Update GitInfo discovery and CI environment logging. - Update impacted tests for new git discovery behavior. JIRA: SDTEST-3226 ## Reason for change Reduce git command overhead and improve visibility into CI git metadata collection. ## Implementation details - Disk cache keyed by RunId in GitCommandHelper with safe.directory handling. - JSON annotations in ProcessHelpers for cache serialization. - Adjusted CI runner git discovery logic and tests. ## Test coverage CI passes then all changes are good. ## Other details - Stacked PRs (current marked): - PR1 #8070 - PR2 #8071 - **CURRENT**: PR3 #8072 - PR4 #8073
tonyredondo
added a commit
that referenced
this pull request
Feb 3, 2026
## Summary of changes - Add cached/file Test Optimization clients and update client wrappers. - Reduce feature init overhead and adjust background initialization. - Improve runner logging and CI workspace cache usage. ## Reason for change Reduce repeated HTTP calls and improve initialization performance for CI visibility features. ## Implementation details - New cached/file client implementations for disk/memory caching. - Feature creation updated to reduce dependencies on client calls. - CodeDuration instrumentation added to CI initialization paths. ## Test coverage CI passes then all changes are good. ## Other details - Stacked PRs (current marked): - PR1 #8070 - PR2 #8071 - PR3 #8072 - **CURRENT**: PR4 #8073
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
_DD_INTERNAL_TOPT_RUNIDto config mapping/docs and regenerate keys.Reason for change
Provide a stable RunId before caching work that depends on it and keep configuration in sync.
Implementation details
ITestOptimization/TestOptimizationwith environment propagation.Test coverage
CI passes then all changes are good.
Other details