Skip to content

Collecting some build data for tasks/targets telemetry#11359

Merged
JanKrivanek merged 28 commits intomainfrom
proto/task-telemetry-data-plan-b
Feb 20, 2025
Merged

Collecting some build data for tasks/targets telemetry#11359
JanKrivanek merged 28 commits intomainfrom
proto/task-telemetry-data-plan-b

Conversation

@JanKrivanek
Copy link
Copy Markdown
Member

@JanKrivanek JanKrivanek commented Jan 29, 2025

Fixes #10946

If split into separate PRs is prefered - please indicate so in comments

Goal

Obtaining information about build composition from microsoft versus 3rd party tasks and targets.

Approach

Since the information is present in the building worker nodes - the code collects it there and then (if requested) send it via WorkerNodeTelemetryEventArgs to the main node.

The classification of 3rd party versus 1st party is for simplicity being done based on location of defining msbuild project and naming of the assembly.

Changes

  • Added class for the exexution statistics of tasks - this is contained in TaskRegistry as well as in TaskFactoryWrappers
  • RequestBuilder is the orchestration here, that decides whether statistics are needed and if yes - traverses the TaskRegistry, BuildResult and ProjectCollection in order to accumulate and populate the statistics
  • Added WorkerNodeTelemetryEventArgs that holds data, InternalTelemeteryConsumingLogger and InternalTelemeteryForwardingLogger to transfer the data (only if telemetry is sampled in)

Performance considerations

By default the data collection logic is off and hence statistics are not collected on worker node, nor serialized to the event args. The perf impact of collection and serialization was though beyond the recognition level of basic 'full duration' testing of full and incremental build of small console and bigger size projects.

Sample of collected data

This is stringified flushed version of stats collected and aggregated from nodes.
Once inputing into our telemetry client, we might send just subset (topN).
We collect the info about whether the task/target is 1st or 3rd party ('C:' prefix), whether it's comming from nuget cache ('N:' prefix) or is generated within metaproj, the task/target name (will be hashed for 3rd party), the duration of the task, the memory consumption of the task and the number of executions of the task

==========================================
Tasks: (258)
Custom tasks:
C:ReplaceFileText
==========================================
Tasks by time:
Microsoft.Build.Tasks.ResolveAssemblyReference - 00:00:34.5173355
Microsoft.CodeAnalysis.BuildTasks.CopyRefAssembly - 00:00:20.0824767
Microsoft.Build.Tasks.WriteLinesToFile - 00:00:08.7755037
Microsoft.NET.Build.Tasks.ResolvePackageAssets - 00:00:07.6207085
Microsoft.CodeAnalysis.BuildTasks.Csc - 00:00:06.6906938
(...)
==========================================
Tasks by memory consumption:
Microsoft.Build.Tasks.ResolveAssemblyReference - 10751256.03 kB
Microsoft.Build.Tasks.AssignProjectConfiguration - 1189559.04 kB
Microsoft.Build.Tasks.AssignTargetPath - 393482.46 kB
Microsoft.Build.Tasks.SetRidAgnosticValueForProjects - 294210.73 kB
(...)
N:Microsoft.Build.Tasks.Git.GetUntrackedFiles - 3543.50 kB
(...)
==========================================
Tasks by Executions count:
GetPackageDirectory - 4140
Microsoft.Build.Tasks.AssignTargetPath - 3050
Microsoft.Build.Tasks.FindUnderPath - 2126
(...)
N:Microsoft.SourceLink.GitHub.GetSourceLinkUrl - 1242
Microsoft.Build.Tasks.RemoveDuplicates - 1086
(...)
=========================================
Targets (1621) - name : executed:
AfterSdkPublish : False
C:MakeWebRootFolder : False
(...)

Performance impact considerations

Somethings always costs more than nothing :-) Beyond that - those changes should be with hardly observable impact.

If telemetry is opted out - there is no extra data collection, processing and sending applied.
When telemetry is opted in - we keep and update hashtables with stats for all tasks and targets in all nodes and at the end of build send those stats to the main node. This minimizes costly traffic during the build.

When measured with 'wall clock time' on incremental build of OrchardCore via:

pskill msbuild; pskill dotnet; pskill vbcscompiler
Measure-Command { & "C:\src\msbuild-2\artifacts\bin\bootstrap\net472\MSBuild\Current\Bin\MSBuild.exe" -v:m | Out-Default }

And excluding warmup and slowest and fastest runs - the versions seems indistinguishable:

main (@aff5455):

Minutes : 2
Seconds : 34
Milliseconds : 352

Minutes : 2
Seconds : 30
Milliseconds : 665

Minutes : 2
Seconds : 30
Milliseconds : 109

This PR (with _isTelemetryEnabled forced to true and $env:MSBUILDOUTPUTNODESTELEMETRY=1):

Minutes : 3
Seconds : 36
Milliseconds : 707

Minutes : 2
Seconds : 37
Milliseconds : 763

Minutes : 2
Seconds : 35
Milliseconds : 794

@JanProvaznik JanProvaznik requested a review from Copilot January 30, 2025 11:36
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 19 out of 34 changed files in this pull request and generated 5 comments.

Files not reviewed (15)
  • src/Build/Microsoft.Build.csproj: Language not supported
  • src/Build.UnitTests/BackEnd/NodePackets_Tests.cs: Evaluated as low risk
  • src/Build/BackEnd/Components/BuildComponentFactoryCollection.cs: Evaluated as low risk
  • src/Build/Definition/Toolset.cs: Evaluated as low risk
  • src/Build/BackEnd/Components/RequestBuilder/TaskBuilder.cs: Evaluated as low risk
  • src/Build/BackEnd/Components/Logging/CentralForwardingLogger.cs: Evaluated as low risk
  • src/Build/BackEnd/Components/RequestBuilder/TargetBuilder.cs: Evaluated as low risk
  • src/Build/BackEnd/Components/IBuildComponentHost.cs: Evaluated as low risk
  • src/Framework.UnitTests/FileClassifier_Tests.cs: Evaluated as low risk
  • src/Build/BackEnd/BuildManager/BuildParameters.cs: Evaluated as low risk
  • src/Build/BackEnd/Components/BuildRequestEngine/BuildRequestEngine.cs: Evaluated as low risk
  • src/Build/BackEnd/Shared/TargetResult.cs: Evaluated as low risk
  • src/Build/BackEnd/Components/Logging/EventSourceSink.cs: Evaluated as low risk
  • src/Build/Instance/TaskFactoryWrapper.cs: Evaluated as low risk
  • src/Build/BackEnd/BuildManager/BuildManager.cs: Evaluated as low risk
Comments suppressed due to low confidence (2)

src/Build/BackEnd/Components/RequestBuilder/RequestBuilder.cs:1263

  • The new telemetry collection logic in 'UpdateStatisticsPostBuild' is not covered by tests. Add test cases to ensure this logic is tested.
private void UpdateStatisticsPostBuild()

src/Build/Instance/TaskRegistry.cs:1182

  • The 'Stats' class should be instantiated with 'new Stats()' instead of 'new Stats'.
internal class Stats()

@dotnet dotnet deleted a comment from Copilot AI Jan 30, 2025
@surayya-MS surayya-MS self-requested a review February 4, 2025 09:20
Copy link
Copy Markdown
Member

@surayya-MS surayya-MS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the first round of review it looks good! I'll look more into this

@JanKrivanek JanKrivanek marked this pull request as ready for review February 5, 2025 10:00
Copy link
Copy Markdown
Member

@surayya-MS surayya-MS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

Copy link
Copy Markdown
Member

@JanProvaznik JanProvaznik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a few suggestions if you agree with the naming. All in all: :shipit:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Define + implement initial metric to collect

4 participants