Add a pipeline for MSBuild -mt validation#4487
Conversation
d46bc5c to
b0dfbea
Compare
|
|
|
@JanProvaznik every repo builds with the LKG msbuild in stage1 builds. Only source-build's stage2 builds with the live msbuild. But if we get the msbuild fix merged into main, we can just easily do another re-bootstrap in the VMR.
Clarification: We want to merge this PR but have the job not run by default. Only via the VMR pipeline that we will add to msbuild and sdk. |
9c566bc to
9ccc03e
Compare
|
FWIW we usually don't put business logic (the filtering) in YML so that the local build behaves the same as the YML build. This is important as our source-build partners don't use our YML to build the VMR. Not important here as source-build partners likely don't want to build the VMR with the -mt mode but I could see myself wanting to build the VMR with it. |
|
@ViktorHofer is there a possible issue using the same config as the sourcebuild jobs? I know we want to avoid windows due to the pool being small, but I'm not sure if I'm overusing some other resources with sourcebuild config. (Extracting to separate pipeline with separate trigger is pending once I get this working with the 2 stage build). |
d344b7e to
aa45bb6
Compare
1990835 to
030c764
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds infrastructure to validate MSBuild's new multithreaded (-mt) mode by creating a manually-triggered pipeline that syncs PRs from constituent repositories into the VMR and performs a 2-stage source build with -mt mode enabled. The pipeline allows the MSBuild team to test their multithreaded feature against the full .NET ecosystem build.
Changes:
- Added a manually-triggered Azure Pipeline for validating MSBuild -mt mode with PR syncing
- Modified Arcade build scripts (tools.sh and tools.ps1) to conditionally enable -mt flag via environment variable
- Added MSBuild -mt mode configuration and exclusion management in repo-projects/Directory.Build.props
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| eng/pipelines/msbuild-mt-2stage-validation.yml | New pipeline that syncs a PR into VMR and runs 2-stage builds with optional -mt mode testing |
| repo-projects/Directory.Build.props | Adds configuration for -mt mode exclusions and environment variable handling to selectively enable the feature per repository |
| src/arcade/eng/common/tools.sh | Conditionally adds -mt flag to MSBuild commands when MSBUILD_MT_ENABLED=1 environment variable is set |
| src/arcade/eng/common/tools.ps1 | PowerShell equivalent of the bash changes for Windows builds |
| eng/common/tools.sh | Same changes as src/arcade version to maintain consistency across both locations |
| eng/common/tools.ps1 | Same changes as src/arcade version to maintain consistency across both locations |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
e12bdc6 to
de70987
Compare
|
@dotnet/source-build can you please review the YML changes around |
mthalman
left a comment
There was a problem hiding this comment.
Is there a test run available that we can see how this pipeline works?
Is this pipeline intended to be temporary until mt validation is completed?
Points to VMR branch msbuild-mt-validation-ci which has extraPropertiesStage2 support. Once the VMR PR dotnet/dotnet#4487 is merged, update ref to main.
1e1192f to
36c65c5
Compare
Points to VMR branch msbuild-mt-validation-ci which has extraPropertiesStage2 support. Once the VMR PR dotnet/dotnet#4487 is merged, update ref to main.
Points to VMR branch msbuild-mt-validation-ci which has extraPropertiesStage2 support. Once the VMR PR dotnet/dotnet#4487 is merged, update ref to main.
36c65c5 to
f4dad99
Compare
Points to VMR branch msbuild-mt-validation-ci which has extraPropertiesStage2 support. Once the VMR PR dotnet/dotnet#4487 is merged, update ref to main.
Points to VMR branch msbuild-mt-validation-ci which has extraPropertiesStage2 support. Once the VMR PR dotnet/dotnet#4487 is merged, update ref to main.
This adds infrastructure to validate MSBuild's multithreaded (-mt) mode: 1. New on-demand pipeline (eng/pipelines/mt-validation.yml): - Syncs a PR from a constituent repo into the VMR - Runs 2-stage source-build with -mt mode enabled - Configurable exclusion list for repos not compatible with -mt 2. Build script support (eng/common/tools.ps1, tools.sh): - Adds -mt flag when MSBUILD_MT_ENABLED=1 env var is set 3. VMR configuration (repo-projects/Directory.Build.props): - Default exclusion list for repos built before MSBuild fix is available - Environment variable override for custom exclusions - Sets MSBUILD_MT_ENABLED=1 when DotNetBuildMT=true 4. Arcade opt-out (repo-projects/arcade.proj): - Arcade builds before msbuild is bootstrapped, so must use LKG Resolves #4451
…repo - Replace custom 227-line pipeline with runtime-style VMR template extension - Add extraPropertiesStage2 parameter to vmr-build.yml → source-build-stages.yml → source-build-and-validate.yml - -mt mode only applies to stage 2 (source-built MSBuild), not stage 1 (LKG) - Add MSBuildMTExcludedReposOverride property support in Directory.Build.props - Use %3B encoding for semicolons to avoid MSBuild command-line parsing issues - Create src/msbuild/azure-pipelines/vmr-mt-validation.yml (42 lines)
c01c45d to
7f51244
Compare
#4451
What
Adds infrastructure for validating MSBuild's multithreaded (-mt) mode by building the VMR with it.
Approach
Following the same pattern as runtime, the pipeline lives in the msbuild repo (
src/msbuild/azure-pipelines/vmr-mt-validation.yml) and extends the existing VMR build templates (vmr-build.yml).The key design choice is that -mt mode is only applied to stage 2 (which builds with the source-built MSBuild from stage 1). Stage 1 uses the LKG SDK whose MSBuild has known -mt bugs.
Changes
VMR template changes
vmr-build.yml: AddextraPropertiesStage2parameter (threaded to source-build-stages → source-build-and-validate)source-build-stages.yml: ThreadextraPropertiesStage2tosource-build-and-validate.ymlsource-build-and-validate.yml: ApplyextraPropertiesStage2only to legs withreuseBuildArtifactsFrom(i.e., stage 2 legs)Exclusion logic
repo-projects/Directory.Build.props: AddMSBuildMTExcludedReposOverrideMSBuild property override (in addition to existingMSBUILD_MT_EXCLUDED_REPOSenv var). Uses%3Bencoding on the command line since MSBuild treats semicolons as property separators.arcade;runtime;roslyn;aspnetcore;sdk;efcore;winforms;wpf;razor;source-build-reference-packagesMSBuild repo pipeline (new)
src/msbuild/azure-pipelines/vmr-mt-validation.yml(~42 lines): Manually triggered pipeline with configurableexcludedReposparameter. Uses the samevmr-build.yml@vmrpattern as runtime.Removed
eng/pipelines/msbuild-mt-2stage-validation.yml: Replaced by the simpler msbuild-repo pipeline above (was 227 lines of custom YAML duplicating VMR template logic).Pipeline parameters
excludedRepos: Repos to exclude from -mt mode (%3B-separated). Configurable per-run to test MSBuild fixes against previously-broken repos.Usage
dotnet/msbuildrepo, fileazure-pipelines/vmr-mt-validation.ymlexcludedReposto test fixesOnce MSBuild team is confident in -mt correctness and performance, this mode would become the default for regular CI.