Skip to content

Eliminate PR build bottleneck from standalone documentation check job#13084

Merged
rainersigwald merged 3 commits intomainfrom
copilot/optimize-test-results-execution
Jan 23, 2026
Merged

Eliminate PR build bottleneck from standalone documentation check job#13084
rainersigwald merged 3 commits intomainfrom
copilot/optimize-test-results-execution

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Jan 21, 2026

Context

The IfOnlyDocumentionChanged job ran as a standalone dependency for all test jobs. When this single job queued waiting for an agent (often 10+ minutes), it blocked all downstream test execution despite agents being available for the actual tests.

Changes Made

  • Removed standalone job: Deleted IfOnlyDocumentionChanged job and all dependsOn references
  • Created reusable template: azure-pipelines/check-documentation-only-change.yml encapsulates the git diff logic
  • Inlined check: Each test job (Windows Full/Core/Release, Linux, macOS) now runs the template as first step (~1-2 sec overhead)
  • Updated CodeCoverage job: References output variable from BootstrapMSBuildOnFullFrameworkWindows instead of deleted job
  • Fixed YAML syntax: Removed empty variables: declarations from CoreBootstrappedOnLinux and CoreOnMac jobs that were left after removing variable references

Net result: -6 lines, eliminated scheduling dependency, same functional behavior.

Testing

  • YAML syntax validated and passes after syntax error fixes
  • Variable references verified across all 5 test jobs
  • Template outputs both local and cross-job variables for downstream consumption

Notes

The check is now redundantly executed 5 times (once per job) instead of once. This is intentional - the ~5-10 seconds of redundant computation is negligible compared to eliminating agent queue blocking. Each job can now start immediately when an agent is available rather than waiting for a dedicated check job to complete.

Original prompt

This section details on the original issue you should resolve

<issue_title>"Check whether Test Results need to be executed" can stall a PR build for a long time</issue_title>
<issue_description>For an example, see https://dev.azure.com/dnceng-public/public/_build/results?buildId=1261235

Our PR builds run this step to try to figure out if it's worth spending time running tests

https://github.com/dotnet/msbuild/blob/fe494f4420703fce289a0b1ea27cf477771e49b6/.vsts-dotnet-ci.yml#L62C1-L80C1

However, because this is an independent job, it can get stalled waiting for a machine, and because all of the actual test jobs depend on it, that can delay PR builds from completion for a very long time.

We should consider moving that computation inside the individual OS/scenario test jobs. That's redundant since it only needs to run once, but I think it would speed up overall execution time since the script is small/fast and it'd avoid this stall.</issue_description>

<agent_instructions>Make sure the result is clean and maintainable--don't just copy/paste the script into each job.</agent_instructions>

Comments on the Issue (you are @copilot in this section)


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@dotnet-policy-service
Copy link
Copy Markdown
Contributor

Hello @@copilot, I noticed that you’re changing an .swr file or any file under src/Package/MSBuild.VSSetup.. Please make sure to validate this change by an experimental VS insertion. This is accomplished by pushing to an exp/* branch, which requires write permissions to this repo.

- Create reusable template for documentation check (check-documentation-only-change.yml)
- Remove standalone IfOnlyDocumentionChanged job that was causing PR build delays
- Integrate documentation check inline at start of each test job
- Remove blocking dependencies on IfOnlyDocumentionChanged job from all test jobs
- CodeCoverage job now gets onlyDocChanged variable from one of the test jobs

Co-authored-by: rainersigwald <3347530+rainersigwald@users.noreply.github.com>
Copilot AI changed the title [WIP] Optimize execution of test results check in PR builds Eliminate PR build bottleneck from standalone documentation check job Jan 21, 2026
Copilot AI requested a review from rainersigwald January 21, 2026 16:17
@rainersigwald
Copy link
Copy Markdown
Member

@copilot

/.vsts-dotnet-ci.yml (Line: 339, Col: 13): Unexpected value ''

Removed empty 'variables:' lines from CoreBootstrappedOnLinux and CoreOnMac jobs that were causing YAML parsing errors.

Co-authored-by: rainersigwald <3347530+rainersigwald@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Jan 21, 2026

@copilot

...

Fixed in 0c2f224. Removed the empty variables: declarations from the CoreBootstrappedOnLinux and CoreOnMac jobs that were left over when removing the variable references.

@rainersigwald rainersigwald marked this pull request as ready for review January 22, 2026 16:05
@rainersigwald rainersigwald requested a review from a team as a code owner January 22, 2026 16:05
Copilot AI review requested due to automatic review settings January 22, 2026 16:05
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR eliminates a critical bottleneck in the CI pipeline by removing a standalone documentation check job that was blocking test execution while waiting for agent availability.

Changes:

  • Removed standalone IfOnlyDocumentionChanged job that could delay PR builds by 10+ minutes
  • Created reusable template azure-pipelines/check-documentation-only-change.yml that encapsulates the git diff logic
  • Inlined the documentation check as the first step in all 5 test jobs (Windows Full/Core/Release, Linux, macOS) with ~1-2 second overhead each
  • Updated CodeCoverage job to reference the output variable from BootstrapMSBuildOnFullFrameworkWindows instead of the deleted standalone job
  • Removed empty variables: sections from CoreBootstrappedOnLinux and CoreOnMac jobs after removing variable references

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
azure-pipelines/check-documentation-only-change.yml New reusable template containing PowerShell script to check if only documentation files changed, sets both local and cross-job output variables
.vsts-dotnet-ci.yml Removed standalone IfOnlyDocumentionChanged job, added template inclusion to all 5 test jobs, updated CodeCoverage job dependencies and variable reference, cleaned up empty variable sections

@rainersigwald rainersigwald merged commit 60cd5bd into main Jan 23, 2026
15 checks passed
@rainersigwald rainersigwald deleted the copilot/optimize-test-results-execution branch January 23, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"Check whether Test Results need to be executed" can stall a PR build for a long time

4 participants