Skip to content

Limit Parallel.ForEach concurrency in SourcelinkTests#5373

Merged
steveisok merged 1 commit intomainfrom
fix-sourcelink-parallelism
Mar 11, 2026
Merged

Limit Parallel.ForEach concurrency in SourcelinkTests#5373
steveisok merged 1 commit intomainfrom
fix-sourcelink-parallelism

Conversation

@steveisok
Copy link
Copy Markdown
Member

Fix

The VerifySourcelinks test launches hundreds of dotnet-sourcelink processes simultaneously via Parallel.ForEach with no concurrency limit. Under resource contention, some processes exceed the 60-second timeout in ExecuteHelper.ExecuteProcess, get killed, and report false failures — the tool writes "validated" to stdout but gets killed before it can exit cleanly with code 0.

Root Cause Analysis (from #5259 (comment))

  1. Parallel.ForEach launches dotnet-sourcelink test --offline for every PDB — potentially hundreds simultaneously
  2. Each process has a 60-second timeout (millisecondTimeout: 60000)
  3. Under contention, some take >60s to fully exit
  4. The tool validates the file and writes "validated" to stdout, but gets killed during shutdown/cleanup
  5. The killed process has a non-zero exit code → added to failedFiles
  6. Different PDBs fail each time because it depends on which processes run during peak contention

Change

Add new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount } to bound concurrent process launches to the number of CPU cores.

Fixes #5259

The VerifySourcelinks test launches hundreds of dotnet-sourcelink processes
simultaneously with no concurrency limit. Under resource contention, some
processes exceed the 60-second timeout in ExecuteHelper, get killed, and
report false failures — the tool writes 'validated' to stdout but gets
killed before exiting cleanly with exit code 0.

Bound concurrency to Environment.ProcessorCount to prevent resource
contention that causes timeout-driven false failures.

Fixes #5259

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@steveisok steveisok requested review from a team as code owners March 10, 2026 21:32
Copilot AI review requested due to automatic review settings March 10, 2026 21:32
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces flakiness in SourcelinkTests.VerifySourcelinks by bounding the number of concurrent dotnet-sourcelink process launches, preventing excessive resource contention that can trigger the existing 60-second per-process timeout and cause false failures.

Changes:

  • Add ParallelOptions with MaxDegreeOfParallelism = Environment.ProcessorCount to the Parallel.ForEach used for PDB validation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

@steveisok
Copy link
Copy Markdown
Member Author

Failure is due to #5391

@steveisok steveisok merged commit b3b656c into main Mar 11, 2026
15 checks passed
@steveisok steveisok deleted the fix-sourcelink-parallelism branch March 11, 2026 19:48
@akoeplinger
Copy link
Copy Markdown
Member

akoeplinger commented Mar 12, 2026

Under resource contention, some processes exceed the 60-second timeout in ExecuteHelper.ExecuteProcess, get killed, and report false failures — the tool writes "validated" to stdout but gets killed before it can exit cleanly with code 0.

I don't think this is necessarily true, we should've still got a message in the log about the process being killed due to timeout: #5259 (comment)

but it's possible this still helps or fixes the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

VerifySourcelinks fails, passes on rerun

5 participants