[release/10.0] Backport Reliability fixes for the Thread Pool #122320
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Tagging subscribers to this area: @mangod9 |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
902b30e to
a4c985e
Compare
Member
Author
|
/backport to release/9.0-staging |
Contributor
|
Started backporting to |
Contributor
|
@VSadov backporting to git am output$ git am --3way --empty=keep --ignore-whitespace --keep-non-patch changes.patch
Applying: Reliability fixes for ThreadPool
Using index info to reconstruct a base tree...
M src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs
Falling back to patching base and 3-way merge...
Auto-merging src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs
CONFLICT (content): Merge conflict in src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 Reliability fixes for ThreadPool
Error: The process '/usr/bin/git' failed with exit code 128 |
Contributor
There was a problem hiding this comment.
Pull request overview
This backport fixes a subtle but critical reliability regression in the ThreadPool's enqueuer/worker handshake algorithm that could lead to deadlocks in rare scenarios. The fix reverts from a complex three-state machine (NotScheduled/Determining/Scheduled) to a simpler binary flag-based approach (_hasOutstandingThreadRequest) across three ThreadPool-like components.
Key changes:
- Replaces state machine with a simpler flag that guarantees at least one worker will check work queues after being set
- Ensures workers always request another worker when dequeuing items to prevent deadlock scenarios where work items depend on other queued work
- Applies the same pattern consistently to ThreadPoolWorkQueue, ThreadPoolTypedWorkItemQueue, and SocketAsyncEngine
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs | Reverts ThreadPoolWorkQueue and ThreadPoolTypedWorkItemQueue from complex state machine to simpler flag-based approach; simplifies Dispatch logic to unconditionally request workers when processing items |
| src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs | Applies same flag-based worker request pattern to SocketAsyncEngine event processing to maintain consistency with ThreadPool reliability fixes |
src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs
Outdated
Show resolved
Hide resolved
src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs
Outdated
Show resolved
Hide resolved
src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs
Outdated
Show resolved
Hide resolved
…adPoolWorkQueue.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
VSadov
commented
Dec 9, 2025
src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs
Outdated
Show resolved
Hide resolved
…adPoolWorkQueue.cs
agocke
reviewed
Dec 10, 2025
src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs
Outdated
Show resolved
Hide resolved
agocke
requested changes
Dec 10, 2025
Member
agocke
left a comment
There was a problem hiding this comment.
holding until we verify the correctness
This was referenced Dec 10, 2025
Member
|
/ba-g ios failure is unrelated |
Member
Author
|
Thanks! |
3 tasks
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes: #121608
Backport of #121887
The ThreadPool in rare cases allows a scenario when an enqueued workitem is not guaranteed to be executed unless more workitems are enqueued. In some scenarios execution of a particular workitem may be necessary before more work is enqueued, thus leading to a deadlock.
This is a subtle regression introduced by a change in enqueuer/worker handshake algorithm.
The same pattern is used in 2 other ThreadPool-like internal features in addition to the ThreadPool.
Customer Impact
Regression
Testing
Standard test pass for deterministic regressions.
A targeted stress application that demonstrates the issue.
(no fix: hangs within 1-2 minutes, with the fix: runs for 20+ min. on the same system)
Risk
Low. This is a revert to the preexisting algorithm for the enqueuer/worker handshake. (in all 3 places)