Fix race condition in LoggingService.ProcessLoggingEvent after shutdown#13450
Merged
YuliiaKovalova merged 2 commits intomainfrom Mar 27, 2026
Merged
Fix race condition in LoggingService.ProcessLoggingEvent after shutdown#13450YuliiaKovalova merged 2 commits intomainfrom
YuliiaKovalova merged 2 commits intomainfrom
Conversation
ProcessLoggingEvent accessed _eventQueue, _dequeueEvent, and _enqueueEvent directly without checking if the service had been shut down. When an external callback (e.g., Process.Exited from a ProjectCachePlugin) attempted to log after ShutdownComponent() had nullified these fields, a NullReferenceException (access violation 0xC0000005) would occur. The fix: 1. Early return when _serviceState == LoggingServiceState.Shutdown, matching the guard pattern already used by RegisterLogger, RegisterDistributedLogger, InitializeNodeLoggers, and InitializeComponent. 2. Capture field references to local variables before use, matching the existing pattern in LoggingEventProc (the queue pump thread). 3. Null-check the captured locals as a second defense against the race between the _serviceState check and field reads during concurrent shutdown.
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a race where LoggingService.ProcessLoggingEvent could be invoked after shutdown (e.g., by external callbacks holding references) and crash due to shutdown-cleaned resources.
Changes:
- Add an early-return guard in
ProcessLoggingEventwhen the service is already shut down, and use local copies of async-queue fields to reduce null races. - Add unit tests covering late logging after shutdown (sync/async) and concurrent shutdown vs. logging calls.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/Build/BackEnd/Components/Logging/LoggingService.cs | Adds shutdown guard + local captures in async path to mitigate post-shutdown races. |
| src/Build.UnitTests/BackEnd/LoggingService_Tests.cs | Adds regression tests for logging-after-shutdown and a concurrent shutdown/logging scenario. |
…tion, Shouldly consistency - Wrap wait-handle operations (WaitOne/Set) in try/catch for ObjectDisposedException, since shutdown can dispose handles after they are captured into locals but before they are used. - Assert logThread.Join result to prevent silent test hangs. - Switch Assert.Equal to Shouldly ShouldBe for consistency with the rest of the test file.
dfederm
approved these changes
Mar 26, 2026
JanProvaznik
approved these changes
Mar 27, 2026
SimaTian
pushed a commit
that referenced
this pull request
Apr 7, 2026
…wn (#13450) ## Summary Fix a race condition crash (access violation 0xC0000005 / NullReferenceException) in `LoggingService.ProcessLoggingEvent` when the method is called after the logging service has been shut down. ## Problem `ProcessLoggingEvent` directly accesses `_eventQueue`, `_dequeueEvent`, and `_enqueueEvent` without any shutdown guard. During `ShutdownComponent()`, `CleanLoggingEventProcessing()` disposes and nullifies all of these fields. If an external callback (e.g., `Process.Exited` from a `ProjectCachePlugin`) attempts to log a message after shutdown has completed, the null field access causes an unrecoverable crash. ### Reproduction Scenario 1. A build runs with a `ProjectCachePlugin` 2. The build completes and `LoggingService.ShutdownComponent()` is called 3. The cache plugin's external process exits, triggering `Process.OnExited` on a ThreadPool thread 4. The callback logs "Project cache service process exited" via `LoggingServiceToPluginLoggerAdapter.LogMessage` 5. This calls `ProcessLoggingEvent()` which accesses `_eventQueue.Count` → **null reference crash** ### Debugger Evidence ``` _serviceState = Shutdown _eventQueue = null _dequeueEvent = null _enqueueEvent = null _emptyQueueEvent = null _loggingEventProcessingCancellation = null buildEvent.Message = "Project cache service process exited" Call: System.Diagnostics.Process.OnExited -> ProjectCachePlugin.dll -> LoggingServiceToPluginLoggerAdapter.LogMessage ``` ## Root Cause `ProcessLoggingEvent` lacked two protections that already exist elsewhere in the codebase: 1. **Shutdown state guard** - Other public methods (`RegisterLogger`, `RegisterDistributedLogger`, `InitializeNodeLoggers`, `InitializeComponent`) all check `_serviceState != LoggingServiceState.Shutdown`. `ProcessLoggingEvent` did not. 2. **Local field capture** - The `LoggingEventProc` queue pump thread already captures `_eventQueue`, `_dequeueEvent`, `_emptyQueueEvent`, and `_enqueueEvent` into local variables to prevent races with `CleanLoggingEventProcessing()`. `ProcessLoggingEvent` accessed the fields directly. ## Fix Three layers of defense, all following patterns already established in the same file: 1. **Early return** when `_serviceState == LoggingServiceState.Shutdown` - silently drops the event, since the service is no longer operational. 2. **Capture fields to locals** before use - prevents null dereference if `CleanLoggingEventProcessing()` runs concurrently between the state check and the field reads. 3. **Null-check the captured locals** - second defense against the TOCTOU race between step 1 and step 2. ## Risk **Low.** The fix only adds early-exit guards - no behavioral change for the normal (non-shutdown) code path. The patterns used are already established in the same file.
dfederm
pushed a commit
to dfederm/msbuild
that referenced
this pull request
Apr 9, 2026
…wn (dotnet#13450) ## Summary Fix a race condition crash (access violation 0xC0000005 / NullReferenceException) in `LoggingService.ProcessLoggingEvent` when the method is called after the logging service has been shut down. ## Problem `ProcessLoggingEvent` directly accesses `_eventQueue`, `_dequeueEvent`, and `_enqueueEvent` without any shutdown guard. During `ShutdownComponent()`, `CleanLoggingEventProcessing()` disposes and nullifies all of these fields. If an external callback (e.g., `Process.Exited` from a `ProjectCachePlugin`) attempts to log a message after shutdown has completed, the null field access causes an unrecoverable crash. ### Reproduction Scenario 1. A build runs with a `ProjectCachePlugin` 2. The build completes and `LoggingService.ShutdownComponent()` is called 3. The cache plugin's external process exits, triggering `Process.OnExited` on a ThreadPool thread 4. The callback logs "Project cache service process exited" via `LoggingServiceToPluginLoggerAdapter.LogMessage` 5. This calls `ProcessLoggingEvent()` which accesses `_eventQueue.Count` → **null reference crash** ### Debugger Evidence ``` _serviceState = Shutdown _eventQueue = null _dequeueEvent = null _enqueueEvent = null _emptyQueueEvent = null _loggingEventProcessingCancellation = null buildEvent.Message = "Project cache service process exited" Call: System.Diagnostics.Process.OnExited -> ProjectCachePlugin.dll -> LoggingServiceToPluginLoggerAdapter.LogMessage ``` ## Root Cause `ProcessLoggingEvent` lacked two protections that already exist elsewhere in the codebase: 1. **Shutdown state guard** - Other public methods (`RegisterLogger`, `RegisterDistributedLogger`, `InitializeNodeLoggers`, `InitializeComponent`) all check `_serviceState != LoggingServiceState.Shutdown`. `ProcessLoggingEvent` did not. 2. **Local field capture** - The `LoggingEventProc` queue pump thread already captures `_eventQueue`, `_dequeueEvent`, `_emptyQueueEvent`, and `_enqueueEvent` into local variables to prevent races with `CleanLoggingEventProcessing()`. `ProcessLoggingEvent` accessed the fields directly. ## Fix Three layers of defense, all following patterns already established in the same file: 1. **Early return** when `_serviceState == LoggingServiceState.Shutdown` - silently drops the event, since the service is no longer operational. 2. **Capture fields to locals** before use - prevents null dereference if `CleanLoggingEventProcessing()` runs concurrently between the state check and the field reads. 3. **Null-check the captured locals** - second defense against the TOCTOU race between step 1 and step 2. ## Risk **Low.** The fix only adds early-exit guards - no behavioral change for the normal (non-shutdown) code path. The patterns used are already established in the same file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix a race condition crash (access violation 0xC0000005 / NullReferenceException) in
LoggingService.ProcessLoggingEventwhen the method is called after the logging service has been shut down.Problem
ProcessLoggingEventdirectly accesses_eventQueue,_dequeueEvent, and_enqueueEventwithout any shutdown guard. DuringShutdownComponent(),CleanLoggingEventProcessing()disposes and nullifies all of these fields. If an external callback (e.g.,Process.Exitedfrom aProjectCachePlugin) attempts to log a message after shutdown has completed, the null field access causes an unrecoverable crash.Reproduction Scenario
ProjectCachePluginLoggingService.ShutdownComponent()is calledProcess.OnExitedon a ThreadPool threadLoggingServiceToPluginLoggerAdapter.LogMessageProcessLoggingEvent()which accesses_eventQueue.Count→ null reference crashDebugger Evidence
Root Cause
ProcessLoggingEventlacked two protections that already exist elsewhere in the codebase:Shutdown state guard - Other public methods (
RegisterLogger,RegisterDistributedLogger,InitializeNodeLoggers,InitializeComponent) all check_serviceState != LoggingServiceState.Shutdown.ProcessLoggingEventdid not.Local field capture - The
LoggingEventProcqueue pump thread already captures_eventQueue,_dequeueEvent,_emptyQueueEvent, and_enqueueEventinto local variables to prevent races withCleanLoggingEventProcessing().ProcessLoggingEventaccessed the fields directly.Fix
Three layers of defense, all following patterns already established in the same file:
_serviceState == LoggingServiceState.Shutdown- silently drops the event, since the service is no longer operational.CleanLoggingEventProcessing()runs concurrently between the state check and the field reads.Risk
Low. The fix only adds early-exit guards - no behavioral change for the normal (non-shutdown) code path. The patterns used are already established in the same file.