In multithreaded (-mt) mode, multiple in-proc BuildRequestEngine instances run in the same process. Each engine's TraceEngine() method wrote to the same EngineTrace_{PID}.txt file using lock(this), which only serialized within that single instance. When two engines traced concurrently, the second FileStream open (with FileShare.Read) threw IOException, which propagated through QueueAction's error handler and fatally crashed the ActionBlock work queue. All subsequent Post() calls were silently rejected, causing request completions to be dropped and the scheduler to deadlock.

Root Cause

In -mt mode (same process, shared PID):

  Engine A: lock(this_A) -> opens EngineTrace_{PID}.txt  OK
  Engine B: lock(this_B) -> opens EngineTrace_{PID}.txt  IOException
                                                         |
                                            QueueAction catch -> throw
                                                         |
                                            ActionBlock faulted -> deadlock

lock(this) only serializes within ONE engine instance. Cross-instance contention on the shared file was unprotected.

Fix

• Replace lock(this) with a static lock (s_traceLock) so all engine instances in the same process serialize trace writes to a single file. This keeps one unified trace file (better debugging UX than per-node files).
• Catch non-critical exceptions in TraceEngine() using catch (Exception e) when (!ExceptionHandling.IsCriticalException(e)), matching the defensive pattern already used by Scheduler.TraceScheduler (Scheduler.cs:2683). Trace file failures must never crash the build engine.

Testing

Added regression test MultiThreadedBuild_WithDebugSchedulerTracing_DoesNotDeadlock:

• Enables MSBUILDDEBUGSCHEDULER=1 during a multithreaded build with 4 parallel child projects
• Uses [Fact(Timeout = 30_000)] so a deadlock regression fails fast instead of hanging CI
• Verified on both net10.0 and net472

Red-green validated locally: without the fix, the test aborts with IOException: The process cannot access the file 'EngineTrace_{PID}.txt' because it is being used by another process followed by InternalErrorException: Failure during engine shutdown. With the fix, the test passes in ~400ms.