It's something in MSBuild. Trying to get a dump -- tricky when it's consuming all my memory.

Yeah, something has definitely changed with MSBuild memory use.

❯ MiB Mem :  32019.2 total,    194.4 free,  31137.1 used,   2706.5 buff/cache
  MiB Swap:   8192.0 total,     22.5 free,   8169.5 used.    882.0 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  48454 andy      20   0   71.1g   3.8g  40560 S  22.9  12.1   0:36.90 /home/andy/code/vmr/.dotnet/dotnet /home/andy/cod+
  48462 andy      20   0   71.2g   3.8g  41788 S  26.8  12.1   0:39.18 /home/andy/code/vmr/.dotnet/dotnet /home/andy/cod+
  48461 andy      20   0   71.0g   3.5g  51364 S  36.3  11.1   0:39.31 /home/andy/code/vmr/.dotnet/dotnet /home/andy/cod+
  48459 andy      20   0   70.7g   3.3g  38344 S  25.2  10.4   0:37.61 /home/andy/code/vmr/.dotnet/dotnet /home/andy/cod+
  49523 andy      20   0   69.7g   3.1g  38588 S  23.5  10.1   0:32.14 /home/andy/code/vmr/.dotnet/dotnet /home/andy/cod+
  48456 andy      20   0   69.9g   2.8g  55572 S  44.1   9.0   0:38.30 /home/andy/code/vmr/.dotnet/dotnet /home/andy/cod+
  47687 andy      20   0   73.1g   2.8g  54964 S  44.8   8.9   2:19.79 /home/andy/code/vmr/.dotnet/dotnet msbuild /m /no+
  48359 andy      20   0   71.4g   2.4g  61544 S  22.5   7.6   1:59.89 /home/andy/code/vmr/.dotnet/dotnet msbuild /m /no+
  52306 andy      20   0  267.3g   1.0g 145388 S 520.3   3.3   4:03.11 /home/andy/code/vmr/.dotnet/sdk/11.0.100-preview.+
  52085 andy      20   0  262.7g 330696  70172 R  37.9   1.0   0:29.19 /home/andy/code/vmr/.d

@rainersigwald could you take a look? Those are the MSBuild worker processes.

Nothing is jumping out at me in dotnet/msbuild@6b7a28c...81815e5 as likely to cause a problem like that. Will try to get a local repro.

Behavior appears different on Windows vs Linux; my Windows build is failing but not crashing but my Linux build crashed the entire WSL VM?

Looks like it's not a single process going bananas but an accumulation of them--my initial WSL attempt crashed during runtime restore (as best as I can tell from logs) but continuing from that point got through runtime and then crashed while building sourcelink + efcore.

Could/should we try reverting MSBuild to the prior state to see if it's in MSBuild vs. runtime?

FWIW vbcscompiler seemed relatively similar to where it was before, so that's why I hypothesized MSBuild, not CLR.

Could/should we try reverting MSBuild to the prior state to see if it's in MSBuild vs. runtime?

Tried to simulate this on my machine by manually patching the MSBuild bits back to the LKG:

cp ~/src/lkg_sdk/sdk/11.0.100-preview.1.26104.118/{MSBuild,Microsoft.Build{,.{Framework,{Tasks,Utilities}.Core}}}.dll .dotnet/sdk/11.0.100-preview.2.26117.112/

And see very similar crash symptoms (during runtime restore). Since I keep seeing it during restore I'm going to try patching NuGet back next.

I'm going to try patching NuGet back next.

Still crashes with NuGet built at the commit from the old SDK.

I think that leaves SDK and runtime as the likeliest suspects?

Nope, spoke too soon, runtime failed with pinned-back NuGet but it didn't crash.

Duplicate project System.Threading [/home/raines/src/dotnet/src/runtime/Build.proj]
   at NuGet.ProjectModel.PackageSpecReferenceDependencyProvider..ctor(IEnumerable`1 externalProjects, IEnvironment
   at NuGet.ProjectModel.PackageSpecReferenceDependencyProvider..ctor(IEnumerable`1 externalProjects, ILogger logg
   at NuGet.Commands.RestoreCommand.ExecuteRestoreAsync(NuGetv3LocalRepository userPackageFolder, IReadOnlyList`1 

^ and that's because I built debug, I think. Trying with release nuget.

With just NuGet@1f8b2d8c release overlaid on the SDK I still get a crash, this one seems to be when restoring Roslyn and runtime (and doing various other things).

❯ rg '\] Building' C:\Users\raines\Downloads\overall.log
42:  [19:36:41.49] Building arcade
61:  [19:39:05.70] Building arcade...done
107:  [19:39:06.64] Building razor
133:  [19:39:06.66] Building winforms
159:  [19:39:06.76] Building templating
175:  [19:39:06.77] Building command-line-api
211:  [19:39:06.77] Building cecil
237:  [19:39:06.79] Building symreader
263:  [19:39:06.80] Building deployment-tools
289:  [19:39:06.81] Building nuget-client
315:  [19:39:06.83] Building emsdk
341:  [19:39:06.84] Building msbuild
367:  [19:39:06.88] Building fsharp
393:  [19:39:06.90] Building diagnostics
419:  [19:39:06.99] Building xdt
445:  [19:39:22.56] Building cecil...done
449:  [19:39:36.04] Building xdt...done
453:  [19:39:45.76] Building deployment-tools...done
456:  [19:39:55.31] Building symreader...done
460:  [19:40:05.70] Building diagnostics...done
464:  [19:40:08.18] Building winforms...done
470:  [19:40:15.86] Building vstest
496:  [19:40:18.14] Building wpf
523:  [19:40:18.22] Building command-line-api...done
529:  [19:40:34.22] Building runtime
555:  [19:40:39.98] Building roslyn
581:  [19:41:50.04] Building templating...done
619:  [19:42:03.12] Building wpf...done
625:  [19:42:06.28] Building windowsdesktop
654:  [19:42:44.87] Building windowsdesktop...done

@agocke is there a good way to falsify CLR involvement here? Maybe a set of files I can patch in the SDK to keep that constant while updating everything else?

You mean like the shared framework? netcoreapp, basically?

Does this replicate when we build from source? Without using the LKG? That would let us bisect commit-by-commit, presumably.