/azp run

Looking at this failure now.

Error message:

Emitting PerfMap file: /__w/1/s/artifacts/source-build/self/src/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/net7.0/linux-x64/Microsoft.Net.Http.Headers.ni.r2rmap
  EXEC : error : Multiple input files matching same simple name /__w/1/s/artifacts/source-build/self/src/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/net7.0/linux-x64/System.Text.Json.dll /__w/1/s/artifacts/source-build/self/package-cache/microsoft.netcore.app.runtime.linux-x64/7.0.0/runtimes/linux-x64/lib/net7.0/System.Text.Json.dll [/__w/1/s/artifacts/source-build/self/src/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj]
##[error]EXEC(0,0): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Multiple input files matching same simple name /__w/1/s/artifacts/source-build/self/src/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/net7.0/linux-x64/System.Text.Json.dll /__w/1/s/artifacts/source-build/self/package-cache/microsoft.netcore.app.runtime.linux-x64/7.0.0/runtimes/linux-x64/lib/net7.0/System.Text.Json.dll
Internal.CommandLine.CommandLineException: Multiple input files matching same simple name /__w/1/s/artifacts/source-build/self/src/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/net7.0/linux-x64/System.Text.Json.dll /__w/1/s/artifacts/source-build/self/package-cache/microsoft.netcore.app.runtime.linux-x64/7.0.0/runtimes/linux-x64/lib/net7.0/System.Text.Json.dll

The specific DLL is not always the same but it does seem to always be the MS.AspNetCore.App.Runtime package.

@MichaelSimons and I have been looking at this issue for quite a while now and so far we've discovered:

• This repros consistently in the full tarball source-build.
• It repros inconsistently in the aspnetcore source-build and non-source-build configurations.
• It repros in both online and offline source-builds, so we think it's unlikely that it's the NuGet restore nondeterminism possibility that was raised.
  • The CopyLocal nondeterminism still seems possible but we don't know where that would be or how it would happen.
• This is the first time we know of the issue showing up in source-build.
• The changes from runtime and aspnetcore in this PR do not appear to be related to the issue.

We're now building a 7.0 tip full tarball source-build to compare against this PR, but we're kind of at a loss and could use some help diagnosing the issue. Any ideas? cc some source-build champs: @janvorli @rainersigwald @wtgodbe

• @dougbu - please see @crummel's previous comment

The command line args for crossgen2 are

CommandLineArguments = "/repos/tarball-7.0.1xx/src/aspnetcore/artifacts/source-build/self/package-cache/microsoft.netcore.app.crossgen2.linux-x64/7.0.0/tools/crossgen2" --composite --targetarch:x64 --targetos:linux /repos/tarball-7.0.1xx/src/aspnetcore/artifacts/source-build/self/package-cache/microsoft.netcore.app.runtime.linux-x64/7.0.0/runtimes/linux-x64/lib/net7.0/*.dll /repos/tarball-7.0.1xx/src/aspnetcore/artifacts/source-build/self/src/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/net7.0/linux-x64/*.dll --out:"/repos/tarball-7.0.1xx/src/aspnetcore/artifacts/source-build/self/src/artifacts/obj/Composite/Release/full-composite.r2r.dll"

Comparing the layout of 7.0.1xx tip and this PR I see a difference in the contents of /repos/tarball-7.0.1xx/src/aspnetcore/artifacts/source-build/self/src/artifacts/bin/Microsoft.AspNetCore.App.Runtime/Release/net7.0/linux-x64/*.dll

This PR is includes System.Formats.Asn1.dll, System.Text.Encodings.Web.dll and System.Text.Json.dll

I have binlogs available for the two environments if needed.

/cc @wtgodbe

@crummel @MichaelSimons in dotnet/aspnetcore, we hit exactly the same issue prior to our equivalent of this PR going in. The problem turned out to be mixing a prior public stable-versioned feed for a dotnet/runtime 7.0.0 build w/ the dotnet-public feed containing same-versioned copies of packages. In the System.Text.Json and System.Text.Encodings.Web cases, there were differences in the contained assemblies that caused the problem.

In the System.Text.Json and System.Text.Encodings.Web cases, there were differences in the contained assemblies that caused the problem.

@dougbu - what differences did you see?

Over to @wtgodbe for the details. In brief, one assembly exposed different APIs than expected and the other caused the Multiple input files matching same simple name errors in our release/7.0 builds (and PRs targeting that branch) after dotnet-public was updated and before we merged our merge-from-internal PR.
I suspect either source-build has something similar going on (perhaps still) or everything targeting release/7.0.1xx is failing and will continue to fail until you're using newer (7.0.1) runtime and aspnetcore bits here.