Apparent heap corruption in crossgen2 on NativeAOT Arm64 Linux #72831
Description
Very easy to hit on a Raspberry Pi 4 on AArch64 Ubuntu 22.04. We also have a customer report for hitting this on a beefier Ampere device (there it requires running as /usr/bin/taskset --all-tasks --cpu-list 1,2,3 to limit the number of CPUs). Probably any ARM64 device would hit this.
The repro I just tried on Ubuntu:
curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin --version 7.0.100-rc.1.22375.3 --install-dir ~/netrc/
~/netrc/dotnet new webapi -o webapi
cd webapi
cat /dev/tty > nuget.config
<configuration>
<packageSources>
<add key="dotnet7" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json" />
</packageSources>
</configuration>
^Z
~/netrc/dotnet publish -r linux-arm64 -c Release /p:PublishTrimmed=true /p:PublishReadyToRun=true
The stack traces are all over the place:
error : Error: Object reference not set to an instance of an object. [/home/michal/webapi/webapi.csproj]
error : System.NullReferenceException: Object reference not set to an instance of an object. [/home/michal/webapi/webapi.csproj]
error : at System.Threading.Lock.ReleaseContended() + 0x50 [/home/michal/webapi/webapi.csproj]
error : at System.Threading.Lock.Release() + 0x70 [/home/michal/webapi/webapi.csproj]
error : at System.Collections.Concurrent.ConcurrentDictionary`2.TryAddInternal(TKey, Nullable`1, TValue, Boolean, Boolean, TValue&) + 0x348 [/home/michal/webapi/webapi.csproj]
error : at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2) + 0xbc [/home/michal/webapi/webapi.csproj]
error : at ILCompiler.DependencyAnalysis.ReadyToRun.CopiedMetadataBlobNode.WriteMethodTableRvas(NodeFactory, ObjectDataBuilder&, BlobReader&) + 0xd0 [/home/michal/webapi/webapi.csproj]
error : at ILCompiler.DependencyAnalysis.ReadyToRun.CopiedMetadataBlobNode.GetData(NodeFactory, Boolean) + 0x1b0 [/home/michal/webapi/webapi.csproj]
error : at ILCompiler.DependencyAnalysis.ObjectNode.GetStaticDependencies(NodeFactory) + 0x48 [/home/michal/webapi/webapi.csproj]
error : at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependenciesImpl(DependencyNodeCore`1) + 0x48 [/home/michal/webapi/webapi.csproj]
error : at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.GetStaticDependencies(DependencyNodeCore`1) + 0x40 [/home/michal/webapi/webapi.csproj]
error : at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ProcessMarkStack() + 0xd4 [/home/michal/webapi/webapi.csproj]
error : at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ComputeMarkedNodes() + 0x4c [/home/michal/webapi/webapi.csproj]
error : at ILCompiler.ReadyToRunCodegenCompilation.Compile(String) + 0x3c [/home/michal/webapi/webapi.csproj]
error : at ILCompiler.Program.RunSingleCompilation(Dictionary`2, InstructionSetSupport, String, Dictionary`2, HashSet`1, CompilerTypeSystemContext) + 0x1418 [/home/michal/webapi/webapi.csproj]
error : at ILCompiler.Program.Run(String[]) + 0xbbc [/home/michal/webapi/webapi.csproj]
error : at ILCompiler.Program.Main(String[]) + 0x34 [/home/michal/webapi/webapi.csproj]
error : ---> ILCompiler.CodeGenerationFailedException: Code generation failed for method '[Microsoft.AspNetCore.Http.Abstractions]Microsoft.AspNetCore.Http.PathString.StartsWithSegments(PathString,StringComparison,PathString&)' [/home/michal/webapi/webapi.csproj]
error : ---> System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. [/home/michal/webapi/webapi.csproj]
error : at System.ThrowHelper.ThrowArgumentOutOfRangeException() + 0x28 [/home/michal/webapi/webapi.csproj]
error : at System.Text.Encoding.GetByteCountWithFallback(Char*, Int32, Int32) + 0x50 [/home/michal/webapi/webapi.csproj]
error : at System.Text.UTF8Encoding.GetByteCount(String) + 0x84 [/home/michal/webapi/webapi.csproj]
error : at Internal.JitInterface.CorInfoImpl.StringToUTF8(String) + 0x30 [/home/michal/webapi/webapi.csproj]
error : at Internal.JitInterface.CorInfoImpl.getMethodNameFromMetadata(CORINFO_METHOD_STRUCT_*, Byte**, Byte**, Byte**) + 0x140 [/home/michal/webapi/webapi.csproj]
error : at Internal.JitInterface.CorInfoImpl._getMethodNameFromMetadata(IntPtr, IntPtr*, CORINFO_METHOD_STRUCT_*, Byte**, Byte**, Byte**) + 0x5c [/home/michal/webapi/webapi.csproj]
I have a crashdump that I was looking at that has:
[0x3] crossgen2!SystemNative_Abort + 0x8
[0x4] crossgen2!S_P_CoreLib_System_RuntimeExceptionHelpers__FailFast_1 + 0xb8
[0x5] crossgen2!S_P_CoreLib_System_RuntimeExceptionHelpers__GetRuntimeException + 0x33c
[0x6] crossgen2!S_P_CoreLib_System_Runtime_EH__GetClasslibException + 0x38
[0x7] crossgen2!RhThrowHwEx + 0xd8
[0x8] crossgen2!RhpThrowHwEx + 0x88
[0x9] crossgen2!System_Collections_Concurrent_System_Collections_Concurrent_ConcurrentDictionary_2<System___Canon__Bool>__TryGetValueInternal + 0x44
[0xa] crossgen2!System_Collections_Concurrent_System_Collections_Concurrent_ConcurrentDictionary_2<System___Canon__Bool>__GetOrAdd + 0x78
[0xb] crossgen2!ILCompiler_ReadyToRun_ILCompiler_DependencyAnalysis_ReadyToRun_ModuleTokenResolver__AddModuleTokenForType + 0x13c
[0xc] crossgen2!ILCompiler_ReadyToRun_ILCompiler_DependencyAnalysis_ReadyToRun_ModuleTokenResolver_TokenResolverProvider__GetTypeFromReference + 0x90
[0xd] crossgen2!System_Reflection_Metadata_System_Reflection_Metadata_Ecma335_SignatureDecoder_2<System___Canon__System___Canon>__DecodeTypeHandle + 0x124
[0xe] crossgen2!System_Reflection_Metadata_System_Reflection_Metadata_Ecma335_SignatureDecoder_2<System___Canon__System___Canon>__DecodeType_0 + 0x348
[0xf] crossgen2!System_Reflection_Metadata_System_Reflection_Metadata_Ecma335_SignatureDecoder_2<System___Canon__System___Canon>__DecodeType_0 + 0x2a8
[0x10] crossgen2!ILCompiler_ReadyToRun_ILCompiler_DependencyAnalysis_ReadyToRun_ModuleTokenResolver__AddModuleTokenForType + 0x124
[0x11] crossgen2!ILCompiler_ReadyToRun_ILCompiler_DependencyAnalysis_ReadyToRun_ModuleTokenResolver__AddModuleTokenForMethod + 0x21c
[0x12] crossgen2!ILCompiler_ReadyToRun_Internal_JitInterface_CorInfoImpl__resolveToken + 0x1a8
[0x13] crossgen2!ILCompiler_ReadyToRun_Internal_JitInterface_CorInfoImpl___resolveToken + 0x3c
We've seen a bunch of apparent corruptions around ConcurrentDictionary even before the NativeAOT switchover: see #69192. But I'm not able to repro this with crossgen2 without NativeAOT anymore. So it's either a NativeAOT bug or the bug doesn't repro without regions (we still run GC in a different configuration than the rest of the shipping product unfortunately).