Enable NativeAOT library tests on Apple mobile platforms by kotlarmilos · Pull Request #125437 · dotnet/runtime

kotlarmilos · 2026-03-11T12:46:57Z

Description

Replace NativeAOT runtime test legs with library test legs on iOS/tvOS devices, iOS/tvOS simulators, and Mac Catalyst.

Replace NativeAOT runtime test legs with library test legs on iOS/tvOS devices, iOS/tvOS simulators, and Mac Catalyst. The runtime tests (nativeaot/SmokeTests) provide minimal value on mobile since the compiler doesn't meaningfully distinguish between iOS and macOS. Library tests exercise OS-specific APIs (crypto, networking, etc.) that are the real quality gate. Changes: - ioslike: remove NativeAOT runtime test leg, update existing library test leg to use dynamic smoke test arg - ioslikesimulator: replace NativeAOT runtime test leg with library test leg, fix isiOSLikeSimulatorOnlyBuild parameter - maccatalyst: replace both NativeAOT runtime test legs (regular + AppSandbox) with library test legs, fix coreclrContainsChange variable (was incorrectly using monoContainsChange) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR updates the Apple “extra platforms” CI legs to validate NativeAOT on iOS/tvOS devices, iOS/tvOS simulators, and Mac Catalyst by running libraries tests (via Helix) instead of the minimal NativeAOT runtime SmokeTests, improving OS-specific coverage (crypto/networking/globalization) on mobile.

Changes:

Switch NativeAOT Apple-mobile legs from runtime SmokeTests submission to eng/pipelines/libraries/helix.yml submission of libraries tests.
Use eng/pipelines/libraries/helix-queues-setup.yml and pass NeedsToBuildAppsOnHelix=true for the new legs.
Update naming/args to use $(_runSmokeTestsOnlyArg) and fix path-evaluation variable naming in affected legs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml	Replaces NativeAOT runtime SmokeTests legs with libraries test legs (including App Sandbox variant) using libraries Helix templates/queues.
eng/pipelines/extra-platforms/runtime-extra-platforms-ioslikesimulator.yml	Replaces NativeAOT runtime SmokeTests on iOS/tvOS simulators with libraries tests using libraries Helix templates/queues.
eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml	Adjusts NativeAOT iOS/tvOS device leg to use `$(_runSmokeTestsOnlyArg)` for libraries tests and removes the old NativeAOT runtime SmokeTests device leg.

You can also share your feedback on Copilot code review. Take the survey.

eng/pipelines/extra-platforms/runtime-extra-platforms-ioslikesimulator.yml

…60312.1 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26160.2 -> To Version 11.0.0-prerelease.26162.1

…60312.3 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26160.2 -> To Version 11.0.0-prerelease.26162.3

…60314.3 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26160.2 -> To Version 11.0.0-prerelease.26164.3

…60317.2 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26160.2 -> To Version 11.0.0-prerelease.26167.2

…60318.1 On relative base path root Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Common , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 11.0.0-prerelease.26160.2 -> To Version 11.0.0-prerelease.26168.1

…bb6-53258cb4f260' into enable-nativeaot-libs-tests-mobile

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

You can also share your feedback on Copilot code review. Take the survey.

kotlarmilos · 2026-04-09T11:10:21Z

This PR stands up Native AOT library tests on Apple mobile. It replaces a subset of runtime tests that were not relevant for Apple mobile. Note it enables only System.Runtime.Tests. The full run revealed 66 failures tracked in #126697. I briefly went through some of the failures and they look trimming related.

The follow-up effort should bring these tests into the CI to improve testing coverage of Native AOT along with CoreCLR.

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

src/libraries/Common/tests/AppleTestRunner/AppleTestRunner.cs

src/mono/msbuild/apple/data/ProxyProjectForAOTOnHelix.proj

…eInput handling in ProxyProjectForAOTOnHelix

src/libraries/System.Runtime/tests/System.Runtime.Tests/System/Reflection/ModuleTests.cs

…ple mobile ActiveIssue annotations Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

src/tasks/AppleAppBuilder/Xcode.cs

src/libraries/Common/tests/AppleTestRunner/AppleTestRunner.cs

kotlarmilos · 2026-04-10T08:49:16Z

@jkoritzinsky @matouskozak Please take a look again

jkotas · 2026-04-11T03:31:22Z

src/libraries/Common/tests/System/ModifiedTypeTests.cs

        }

        [Fact]
+        [ActiveIssue("https://github.com/dotnet/runtime/issues/67531", typeof(PlatformDetection), nameof(PlatformDetection.IsNativeAot), nameof(PlatformDetection.IsAppleMobile))]


This issue does not look related to what the tests is testing.

Is this correct issue link?

Likely the same underlying problem as #126730 (comment)

Yes, it is likely related to trimming. Let's try a different approach and trim on build machines.

MichalStrehovsky · 2026-04-13T04:16:51Z

Actually looking at the changes here, all the trimming issues are likely caused by mismatch between how compilation would happen with regular publish vs what ProxyProjectForAOTOnHelix.proj sets up and we are ending up building different configurations.

Do we need to keep doing ProxyProjectForAOTOnHelix.proj? Why can't we let the publish happen on the build machines, like the other platforms?

kotlarmilos · 2026-04-13T08:19:34Z

The only concern is build time on Helix machines. dotnet publish would include app bundling and signing, which is time consuming. However, you are correct. it should be done on build machines, and going forward this approach would only cause problems, needing to match dotnet publish with what we do on Helix.

Let's try to split trimming and publish, and do trimming and ILC on build machines and app bundling on Helix.

steveisok · 2026-04-13T10:01:55Z

Let's try to split trimming and publish, and do trimming and ILC on build machines and app bundling on Helix.

Worth a try but chances are good you'll timeout. Depends on how much extra time it adds on the build step.

MichalStrehovsky · 2026-04-13T11:24:05Z

Worth a try but chances are good you'll timeout. Depends on how much extra time it adds on the build step.

We use a 6 hour timeout for the libraries legs in native AOT testing. In practice it's less than that, but this accounts for Helix delays as well. I just checked a recent ARM64 mac outerloop leg build: it was about 3 hours to build the product and tests (including AOT compiling them). The Build product leg of the maccatalyst leg in this PR was 50 minutes (but we also didn't build all the tests, only System.Runtime, so it's not representative).

It will take longer, but not LLVM-longer. But in return we get compat with all the switches the tests might be setting (trimming settings are not the only things we sometimes tweak per test project - it's also things like invariant globalization and bunch of others that we can set at compile time - root causing the failing tests to a missing compile-time switch can be pretty annoying).

Copilot AI review requested due to automatic review settings March 11, 2026 12:46

github-actions bot added the area-Infrastructure-mono label Mar 11, 2026

dotnet-policy-service bot assigned kotlarmilos Mar 11, 2026

Copilot started reviewing on behalf of kotlarmilos March 11, 2026 12:47 View session

kotlarmilos added area-Infrastructure-coreclr os-ios Apple iOS and removed area-Infrastructure-mono labels Mar 11, 2026

github-project-automation bot added this to Runtime Infra Mar 11, 2026

kotlarmilos requested review from MichalStrehovsky, jkoritzinsky, matouskozak and vitek-karas March 11, 2026 12:50

Copilot AI reviewed Mar 11, 2026

View reviewed changes

eng/pipelines/extra-platforms/runtime-extra-platforms-ioslikesimulator.yml Show resolved Hide resolved

matouskozak approved these changes Mar 11, 2026

View reviewed changes

build-analysis bot mentioned this pull request Mar 11, 2026

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." dotnet/dnceng#1883

Open

3 tasks

jkoritzinsky approved these changes Mar 11, 2026

View reviewed changes

dotnet-maestro bot and others added 5 commits March 12, 2026 10:30

Merge branch 'main' into enable-nativeaot-libs-tests-mobile

9e2a877

This was referenced Mar 17, 2026

XHarness package install failure on iOS due to devicectl NSPOSIXErrorDomain error 49 #123796

Open

MsQuic fails with QUIC_STATUS_OUT_OF_MEMORY on AzureLinux #123216

Open

kotlarmilos marked this pull request as draft March 18, 2026 12:57

Merge remote-tracking branch 'upstream/darc-main-98c14788-b6c2-47d8-8…

98a831a

…bb6-53258cb4f260' into enable-nativeaot-libs-tests-mobile

Copilot AI review requested due to automatic review settings March 18, 2026 13:02

Copilot started reviewing on behalf of kotlarmilos March 18, 2026 13:03 View session

Copilot AI reviewed Mar 18, 2026

View reviewed changes

kotlarmilos marked this pull request as ready for review April 9, 2026 11:07

kotlarmilos requested a review from steveisok as a code owner April 9, 2026 11:07

Copilot AI review requested due to automatic review settings April 9, 2026 11:07

Copilot started reviewing on behalf of kotlarmilos April 9, 2026 11:09 View session

kotlarmilos requested review from davidnguyen-tech, jkoritzinsky and matouskozak April 9, 2026 11:10

Copilot AI reviewed Apr 9, 2026

View reviewed changes

src/libraries/Common/tests/AppleTestRunner/AppleTestRunner.cs Show resolved Hide resolved

src/mono/msbuild/apple/data/ProxyProjectForAOTOnHelix.proj Show resolved Hide resolved

src/mono/msbuild/apple/data/ProxyProjectForAOTOnHelix.proj Outdated Show resolved Hide resolved

Update AppleTestRunner to use test assembly name and refine IlcCompil…

a8eae52

…eInput handling in ProxyProjectForAOTOnHelix

kotlarmilos force-pushed the enable-nativeaot-libs-tests-mobile branch from 01d5d83 to a8eae52 Compare April 9, 2026 11:34

jkotas reviewed Apr 9, 2026

View reviewed changes

src/libraries/System.Runtime/tests/System.Runtime.Tests/System/Reflection/ModuleTests.cs Outdated Show resolved Hide resolved

kotlarmilos mentioned this pull request Apr 9, 2026

Module reflection tests fail on NativeAOT Apple mobile #126730

Open

Replace closed issue dotnet#51912 with dotnet#126730 for NativeAOT Ap…

3d47dc6

…ple mobile ActiveIssue annotations Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 9, 2026 20:33

Copilot started reviewing on behalf of kotlarmilos April 9, 2026 20:35 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

src/tasks/AppleAppBuilder/Xcode.cs Show resolved Hide resolved

src/libraries/Common/tests/AppleTestRunner/AppleTestRunner.cs Show resolved Hide resolved

src/libraries/Common/tests/AppleTestRunner/AppleTestRunner.cs Show resolved Hide resolved

jkoritzinsky approved these changes Apr 10, 2026

View reviewed changes

jkotas reviewed Apr 11, 2026

View reviewed changes

Copilot AI mentioned this pull request Apr 11, 2026

Fix WasmTestOnChrome: exclude browser/wasi from gen-debug-dump-docs.py helix post-command #126789

Open

1 task

matouskozak approved these changes Apr 13, 2026

View reviewed changes

kotlarmilos marked this pull request as draft April 13, 2026 11:44

Conversation

kotlarmilos commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

kotlarmilos commented Apr 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kotlarmilos commented Apr 10, 2026

Uh oh!

jkotas Apr 11, 2026

Choose a reason for hiding this comment

Uh oh!

jkotas Apr 11, 2026

Choose a reason for hiding this comment

Uh oh!

kotlarmilos Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

MichalStrehovsky commented Apr 13, 2026

Uh oh!

kotlarmilos commented Apr 13, 2026

Uh oh!

steveisok commented Apr 13, 2026

Uh oh!

MichalStrehovsky commented Apr 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

kotlarmilos commented Mar 11, 2026 •

edited

Loading