Skip to content

Move libraries tests to xunit3#126277

Draft
Copilot wants to merge 18 commits intomainfrom
copilot/update-runtime-governance-docs
Draft

Move libraries tests to xunit3#126277
Copilot wants to merge 18 commits intomainfrom
copilot/update-runtime-governance-docs

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 29, 2026

Note

This PR was created with the assistance of GitHub Copilot.

This PR ports the test-only content from #125019 — moving libraries tests from xunit v2 to xunit v3. Production code is intentionally left unchanged.

Description

Key changes include:

  • xunit v3 package updates (eng/Versions.props, eng/testing/xunit/): Updates xunit dependencies from v2 to v3 packages
  • Build infrastructure (Directory.Build.targets, eng/testing/): Adds UseAppHost overrides and build validation for xunit v3 test projects
  • NuGet source (NuGet.config): Adds nuget.org feed for xunit v3 packages
  • Test runner updates (eng/testing/RunnerTemplate.sh, eng/testing/RunnerTemplate.cmd): Updates test execution to use xunit v3 runner
  • Test infrastructure (src/libraries/sendtohelix-browser.targets, src/libraries/sendtohelix-wasi.targets): Updates xunit trait filter syntax (-notrait-trait-) for xunit v3 compatibility
  • SingleFileTestRunner (src/libraries/Common/tests/SingleFileTestRunner/SingleFileTestRunner.cs): Updates for xunit v3 in-process runner APIs with AssemblyMetadata(3, ...) for NativeAOT compatibility
  • NativeAOT single-file targets (eng/testing/tests.singlefile.targets): Disables xunit auto-generated entry point for NativeAOT tests (uses SingleFileTestRunner instead)
  • Test code migrations (~640 test files): Updates test patterns for xunit v3 compatibility including:
    • TheoryData → typed TheoryData<T> with .Data accessor
    • IAsyncLifetime adjustments
    • Assert.ThrowsAsync pattern updates
    • Collection/class fixture changes
    • Skip mechanism updates (throw SkipTestExceptionAssert.SkipUnless/Assert.SkipWhen/Assert.Skip)
  • De Morgan's law skip condition fixes (6 total across 5 files): Fixed inverted skip conditions where compound negated if conditions were incorrectly converted to Assert.SkipUnless/Assert.SkipWhen without properly applying De Morgan's law:
    • HttpClientHandlerTest.Authentication.cs: if (!A || !B) throw SkipAssert.SkipUnless(A && B) (was incorrectly A || !B)
    • HttpClientHandlerTest.Headers.cs: if (!A && B) throw SkipAssert.SkipWhen(!A && B) (was incorrectly Assert.SkipUnless(A && B))
    • TextWriterTests.cs (2 instances): if (!A && !B) throw SkipAssert.SkipUnless(A || B) (was incorrectly A && !B)
    • BoundedChannelTests.cs: if (!A && !B) throw SkipAssert.SkipUnless(A || B) (was incorrectly A && !B)
    • RendezvousChannelTests.cs: if (!A && !B) throw SkipAssert.SkipUnless(A || B) (was incorrectly A && !B)
  • Documentation (docs/workflow/testing/libraries/xunit3-migration.md): Adds migration guide for common xunit v2→v3 patterns

All changes to production/source code (coreclr JIT, nativeaot CoreLib, System.Text.RegularExpressions, native libs, AndroidAppBuilder) have been reverted per reviewer feedback. Only test code, test infrastructure, and documentation changes remain.

See #125019 for full context and discussion.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Port of #125019

Co-authored-by: agocke <515774+agocke@users.noreply.github.com>
@dotnet-policy-service dotnet-policy-service bot added the linkable-framework Issues associated with delivering a linker friendly framework label Mar 29, 2026
@dotnet-policy-service
Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

The merge from upstream/main reintroduced xunit v2 patterns in files that
were modified on both branches:

- EdgeCaseTests.cs: [ConditionalTheory] -> [Theory] (fixes CS0618 build
  error blocking 21 CI jobs), SkipTestException -> SkipException.ForSkip()
- Regex.Match.Tests.cs: SkipTestException -> SkipException.ForSkip()
- DumpTestBase.cs: SkipTestException -> SkipException.ForSkip()

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

@agocke
Copy link
Copy Markdown
Member

agocke commented Mar 29, 2026

@copilot please revert all changes to production code

Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/366384e9-ee01-45a6-8bc0-e3ec75e0d12d

Co-authored-by: agocke <515774+agocke@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Mar 29, 2026

@copilot please revert all changes to production code

Reverted all 19 production code files to their original state in 7167df8. Files reverted:

  • src/coreclr/jit/codegenwasm.cpp, regallocwasm.cpp
  • src/coreclr/nativeaot/System.Private.CoreLib/src/.../ThunkedApis.cs
  • src/libraries/System.Text.RegularExpressions/src/... (3 files)
  • src/native/libs/... (9 files: JavaScript loader, CMakeLists, pal_*.c/.h/.m)
  • src/tasks/AndroidAppBuilder/Templates/monodroid-nativeaot.cs
  • src/libraries/sendtohelix-browser.targets, sendtohelix-wasi.targets

Only test code, test infrastructure (eng/testing/, Directory.Build.targets, NuGet.config, eng/Versions.props), and documentation changes remain.

@agocke
Copy link
Copy Markdown
Member

agocke commented Mar 29, 2026

@copilot these changes are test infra: src/libraries/sendtohelix-browser.targets, sendtohelix-wasi.targets

…-wasi.targets (test infra)

Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/257c3fec-6b69-4b38-b147-c12d2df91422

Co-authored-by: agocke <515774+agocke@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Mar 29, 2026

@copilot these changes are test infra: src/libraries/sendtohelix-browser.targets, sendtohelix-wasi.targets

Restored the xunit3 changes (-notrait-trait-) to both files in 4dd3f11.

</PropertyGroup>

<!--
Override the xunit.v3.core.mtp-v1 validation target. The original target
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like a situation where you could be using xunit.v3.extensibility.core instead of xunit.v3.core, especially if your custom runner replaces what xunit.v3.runner.inproc.console does. That would make sense in your situation (with a custom runner) and then these rules wouldn't end up being applied. In essence, xunit.v3.core is the package we think of as the package that says "you're planning to run your tests exactly how we intended for .NET Framework and/or .NET", and when you don't fit that mold, stepping away from that package makes sense to me.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I make this comment without understanding how your custom runner slots into anything. Feel free to ignore me. 😂)

xunit v3 relies on Assembly.Location which returns empty string in
NativeAOT, causing 'Argument was empty (Parameter AssemblyName)'.
xunit v4 (4.0.0-pre.33) fixes this internally.

Changes:
- Override XUnitV3Version to 4.0.0-pre.33 for TestNativeAot builds
- Update SingleFileTestRunner.cs for v4 API changes:
  - Namespace moves (ConsoleHelper, AssemblyMetadata, etc.)
  - Replace removed GetTargetFramework() with TargetFrameworkAttribute
  - Use non-obsolete Run() overload with resultWriters parameter

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

@agocke
Copy link
Copy Markdown
Member

agocke commented Mar 30, 2026

@copilot looks like there are api changes in the new xunit versions for the singlefiletestrunner: these failures are in the arm64 naot run: Running assembly:System.Collections.Tests, Version=11.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
Unhandled exception. System.ArgumentException: xunitVersion must be between 0 and 3 (Parameter 'xunitVersion')
at Xunit.Runner.Common.AssemblyMetadata..ctor(Int32 xunitVersion, String targetFramework) in //src/xunit.v3.runner.common/Utility/AssemblyMetadata.cs:line 42
at SingleFileTestRunner.Main(String[]) in /
/src/libraries/Common/tests/SingleFileTestRunner/SingleFileTestRunner.cs:line 60
at SingleFileTestRunner.Main(String[] args)
--- End of stack trace from previous location ---
at SingleFileTestRunner.

(String[] args)
DOTNET_DbgEnab

@github-actions

This comment has been minimized.

@agocke
Copy link
Copy Markdown
Member

agocke commented Apr 3, 2026

@bradwilson it looks like the regular runner isn’t compatible with single-file deployment (Assembly.Location returns empty string). Is that expected?

@bradwilson
Copy link
Copy Markdown

I don't know what you mean by "regular runner".

xunit v3 3.2.2 crashes in NativeAOT because Assembly.Location returns ""
and InProcessFrontController uses it without fallback. The existing
SingleFileTestRunner workaround patches XunitProjectAssembly but cannot
fix xunit's internal code paths.

Switch NativeAOT test builds to the xunit v4 AOT packages
(xunit.v3.core.aot 4.0.0-pre.33) which use source generators for test
discovery instead of runtime reflection, completely avoiding the
Assembly.Location dependency.

Key changes:
- eng/testing/xunit/xunit.props: Conditionally reference xunit.v3.core.aot
  when TestNativeAot=true; exclude Microsoft.DotNet.XUnitV3Extensions
  (incompatible with sealed FactAttribute/TheoryAttribute in AOT packages)
- eng/testing/tests.singlefile.targets: Enable xunit's auto-generated
  entry point for NativeAOT (no more SingleFileTestRunner for AOT builds)
- XUnitExtensionsShims.cs: Compile-time stubs for Extensions types
  (ConditionalFact, ActiveIssue, OuterLoop, etc.) so test source compiles
  without the Extensions package
- eng/Version.Details.props: Add XUnitV3AotVersion property

Trade-off: Only [Fact] and [Theory] tests are discovered in NativeAOT.
Tests using ConditionalFact/ConditionalTheory compile but are silently
excluded from AOT test runs (~1030 files affected, ~4900 [Fact] files
still work).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

@agocke
Copy link
Copy Markdown
Member

agocke commented Apr 6, 2026

@bradwilson Non-aot packages, just the regular packages, executing as normal

@bradwilson
Copy link
Copy Markdown

Then that is correct. Single file publishing, like Native AOT publishing, requires using the AOT packages.

@bradwilson
Copy link
Copy Markdown

And to be clear, we don't consider single file publishing to be a scenario anybody will use. 😁 The fact that it works with the AOT packages is a happy coincidence.

@agocke
Copy link
Copy Markdown
Member

agocke commented Apr 6, 2026

Is there a fundamental problem there? I'm surprised that there's such a strong tie to Assembly.Location, as opposed to something like AppContext.BaseDirectory

@bradwilson
Copy link
Copy Markdown

.NET Framework? 😂

@agocke
Copy link
Copy Markdown
Member

agocke commented Apr 6, 2026

AppContext.BaseDirectory is available in net4.6/netstandard2.0, if that helps. My thinking was that you would just check if Assembly.Location returns "", and if it does fall back to one of the other options. That way the existing code should continue to work in all currently working cases.

@bradwilson
Copy link
Copy Markdown

I can certainly add that as a possible fallback when things return null. Is there anything else that needs to be supported for single file deployments? (There are, AFAIK, no compiler errors or Roslyn analyzers that help out here, you just have to run things and 🤞🏼.)

@agocke
Copy link
Copy Markdown
Member

agocke commented Apr 7, 2026

@bradwilson There's an analyzer. You should be able to enable it for a project using <EnableSingleFileAnalyzer>true</EnableSingleFileAnalyzer>. The list of problematic APIs is much shorter: https://learn.microsoft.com/en-us/dotnet/core/deploying/single-file/overview?tabs=cli#api-incompatibility.

In practice I just see Assembly.Location. Note: it specifically returns empty string, not null.

@bradwilson
Copy link
Copy Markdown

I put in the changes suggested by the analyzer, which were just a few, and it's not enough. I'll have to debug and see what the analyzers are missing, but needing just a single change in the in-process runner felt hopelessly optimistic. 😂

@bradwilson
Copy link
Copy Markdown

@agocke At the moment, it looks like a decent amount of the loading/introspection changes that are required for AOT would also need to be used for non-AOT to support Single File, so my inclination is to say that Single File is only "supported" when you link against the AOT libraries.

Copilot AI review requested due to automatic review settings April 8, 2026 17:54
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@github-actions

This comment has been minimized.

@github-actions
Copy link
Copy Markdown
Contributor

🤖 Copilot Code Review — PR #126277

Note

This review was generated by Copilot and should be verified by a human reviewer.

Holistic Assessment

Motivation: The motivation is real and well-understood — xunit v3 crashes in NativeAOT because Assembly.Location returns "" and InProcessFrontController uses it without fallback. The existing SingleFileTestRunner workaround can't fix xunit's internal code paths. Switching to xunit v4 AOT packages that use source generators for test discovery is a sound strategy.

Approach: The core approach — using official AOT-compatible xunit packages — is the right direction. However, the implementation has significant gaps: a pre-release version is hardcoded in 8 locations instead of centralized, tests using conditional attributes (~1,200+ files) are silently excluded from NativeAOT runs with no tracking mechanism, and several utility projects still have unconditional non-AOT xunit references.

Summary: ⚠️ Needs Changes. The approach is sound but the implementation has issues that should be resolved before merge: (1) the hardcoded pre-release version must be centralized per repo conventions, (2) the unconditional Microsoft.DotNet.XUnitV3Extensions reference in installer/TestUtils.csproj will cause package conflicts, and (3) the scope of silently-excluded tests needs to be acknowledged with a tracking mechanism.


Detailed Findings

❌ Hardcoded pre-release version 4.0.0-pre.33 in 8 locations — Must be centralized

The version string 4.0.0-pre.33 appears in 8 separate locations across 5 files:

  • eng/testing/xunit/xunit.targets (2×)
  • src/libraries/Common/tests/TestUtilities/TestUtilities.csproj (3×)
  • src/libraries/Common/tests/StreamConformanceTests/StreamConformanceTests.csproj (2×)
  • src/installer/tests/TestUtils/TestUtils.csproj (1×)

The initial commit message (HEAD~2) states "eng/Version.Details.props: Add XUnitV3AotVersion property" but this change was apparently dropped in subsequent commits. All other xunit versions in this repo are centralized in eng/Version.Details.props (e.g., MicrosoftDotNetXUnitAssertPackageVersion, MicrosoftDotNetXUnitExtensionsPackageVersion). The XUnitV3Version property used for non-AOT packages is set by the xunit NuGet package itself.

The hardcoded version is a maintenance hazard: updating xunit AOT packages requires finding and changing 8 locations. Add a XUnitV3AotVersion property to the central version files and reference it everywhere.

Additionally, using a pre-release package (4.0.0-pre.33) is a risk — pre-release APIs can change or break. This should be tracked as a known dependency that needs to be updated before the release.

⚠️ installer/tests/TestUtils/TestUtils.csproj — Unconditional Microsoft.DotNet.XUnitV3Extensions reference

Lines 18-20 of this file have an unconditional reference:

<PackageReference Include="Microsoft.DotNet.XUnitV3Extensions" Version="$(MicrosoftDotNetXUnitExtensionsVersion)" />

When TestNativeAot=true, this project would reference both Microsoft.DotNet.XUnitV3Extensions (which transitively pulls in non-AOT xunit types with non-sealed FactAttribute) and xunit.v3.extensibility.core.aot (with sealed FactAttribute). This could cause type conflicts or assembly resolution failures. The xunit.v3.extensibility.core reference was correctly made conditional, but the XUnitV3Extensions reference was missed.

This should either be made conditional (like was done in xunit.targets and TestUtilities.csproj), or if this project is never compiled with TestNativeAot=true, that assumption should be documented.

⚠️ Silent test exclusion at scale — ~1,200+ files affected with no tracking

The stub attributes in AttributeStubs.cs (ConditionalFact, ConditionalTheory, ActiveIssue, OuterLoop, PlatformSpecific, SkipOnPlatform, SkipOnMono, SkipOnCoreClr, etc.) are plain [Attribute] types that don't inherit from xunit's FactAttribute/TheoryAttribute. The AOT source generator only discovers [Fact] and [Theory] methods.

By my count:

  • ~1,213 files use ConditionalFact/ConditionalTheory
  • ~1,427 files use other conditional attributes (ActiveIssue, PlatformSpecific, etc.)
  • ~5,587 files use plain [Fact]/[Theory]

The commit message acknowledges this trade-off (~1,030 files affected, ~4,900 still work), but there's no tracking issue, no test infrastructure that validates which tests are expected to run, and no way to detect if this set drifts over time. A new contributor adding [ConditionalFact] to a test won't know it silently vanishes from NativeAOT runs. Consider filing a tracking issue and adding a comment in AttributeStubs.cs linking to it.

⚠️ SkipTestException stub may cause false failures

The stub SkipTestException inherits from plain Exception, not xunit's skip mechanism. Tests that use [Fact] (discoverable by AOT generator) and throw SkipTestException internally will fail instead of skip in NativeAOT mode. I found ~5 such throw sites in src/libraries/ (e.g., SafeFileHandle/GetFileType.Unix.cs, PingTest.cs). While the count is small, these would produce confusing test failures in NativeAOT CI runs.

⚠️ Utility projects with unconditional non-AOT xunit references — Potential future conflicts

Several utility/test projects still have unconditional xunit.v3.extensibility.core or Microsoft.DotNet.XUnitV3Extensions references without TestNativeAot guards:

  • src/libraries/Common/tests/TestUtilities.Unicode/TestUtilities.Unicode.csproj
  • src/libraries/Common/tests/System/Xml/ModuleCore/ModuleCore.csproj
  • src/libraries/System.Private.Xml/tests/System.Private.Xml.Tests.csproj
  • src/libraries/Microsoft.Extensions.DependencyInjection.Specification.Tests/...
  • src/coreclr/tools/aot/ILCompiler.Compiler.Tests/ILCompiler.Compiler.Tests.csproj
  • src/coreclr/tools/aot/ILCompiler.TypeSystem.Tests/ILCompiler.TypeSystem.Tests.csproj

If any of these are ever compiled with TestNativeAot=true, mixing AOT and non-AOT xunit packages would likely cause type conflicts. This may be fine today if these projects are never part of NativeAOT test runs, but it's fragile. At minimum, document which projects are expected to be NativeAOT-aware, or make the guards consistent.

InlineDataTheoryData/MemberData conversion — Correct approach

The SSL test files (HttpClientHandlerTest.AcceptAllCerts.cs, HttpClientHandlerTest.SslProtocols.cs) correctly convert [InlineData] with #if preprocessor directives to static TheoryData<> properties with [MemberData]. The AOT source generator doesn't support conditional compilation in attributes, so this is the right fix. The #pragma warning placement is also correctly maintained.

✅ ArrayBuilderTests pattern — Reasonable workaround

Moving test attributes from the generic base class to concrete subclasses via new method hiding is a pragmatic workaround for the AOT source generator's inability to discover methods on open generic types. The #pragma warning disable xUnit1024 is appropriately scoped and documented. The comment explaining the reason is clear.

#if NET guard in SingleFileTestRunner.cs — Correct

Environment.ProcessPath is a .NET 6+ API. The #if NET guard prevents compilation errors when targeting non-.NETCoreApp TFMs, which is correct since SingleFileTestRunner.cs is compiled for multiple frameworks.

💡 AotXunitExtensions.csproj targets $(NetCoreAppMinimum) — Consider alignment

The project targets $(NetCoreAppMinimum) (net10.0), which works since it's forward-compatible. However, since this project is exclusively consumed in NativeAOT test contexts targeting $(NetCoreAppCurrent), consider targeting $(NetCoreAppCurrent) for alignment. Not blocking.

💡 CS8625 suppression in DirectoryServices test project — Narrow but acceptable

Suppressing CS8625 (cannot convert null literal to non-nullable reference type) globally for NativeAOT builds is a pragmatic workaround for the AOT source generator's code generation limitations. The comment explains the reason clearly. Consider filing an upstream issue with xunit if not already tracked.

Generated by Code Review for issue #126277 ·

@bradwilson
Copy link
Copy Markdown

bradwilson commented Apr 12, 2026

New prerelease builds are available today that include lots of Native AOT bug fixes as well as performance improvements (moving the diagnostics from source generation to the analyzers package). /cc @agocke

https://xunit.net/releases/v3/4.0.0-pre.81
https://xunit.net/releases/analyzers/2.0.0-pre.40
https://xunit.net/releases/visualstudio/4.0.0-pre.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-Meta linkable-framework Issues associated with delivering a linker friendly framework

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

4 participants