Reduce PR validation jobs and clean up test scheduling by radical · Pull Request #15031 · dotnet/aspire

radical · 2026-03-07T02:29:00Z

Description

This change reduces the jobs we run during PR validation and fixes unnecessary jobs that were still being scheduled because of test-enumeration mismatches.

This PR is organized into five focused commits while preserving the same final tree:

Exclude outerloop and quarantined tests from class enumeration
- Updates split-test-projects-for-ci.ps1 so class-based partition discovery does not enumerate tests that regular CI already excludes.
Make Aspire test macOS selection PR-aware
- GitHub-hosted macOS runners are less available and can add unnecessary delay to PR validation.
- Adds IsGithubPullRequest in eng/Testing.props and, for GitHub pull requests only, defaults RunOnGithubActionsMacOS=false for Aspire.* test projects except Aspire.Cli* unless a project explicitly overrides it.
- This keeps fuller CI coverage outside PR validation while reducing queue pressure on PRs.
- Removes now-redundant per-project macOS overrides from Aspire.Deployment.EndToEnd.Tests, Aspire.EndToEnd.Tests, and Aspire.Playground.Tests.
Prefer OuterloopTest in repository guidance
- Switches the remaining template tests on this branch back to the repository's OuterloopTest attribute.
- Adds AGENTS guidance to keep using OuterloopTest in this repo instead of Arcade's OuterLoop attribute.
Require Docker for the built-in starter template test
- Marks BuildAndRunStarterTemplateBuiltInTest_Test with the missing Docker feature requirement so CI only schedules it where Docker is available.
Limit QuarantineTools.Tests scheduling
- Restricts QuarantineTools.Tests to the intended GitHub Actions Linux environment to avoid unnecessary runs on other CI lanes.

Must check before merge

Outerloop workflow works and correctly runs Aspire.Oracle.EntityFrameworkCore.Tests.
Regular CI does not run Aspire.Oracle.EntityFrameworkCore.Tests.

github-actions · 2026-03-07T02:29:09Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 15031

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 15031"

Copilot

Pull request overview

This PR consolidates test-infrastructure cleanup across the repo by standardizing on Arcade’s [OuterLoop] attribute, updating CI trait filters/scripts/docs accordingly, and centralizing some CI/MSBuild test properties (GitHub PR detection and default macOS execution behavior).

Changes:

Replaces the custom OuterloopTestAttribute with Arcade’s [OuterLoop] across test code and updates trait filtering from outerloop=true to category=outerloop.
Adds OuterloopTestProject=true for Oracle EF Core tests and updates the specialized test project discovery script to include csproj-designated outerloop projects.
Centralizes CI defaults in eng/Testing.props (GitHub PR detection and default skip-on-macOS for Aspire.* projects, with Aspire.Cli* excluded).

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/Aspire.TestUtilities/OuterloopTestAttribute.cs	Deletes the custom outerloop trait attribute implementation.
tests/Aspire.Templates.Tests/StarterTemplateRunTestsBase.cs	Switches Playwright tests from `[OuterloopTest]` to `[OuterLoop]`.
tests/Aspire.Templates.Tests/EmptyTemplateRunTests.cs	Switches Playwright test from `[OuterloopTest]` to `[OuterLoop]`.
tests/Aspire.Playground.Tests/Aspire.Playground.Tests.csproj	Removes redundant macOS skip override (relies on centralized defaults).
tests/Aspire.Oracle.EntityFrameworkCore.Tests/AssemblyInfo.cs	Removes assembly-level outerloop marker.
tests/Aspire.Oracle.EntityFrameworkCore.Tests/Aspire.Oracle.EntityFrameworkCore.Tests.csproj	Adds `OuterloopTestProject=true` to designate the project as outerloop.
tests/Aspire.Hosting.Tests/DistributedApplicationTests.cs	Switches several long-running tests to `[OuterLoop]`.
tests/Aspire.Hosting.JavaScript.Tests/AddJavaScriptAppTests.cs	Switches Docker build test to `[OuterLoop]`.
tests/Aspire.EndToEnd.Tests/Aspire.EndToEnd.Tests.csproj	Removes redundant macOS skip override (relies on centralized defaults).
tests/Aspire.Deployment.EndToEnd.Tests/Aspire.Deployment.EndToEnd.Tests.csproj	Removes redundant macOS skip override (relies on centralized defaults).
tests/Aspire.Dashboard.Tests/Integration/Playwright/BrowserTokenAuthenticationTests.cs	Switches Playwright tests to `[OuterLoop]`.
tests/Aspire.Dashboard.Tests/Integration/Playwright/AppBarTests.cs	Switches Playwright test to `[OuterLoop]`.
tests/Aspire.Cli.Tests/Mcp/E2E/McpDocsE2ETests.cs	Switches E2E tests to `[OuterLoop]` and removes old utility using.
tests/Aspire.Cli.EndToEnd.Tests/PlaywrightCliInstallTests.cs	Switches E2E test class to `[OuterLoop]` and removes old utility using.
tests/Aspire.Cli.EndToEnd.Tests/McpDocsE2ETests.cs	Switches E2E tests to `[OuterLoop]` and removes old utility using.
eng/scripts/split-test-projects-for-ci.ps1	Excludes outerloop/quarantined tests during class-based test enumeration.
eng/scripts/generate-specialized-test-projects-list.sh	Adds csproj-based outerloop project discovery (`OuterloopTestProject`).
eng/Testing.props	Updates outerloop trait filters, adds `IsGithubPullRequest`, and centralizes macOS skip defaults for `Aspire.*` tests.
docs/outerloop-tests.md	Updates documentation to `[OuterLoop]` and new trait filter syntax.
docs/ci/TestingOnCI.md	Updates CI filter documentation to `category=outerloop`.
AGENTS.md	Updates agent guidance to use the new outerloop filter syntax.
.github/workflows/tests.yml	Updates default test args to exclude `category=outerloop`.
.github/workflows/tests-outerloop.yml	Updates workflow to select `OuterLoop` attribute name and filter by `category=outerloop`.
.github/workflows/specialized-test-runner.yml	Updates attributeName description text.
.github/skills/test-management/SKILL.md	Updates examples to exclude `category=outerloop`.

tests/Aspire.Oracle.EntityFrameworkCore.Tests/AssemblyInfo.cs

Copilot · 2026-03-07T02:36:47Z

tests/Aspire.Oracle.EntityFrameworkCore.Tests/Aspire.Oracle.EntityFrameworkCore.Tests.csproj


  <PropertyGroup>
    <TargetFrameworks>$(AllTargetFrameworks)</TargetFrameworks>
+    <OuterloopTestProject>true</OuterloopTestProject>


OuterloopTestProject is introduced here, but nothing in the current test execution path consumes it to (a) exclude the project from regular CI runs or (b) include it in outerloop runs. As-is, outerloop workflows still filter on category=outerloop, so this project won’t run there, and it will start running in regular CI. Either tag the tests with [OuterLoop] (assembly/class/method) or update the MSBuild/test runner logic to treat OuterloopTestProject=true as an outerloop designation (skip in normal runs, run without trait filtering in outerloop runs).

Suggested change

<OuterloopTestProject>true</OuterloopTestProject>

eng/scripts/generate-specialized-test-projects-list.sh

Copilot · 2026-03-07T02:36:48Z

.github/workflows/tests-outerloop.yml

+      attributeName: "OuterLoop"
      extraRunSheetBuilderArgs: "-p:RunOuterloopTests=true"
-      extraTestArgs: "--filter-trait outerloop=true"
+      extraTestArgs: "--filter-trait category=outerloop"


With the move to OuterloopTestProject=true (csproj-based outerloop designation), always passing --filter-trait category=outerloop here will prevent those projects from running unless every test is still tagged with the outerloop trait. If the intent is to allow “entire project is outerloop” semantics, consider removing this trait filter from the workflow and letting MSBuild/props decide what to run for outerloop (or alternatively ensure the project’s tests are tagged with [OuterLoop]).

Suggested change

extraTestArgs: "--filter-trait category=outerloop"

docs/outerloop-tests.md

github-actions · 2026-03-07T03:01:02Z

🎬 CLI E2E Test Recordings

The following terminal recordings are available for commit 570921d:

Test	Recording
AddPackageInteractiveWhileAppHostRunningDetached	▶️ View Recording
AddPackageWhileAppHostRunningDetached	▶️ View Recording
AgentCommands_AllHelpOutputs_AreCorrect	▶️ View Recording
AgentInitCommand_DefaultSelection_InstallsSkillOnly	▶️ View Recording
AgentInitCommand_MigratesDeprecatedConfig	▶️ View Recording
AspireUpdateRemovesAppHostPackageVersionFromDirectoryPackagesProps	▶️ View Recording
Banner_DisplayedOnFirstRun	▶️ View Recording
Banner_DisplayedWithExplicitFlag	▶️ View Recording
CreateAndDeployToDockerCompose	▶️ View Recording
CreateAndDeployToDockerComposeInteractive	▶️ View Recording
CreateAndPublishToKubernetes	▶️ View Recording
CreateAndRunAspireStarterProject	▶️ View Recording
CreateAndRunAspireStarterProjectWithBundle	▶️ View Recording
CreateAndRunJsReactProject	▶️ View Recording
CreateAndRunPythonReactProject	▶️ View Recording
CreateAndRunTypeScriptStarterProject	▶️ View Recording
CreateEmptyAppHostProject	▶️ View Recording
CreateStartAndStopAspireProject	▶️ View Recording
CreateStartWaitAndStopAspireProject	▶️ View Recording
CreateTypeScriptAppHostWithViteApp	▶️ View Recording
DescribeCommandResolvesReplicaNames	▶️ View Recording
DescribeCommandShowsRunningResources	▶️ View Recording
DetachFormatJsonProducesValidJson	▶️ View Recording
DoctorCommand_DetectsDeprecatedAgentConfig	▶️ View Recording
DoctorCommand_WithSslCertDir_ShowsTrusted	▶️ View Recording
DoctorCommand_WithoutSslCertDir_ShowsPartiallyTrusted	▶️ View Recording
LogsCommandShowsResourceLogs	▶️ View Recording
PsCommandListsRunningAppHost	▶️ View Recording
PsFormatJsonOutputsOnlyJsonToStdout	▶️ View Recording
SecretCrudOnDotNetAppHost	▶️ View Recording
SecretCrudOnTypeScriptAppHost	▶️ View Recording
StagingChannel_ConfigureAndVerifySettings_ThenSwitchChannels	❌ Upload failed
StopAllAppHostsFromAppHostDirectory	▶️ View Recording
StopAllAppHostsFromUnrelatedDirectory	▶️ View Recording
StopNonInteractiveMultipleAppHostsShowsError	▶️ View Recording
StopNonInteractiveSingleAppHost	❌ Upload failed
StopWithNoRunningAppHostExitsSuccessfully	❌ Upload failed
TypeScriptAppHostWithProjectReferenceIntegration	▶️ View Recording

_{📹 Recordings uploaded automatically from CI run #22794595902}

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

radical requested review from eerhardt and mitchdenny as code owners March 7, 2026 02:29

Copilot AI review requested due to automatic review settings March 7, 2026 02:29

radical marked this pull request as draft March 7, 2026 02:29

Copilot started reviewing on behalf of radical March 7, 2026 02:29 View session

Copilot AI reviewed Mar 7, 2026

View reviewed changes

radical changed the title ~~Test infrastructure cleanup: OuterLoop attribute, skip macOS, IsGithubPullRequest~~ Clean up CI test selection and OuterLoop metadata Mar 7, 2026

radical force-pushed the test-cleanup branch from 64933f5 to 28ae5a9 Compare March 7, 2026 03:05

radical changed the title ~~Clean up CI test selection and OuterLoop metadata~~ Align CI test selection and remove duplicate OuterLoop metadata Mar 7, 2026

radical and others added 3 commits March 6, 2026 22:45

Exclude outerloop and quarantined tests from class enumeration

8008d22

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Make Aspire test macOS selection PR-aware

89bb02c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Prefer OuterloopTest in repository guidance

b7788a1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

radical force-pushed the test-cleanup branch from 28ae5a9 to b7788a1 Compare March 7, 2026 03:45

radical changed the title ~~Align CI test selection and remove duplicate OuterLoop metadata~~ Reduce PR validation jobs and clean up OuterloopTest usage Mar 7, 2026

radical and others added 2 commits March 6, 2026 23:00

Require Docker for built-in starter template test

152c8c6

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Limit QuarantineTools test project scheduling

c497143

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

radical changed the title ~~Reduce PR validation jobs and clean up OuterloopTest usage~~ Reduce PR validation jobs and clean up test scheduling Mar 7, 2026

Fix Oracle EF outerloop CI routing

570921d

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

radical marked this pull request as ready for review March 7, 2026 05:12

radical added the area-engineering-systems infrastructure helix infra engineering repo stuff label Mar 7, 2026

radical requested review from davidfowl and joperezr March 7, 2026 05:13

github-actions bot mentioned this pull request Mar 7, 2026

[pull requests] Review required #13834

Open

radical closed this Mar 7, 2026

radical reopened this Mar 7, 2026

dotnet-policy-service bot added this to the 13.2 milestone Mar 7, 2026

github-actions bot mentioned this pull request Mar 7, 2026

[13.2-burndown] Daily Burndown Report - March 7, 2026 #15039

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce PR validation jobs and clean up test scheduling#15031

Reduce PR validation jobs and clean up test scheduling#15031
radical wants to merge 6 commits intorelease/13.2from
test-cleanup

radical commented Mar 7, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 7, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Mar 7, 2026

Uh oh!

Uh oh!

Copilot AI Mar 7, 2026

Uh oh!

Uh oh!

github-actions bot commented Mar 7, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

radical commented Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Must check before merge

Uh oh!

github-actions bot commented Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Mar 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Mar 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎬 CLI E2E Test Recordings

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

radical commented Mar 7, 2026 •

edited

Loading

github-actions bot commented Mar 7, 2026 •

edited

Loading

github-actions bot commented Mar 7, 2026 •

edited

Loading