[CI] Add PR validation on macOS by radical · Pull Request #9175 · dotnet/aspire

radical · 2025-05-08T19:01:19Z

This adds PR validation for macOS with github actions. This is at par with Windows tests, with Docker based tests, and tests requiring trusted ssl certificate being disabled.

[tests] Run tests workflow on macos
Adjust RequiresDocker and RequiresSSLCertificate to skip macos
Disable tests failing on macos
- FormatDateTime_WithMilliseconds_NewZealandCulture
- AppBar_Change_Theme_ReloadPage
- ProjectWithNoHTTPSRequiresExplicitOverrideWithEnvironmentVariable
Don't override DOTNET_ROOT when getting list of tests
Skip playground tests on windows and macos

.. and add RunOnGithubActionsMacOS

danmoseley

LGTM feel free to merge

RussKie · 2025-05-08T23:20:32Z

We need to update runsheet runners:

I think this where the change will go:

aspire/eng/TestRunsheetBuilder/TestRunsheetBuilder.targets

Lines 54 to 55 in 35fbfd3

    
           <_TestRunsheetFileNameWindows>$(ArtifactsTmpDir)\$(_TestRunsheet).win.runsheet.json</_TestRunsheetFileNameWindows> 
        
           <_TestRunsheetFileNameLinux>$(ArtifactsTmpDir)\$(_TestRunsheet).linux.runsheet.json</_TestRunsheetFileNameLinux>

aspire/eng/TestRunsheetBuilder/TestRunsheetBuilder.targets

Lines 73 to 74 in 35fbfd3

    
           <_TestRunsheetWindows>{ "project": "$(_TestRunsheet)", "os": "windows-latest", "command": "./eng/build.ps1 $(_TestCommand)" }</_TestRunsheetWindows> 
        
           <_TestRunsheetLinux>{ "project": "$(_TestRunsheet)", "os": "ubuntu-latest", "command": "$(_PreCommand)./eng/build.sh $(_TestCommand)" }</_TestRunsheetLinux>

aspire/eng/TestRunsheetBuilder/TestRunsheetBuilder.targets

Lines 85 to 105 in 35fbfd3

    
               <WriteLinesToFile 
        
                       Condition=" '$(RunOnGithubActionsWindows)' == 'true' and '$(_CreateRunsheet)' == 'true' " 
        
                       File="$(_TestRunsheetFileNameWindows)" 
        
                       Lines="$(_TestRunsheetWindows)" 
        
                       Overwrite="true" 
        
                       WriteOnlyWhenDifferent="true" /> 
        
               <WriteLinesToFile 
        
                       Condition=" '$(RunOnGithubActionsLinux)' == 'true' and '$(_CreateRunsheet)' == 'true' " 
        
                       File="$(_TestRunsheetFileNameLinux)" 
        
                       Lines="$(_TestRunsheetLinux)" 
        
                       Overwrite="true" 
        
                       WriteOnlyWhenDifferent="true" /> 
        
               <!-- 
        
                 On Linux there's a bug in MSBuild, which "normalises" all slashes (see https://github.com/dotnet/msbuild/issues/3468). 
        
                 This is a workaround to replace `/"` with the required `\"`. 
        
                 --> 
        
               <Exec Command="pwsh -Command &quot;(Get-Content -Path '$(_TestRunsheetFileNameWindows)') -replace '/\&quot;', '\\\&quot;' | Set-Content -Path '$(_TestRunsheetFileNameWindows)'&quot; " 
        
                     Condition=" Exists('$(_TestRunsheetFileNameWindows)') and '$(BuildOs)' != 'windows' " /> 
        
               <Exec Command="pwsh -Command &quot;(Get-Content -Path '$(_TestRunsheetFileNameLinux)') -replace '/\&quot;', '\\\&quot;' | Set-Content -Path '$(_TestRunsheetFileNameLinux)'&quot; " 
        
                     Condition=" Exists('$(_TestRunsheetFileNameLinux)') and '$(BuildOs)' != 'windows' " />

tests/Directory.Build.targets

RussKie · 2025-05-09T00:05:18Z

Runsheet: https://github.com/dotnet/aspire/actions/runs/14918382872

RussKie · 2025-05-09T00:05:29Z

~~Outerloop is unhappy, looking~~
Outerloop fixed https://github.com/dotnet/aspire/actions/runs/14918714434

RussKie · 2025-05-09T00:33:18Z

Do we need to run E2E on a mac? Those seem to time-out...

RussKie · 2025-05-09T00:38:18Z

tests/Directory.Build.targets

           Text="%24(ExtractTestClassNamesPrefix) should be set, for example - Aspire.Templates.Tests" />

-    <Exec Command="&quot;$(RunCommand)&quot; --filter-not-trait category=failing --list-tests" ConsoleToMSBuild="true" EnvironmentVariables="DOTNET_ROOT=$(DotNetRoot);DOTNET_ROOT_X86=$(DotNetRoot)x86">
+    <Exec Command="&quot;$(RunCommand)&quot; --filter-not-trait category=failing --list-tests" ConsoleToMSBuild="true">


May you please provide the reason behind this change?

This always overrides DOTNET_ROOT using $(DotnetRoot) which overrides the setting in the environment. So, if you have DOTNET_ROOT set then the parent dotnet process will run with that, but the dotnet .. list-tests will run with this overridden setting.

If you don't agree with this btw, then we can discuss and fix in a follow up.

Fails on azdo with:

You must install or update .NET to run this application. App: /mnt/vss/_work/1/s/artifacts/bin/Aspire.Templates.Tests/Release/net8.0/Aspire.Templates.Tests Architecture: x64 Framework: 'Microsoft.NETCore.App', version '8.0.0' (x64) .NET location: /usr/lib/dotnet The following frameworks were found: 6.0.36 at [/usr/lib/dotnet/shared/Microsoft.NETCore.App] Learn about framework resolution: https://aka.ms/dotnet/app-launch-failed To install missing framework, download: https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=8.0.0&arch=x64&rid=ubuntu.22.04-x64 /mnt/vss/_work/1/s/tests/Directory.Build.targets(39,5): error MSB3073: The command ""/mnt/vss/_work/1/s/artifacts/bin/Aspire.Templates.Tests/Release/net8.0/Aspire.Templates.Tests" --filter-not-trait category=failing --list-tests" exited with code 150. [/mnt/vss/_work/1/s/tests/Aspire.Templates.Tests/Aspire.Templates.Tests.csproj] ##[error]tests/Directory.Build.targets(39,5): error MSB3073: (NETCORE_ENGINEERING_TELEMETRY=Build) The command ""/mnt/vss/_work/1/s/artifacts/bin/Aspire.Templates.Tests/Release/net8.0/Aspire.Templates.Tests" --filter-not-trait category=failing --list-tests" exited with code 150.

We can override DotnetRoot specifically then to have this scenario work too. Though where is this property set?

@Youssef1313

.. are timing out. Issue: dotnet#9198

Fixes ``` /home/runner/work/aspire/aspire/eng/Versions.targets(49,67): error MSB4057: The target "GitInfo" does not exist in the project. [/home/runner/work/aspire/aspire/src/Microsoft.Extensions.ServiceDiscovery.Abstractions/Microsoft.Extensions.ServiceDiscovery.Abstractions.csproj] /home/runner/work/aspire/aspire/eng/Versions.targets(49,67): error MSB4057: The target "GitInfo" does not exist in the project. [/home/runner/work/aspire/aspire/src/Microsoft.Extensions.ServiceDiscovery.Abstractions/Microsoft.Extensions.ServiceDiscovery.Abstractions.csproj] /home/runner/work/aspire/aspire/eng/Versions.targets(49,67): error MSB4057: The target "GitInfo" does not exist in the project. [/home/runner/work/aspire/aspire/src/Tools/ConfigurationSchemaGenerator/ConfigurationSchemaGenerator.csproj] /home/runner/work/aspire/aspire/eng/Versions.targets(49,67): error MSB4057: The target "GitInfo" does not exist in the project. [/home/runner/work/aspire/aspire/src/Microsoft.Extensions.ServiceDiscovery.Abstractions/Microsoft.Extensions.ServiceDiscovery.Abstractions.csproj] /home/runner/work/aspire/aspire/eng/Versions.targets(49,67): error MSB4057: The target "GitInfo" does not exist in the project. [/home/runner/work/aspire/aspire/src/Tools/ConfigurationSchemaGenerator/ConfigurationSchemaGenerator.csproj] ```

This reverts commit 23742a9. Reverting this temporarily till 9.3 branches to avoid blocking the merge due to new flaky macos test failures.

This is reverting "Revert "[CI] Add PR validation on macOS (dotnet#9175)" (dotnet#9208)" commit e5b2802. This is from the original commit message: * [tests] Run tests workflow on macos * Adjust RequiresDocker and RequiresSSLCertificate to skip macos * Disable tests failing on macos * Don't override DOTNET_ROOT when getting list of tests * Skip playground tests on windows and macos .. and add RunOnGithubActionsMacOS * Update runsheet runners * fixup! Update runsheet runners * fixup! Update runsheet runners * Address review feedback from @ russkie * Address review feedback from @ russkie - skip E2E tests on macos as they .. are timing out. Issue: dotnet#9198 * Add missing CI: false for outerloop test runs Fixes ``` /home/runner/work/aspire/aspire/eng/Versions.targets(49,67): error MSB4057: The target "GitInfo" does not exist in the project. [/home/runner/work/aspire/aspire/src/Microsoft.Extensions.ServiceDiscovery.Abstractions/Microsoft.Extensions.ServiceDiscovery.Abstractions.csproj] /home/runner/work/aspire/aspire/eng/Versions.targets(49,67): error MSB4057: The target "GitInfo" does not exist in the project. [/home/runner/work/aspire/aspire/src/Microsoft.Extensions.ServiceDiscovery.Abstractions/Microsoft.Extensions.ServiceDiscovery.Abstractions.csproj] /home/runner/work/aspire/aspire/eng/Versions.targets(49,67): error MSB4057: The target "GitInfo" does not exist in the project. [/home/runner/work/aspire/aspire/src/Tools/ConfigurationSchemaGenerator/ConfigurationSchemaGenerator.csproj] /home/runner/work/aspire/aspire/eng/Versions.targets(49,67): error MSB4057: The target "GitInfo" does not exist in the project. [/home/runner/work/aspire/aspire/src/Microsoft.Extensions.ServiceDiscovery.Abstractions/Microsoft.Extensions.ServiceDiscovery.Abstractions.csproj] /home/runner/work/aspire/aspire/eng/Versions.targets(49,67): error MSB4057: The target "GitInfo" does not exist in the project. [/home/runner/work/aspire/aspire/src/Tools/ConfigurationSchemaGenerator/ConfigurationSchemaGenerator.csproj] ``` * Add macos to the quarantined tests report generator --------- Co-authored-by: Igor Velikorossov <igor.velikorossov@microsoft.com> Co-authored-by: Igor Velikorossov <RussKie@users.noreply.github.com>

radical added 5 commits May 8, 2025 13:39

[tests] Run tests workflow on macos

532094e

Adjust RequiresDocker and RequiresSSLCertificate to skip macos

51199bc

Disable tests failing on macos

5bd89d5

Don't override DOTNET_ROOT when getting list of tests

459f5ae

Skip playground tests on windows and macos

38602b2

.. and add RunOnGithubActionsMacOS

radical requested review from RussKie, danmoseley and joperezr May 8, 2025 19:01

radical requested a review from eerhardt as a code owner May 8, 2025 19:01

github-actions bot added the area-integrations Issues pertaining to Aspire Integrations packages label May 8, 2025

radical added the area-engineering-systems infrastructure helix infra engineering repo stuff label May 8, 2025

danmoseley approved these changes May 8, 2025

View reviewed changes

RussKie removed the area-integrations Issues pertaining to Aspire Integrations packages label May 8, 2025

RussKie and others added 3 commits May 9, 2025 09:44

Update runsheet runners

8e8d091

Merge branch 'main' into macos-ci

011ed56

fixup! Update runsheet runners

91cbe64

RussKie reviewed May 8, 2025

View reviewed changes

tests/Directory.Build.targets Outdated Show resolved Hide resolved

fixup! Update runsheet runners

5181658

RussKie reviewed May 9, 2025

View reviewed changes

radical added 7 commits May 9, 2025 10:09

Merge remote-tracking branch 'origin/main' into macos-ci

1b337b0

Address review feedback from @ russkie

67e94ba

Merge remote-tracking branch 'origin/main' into macos-ci

ebd5001

Address review feedback from @ russkie - skip E2E tests on macos as they

e89a9d5

.. are timing out. Issue: dotnet#9198

Add macos to the quarantined tests report generator

0c45d76

Merge remote-tracking branch 'origin/main' into macos-ci

efae1e6

radical merged commit 23742a9 into dotnet:main May 9, 2025
245 checks passed

radical added a commit to radical/aspire that referenced this pull request May 9, 2025

Revert "[CI] Add PR validation on macOS (dotnet#9175)"

5292f4e

This reverts commit 23742a9. Reverting this temporarily till 9.3 branches to avoid blocking the merge due to new flaky macos test failures.

danmoseley pushed a commit that referenced this pull request May 9, 2025

Revert "[CI] Add PR validation on macOS (#9175)" (#9208)

e5b2802

This reverts commit 23742a9. Reverting this temporarily till 9.3 branches to avoid blocking the merge due to new flaky macos test failures.

radical mentioned this pull request May 13, 2025

[CI] Add PR validation on macOS #9287

Merged

github-actions bot locked and limited conversation to collaborators Jun 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Add PR validation on macOS#9175

[CI] Add PR validation on macOS#9175
radical merged 16 commits intodotnet:mainfrom
radical:macos-ci

radical commented May 8, 2025 •

edited

Loading

Uh oh!

danmoseley left a comment

Uh oh!

RussKie commented May 8, 2025 •

edited

Loading

Uh oh!

Uh oh!

RussKie commented May 9, 2025 •

edited

Loading

Uh oh!

RussKie commented May 9, 2025 •

edited

Loading

Uh oh!

RussKie commented May 9, 2025

Uh oh!

RussKie May 9, 2025

Uh oh!

radical May 9, 2025

Uh oh!

radical May 9, 2025

Uh oh!

radical May 9, 2025

Uh oh!

RussKie May 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

radical commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danmoseley left a comment

Choose a reason for hiding this comment

Uh oh!

RussKie commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

RussKie commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RussKie commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RussKie commented May 9, 2025

Uh oh!

RussKie May 9, 2025

Choose a reason for hiding this comment

Uh oh!

radical May 9, 2025

Choose a reason for hiding this comment

Uh oh!

radical May 9, 2025

Choose a reason for hiding this comment

Uh oh!

radical May 9, 2025

Choose a reason for hiding this comment

Uh oh!

RussKie May 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

radical commented May 8, 2025 •

edited

Loading

RussKie commented May 8, 2025 •

edited

Loading

RussKie commented May 9, 2025 •

edited

Loading

RussKie commented May 9, 2025 •

edited

Loading