Skip to content

[ArPow] Run smoke tests for tarball CI#11023

Merged
dseefeld merged 5 commits intodotnet:release/6.0.1xxfrom
omajid:arpow-ci-smoke-tests
Sep 16, 2021
Merged

[ArPow] Run smoke tests for tarball CI#11023
dseefeld merged 5 commits intodotnet:release/6.0.1xxfrom
omajid:arpow-ci-smoke-tests

Conversation

@omajid
Copy link
Member

@omajid omajid commented Jun 30, 2021 •
edited by MichaelSimons
Loading

@MichaelSimons
Copy link
Member

MichaelSimons commented Jun 30, 2021

Friendly FYI, we are not going to be able to run the smoke tests until all repos are included the tarball.

@omajid
Copy link
Member Author

omajid commented Jun 30, 2021

Friendly FYI, we are not going to be able to run the smoke tests until all repos are included the tarball.

Yup, thanks for that! I tried out a variant of this locally and ran into some "interesting" errors:

$ ./build.sh --run-smoke-test
Found bootstrap SDK 6.0.100-preview.5.21225.11, bootstrap Arcade 6.0.0-beta.21310.5, bootstrap SourceLink 1.0.0
Microsoft (R) Build Engine version 16.10.0-preview-21223-01+117a9cbe2 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/dotnet-tarball/.dotnet/sdk/6.0.100-preview.5.21225.11/MSBuild.dll /bl:source-build-test.binlog /clp:v=m /flp:v=detailed /p:OfflineBuild=true /t:RunSmokeTest /dotnet-tarball/build.proj
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21310.5' at '/dotnet-tarball/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21310.5'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.DotNet.Arcade.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21310.5' at '/dotnet-tarball/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21310.5'
[SourceBuiltSdkResolver] Overriding Microsoft.DotNet.Arcade.Sdk with 'ARCADE'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21310.5' at '/dotnet-tarball/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21310.5'
/dotnet-tarball/Directory.Build.targets(58,7): error MSB4184: The expression "[System.IO.File]::ReadAllText(/dotnet-tarball/ProdConFeed.txt)" cannot be evaluated. Could not find file '/dotnet-tarball/ProdConFeed.txt'. [/dotnet-tarball/build.proj]

MichaelSimons
MichaelSimons reviewed Jun 30, 2021
View reviewed changes
@omajid omajid force-pushed the arpow-ci-smoke-tests branch 2 times, most recently from 77aca14 to 90fe359 Compare July 8, 2021 21:50
@omajid omajid force-pushed the arpow-ci-smoke-tests branch from 90fe359 to e52b34a Compare August 19, 2021 20:41
@omajid
Copy link
Member Author

omajid commented Aug 20, 2021 •
edited
Loading

Now the failure is:

  RID to test: fedora.33-x64
  ls: cannot access '/tarball/artifacts/x64/Release/dotnet-sdk-5.0*': No such file or directory
/tarball/build.proj(105,5): error MSB3073: The command "./smoke-test.sh --minimal --projectOutput --configuration Release --archiveRestoredPackages" exited with code 2.

Which is expected until we can build the entire SDK end-to-end.

@MichaelSimons MichaelSimons changed the base branch from main to release/6.0.1xx August 24, 2021 19:31
@MichaelSimons
Copy link
Member

MichaelSimons commented Aug 24, 2021

I re-targeted this PR to release/6.0.1xx. I also updated the VERSION_PREFIX reference in smoke-test.sh from 5.0 to 6.0. A few tests pass now but fail with:

  starting language C#, type xunit
      running new
  No templates found matching: 'xunit'.
  To list installed templates, run 'dotnet new --list'.
  To search for the templates on NuGet.org, run 'dotnet new xunit --search'.
/repos/installer2/artifacts/tarball/build.proj(105,5): error MSB3073: The command "./smoke-test.sh --minimal --projectOutput --configuration Release --archiveRestoredPackages" exited with code 6.

The xunit template is indeed missing.

@omajid omajid marked this pull request as ready for review August 24, 2021 20:16
@omajid omajid requested a review from a team as a code owner August 24, 2021 20:16
@MichaelSimons
Copy link
Member

MichaelSimons commented Aug 24, 2021 •
edited
Loading

Comparing the source-built 6.0 SDK against the MSFT Preview 6 build and the 5.0 source-build, I see the following templates missing from 6.0 source-build.

ASP.NET Core with React.js and Redux          reactredux           [C#]        Web/MVC/SPA
MSTest Test Project                           mstest               [C#],F#,VB  Test/MSTest
NUnit 3 Test Item                             nunit-test           [C#],F#,VB  Test/NUnit
NUnit 3 Test Project                          nunit                [C#],F#,VB  Test/NUnit
xUnit Test Project                            xunit                [C#],F#,VB  Test/xUnit

@omajid
Copy link
Member Author

omajid commented Aug 25, 2021

As discussed in the meeting earlier, I will let @lbussell and others take over this now. Please ping me if there's anything I can/should do.

@MichaelSimons
Copy link
Member

MichaelSimons commented Aug 26, 2021

[Triage] We think it is important to get some level of smoke tests enabled in CI soon. In order to support that goal, we proposed disabling the tests based on the missing templates and then log a follow up issue to track down why the templates mentioned earlier are missing.

@lbussell lbussell mentioned this pull request Aug 30, 2021
Closed
@lbussell
Copy link
Member

lbussell commented Sep 1, 2021

Current status: smoke tests are failing on web tests. Our source built dotnet won't build the default mvc project generated with dotnet new mvc. The generated project is also different from the one produced by the official preview 7 build. This is possibly due to an incoherent build and might be fixed once our source build commit shas are updated.

@MichaelSimons
Copy link
Member

MichaelSimons commented Sep 9, 2021

@lbussell - the commit shas have been updated. Please merge in the latest to see if this fixes the mvc project issues.

@MichaelSimons
Copy link
Member

MichaelSimons commented Sep 14, 2021

/AZP run

@azure-pipelines
Copy link

azure-pipelines bot commented Sep 14, 2021

Azure Pipelines successfully started running 1 pipeline(s).

@MichaelSimons
Copy link
Member

MichaelSimons commented Sep 15, 2021

This is ready for final review. Once merged, I will open new issues to enable the test templates, F#, and C# blazorwasm.

@dseefeld dseefeld merged commit 448400e into dotnet:release/6.0.1xx Sep 16, 2021
MichaelSimons added a commit to MichaelSimons/installer that referenced this pull request Sep 21, 2021
@omajid @MichaelSimons @lbussell
[ArPow] Run smoke tests for tarball CI (dotnet#11023)
888a486 
* [ArPow] Run smoke tests for tarball CI

Fixes: dotnet/source-build#2298

* Update smoke-tests version reference

* comment out tests with missing templates and temporarily accommodate missing xml files

* Disable f# and blazorwasm tests

Co-authored-by: Michael Simons <msimons@microsoft.com>
Co-authored-by: Logan Bussell <36081148+lbussell@users.noreply.github.com>
This was referenced Sep 21, 2021
Merged
Closed
Closed
Closed
Closed
MichaelSimons added a commit that referenced this pull request Sep 24, 2021
@MichaelSimons
[ArPow] Run smoke tests for tarball CI (#11023) (#12055)
3694b8b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MichaelSimons MichaelSimons MichaelSimons left review comments

@lbussell lbussell Awaiting requested review from lbussell

+2 more reviewers

@dseefeld dseefeld dseefeld approved these changes

@crummel crummel crummel approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@omajid @MichaelSimons @lbussell @dseefeld @crummel