Windows CoreFX CI by A-And · Pull Request #18365 · dotnet/coreclr

A-And · 2018-06-07T23:44:37Z

This is connected to #18156

The following commit builds a test host and uses as helper projectt to download and run CoreFX binaries.
There are no Windows-specifics to the implementation other than the batch scripts.

The test lists are included as an example - I've added a command-line parameter to specify a test list according to scenario.

The test urls currently point to CoreRT's CI - again just to be able to test binary download.

cc @RussKeldorph @BruceForstall @jkotas @sergiy-k

A-And · 2018-06-08T01:48:17Z

error MSB4023: Cannot evaluate the item metadata "%(Filename)". The item metadata "%(Filename)" cannot be applied to the path "D:\j\workspace\x64_checked_w---76911707\tests..\Tools\dotnetcli\sdk\NuGetFallbackFolder\runtime.fedora.23-x64.runtime.native.system.security.cryptography.openssl\4.3.0\runtime.fedora.23-x64.runtime.native.system.security.cryptography.openssl.4.3.0.nupkg.sha512". The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.

Seems like we're hitting MAX_PATH in MSBuild.

sergiy-k · 2018-06-08T17:47:52Z

Good job!

A-And · 2018-06-09T02:44:42Z

After enabling all 250+ tests that run on CoreFX test PRs, we're failing some of them because of missing dependencies. Commit with dependencies and test exclusions to follow.

A-And · 2018-06-12T03:33:57Z

All dependencies are now restored correctly. From all executing tests we have ~20 that are failing. The end-to-end test run takes around 3 hours on my local machine.

jkotas · 2018-06-12T03:41:53Z

The end-to-end test run takes around 3 hours on my local machine.

This feels way too long. What is the runtime flavor used for this? Do the tests run in parallel?

A-And · 2018-06-12T17:04:51Z

@jkotas
This is Release x64 builds of both test binaries and CoreCLR.

Xunit itself parallelizes test runs, but the script does not - I ran into the issue of the runner locking assemblies.
The test host itself is 77MB so it would be feasible to copy it across multiple tests and run them, cleaning up as we go.

A-And · 2018-06-13T00:27:50Z

The above comment was a bit off the mark. We can parallelize execution of the runners, but it would have to be managed by the CoreCLR test scripts.

A-And · 2018-06-14T21:05:00Z

The last commit adds a test helper to run tests in parallel. A full run takes around 90 minutes (x64 release) on my local machine. All tests pass with a few fact failures and the notable exception of System.Data.SqlClient.* tests, which hit this issue https://github.com/dotnet/corefx/issues/5252 .

BruceForstall · 2018-06-27T01:20:33Z

Documentation/building/testing-with-corefx.md

 [run-corefx-tests.py](https://github.com/dotnet/coreclr/blob/master/tests/scripts/run-corefx-tests.py) will clone dotnet/corefx and run steps 2-4 above automatically.  It is primarily intended to be run by the dotnet/coreclr CI system, but it might provide a useful reference or shortcut for individuals running the tests locally.

+## Using the built CoreCLR testhost 
+**These instructions are currenly Windows only**


currenly [](start = 25, length = 8)

typo: currenly

BruceForstall · 2018-06-27T01:20:56Z

Documentation/building/testing-with-corefx.md

 [run-corefx-tests.py](https://github.com/dotnet/coreclr/blob/master/tests/scripts/run-corefx-tests.py) will clone dotnet/corefx and run steps 2-4 above automatically.  It is primarily intended to be run by the dotnet/coreclr CI system, but it might provide a useful reference or shortcut for individuals running the tests locally.

+## Using the built CoreCLR testhost 
+**These instructions are currenly Windows only**


currenly [](start = 25, length = 8)

grammar: missing trailing period.

BruceForstall · 2018-06-27T01:23:22Z

Documentation/building/testing-with-corefx.md

+For example to run .NET Core Windows tests from System.Collections.Tests with an x64 Release build of CoreCLR.
+```
+pushd C:\corefx\bin\tests\System.Collections.Tests
+C:\coreclr\bin\tests\Windows_NT.x64.Release\testhost\dotnet.exe .\xunit.console.netcore.exe .\System.Collections.Tests.dll -notrait category=nonnetcoretests -notraint category=nonwindowstests


-notraint [](start = 157, length = 9)

typo? -notraint

BruceForstall · 2018-06-27T01:23:56Z

Documentation/building/testing-with-corefx.md

+```
+
+### CI Script
+CoreCLR has an alternative way to run CoreFX tests, built for PR CI jobs. To run tests against pre-built binaries you can exscute the following from the CoreCLR repo root:


exscute [](start = 122, length = 7)

typo: exscute

BruceForstall · 2018-06-27T01:28:31Z

tests/runtest.cmd

-if /i "%1" == "CoreFXTests"           (exit /b 0)
+if /i "%1" == "CoreFXTests"           (set __CoreFXTests=true&shift&goto Arg_Loop)
+if /i "%1" == "CoreFXTestsAll"        (set __CoreFXTests=true&shift&set __CoreFXTestsRunAllAvailable=true&shift&goto Arg_Loop)
+if /i "%1" == "CoreFXTestList"        (set __CoreFXTests=true&set __CoreFXTestList=%2&shift&shift&goto Arg_Loop)


this shifts twice, so skips the next argument

BruceForstall · 2018-06-27T01:36:05Z

tests/runtest.cmd

+set _CoreFX_TestLogFileName=testResults.xml
+set _CoreFX_TestRunScriptName=CoreCLR_RunTest.cmd
+echo All coreFX tests set to %__CoreFXTestsRunAllAvailable%
+if "%__CoreFXTestsRunAllAvailable%" == "true" ( 


seems like this output is not necessary, since it is an argument in the dotnet command that is shown below

BruceForstall · 2018-06-27T01:37:32Z

tests/runtest.cmd

+
+
+echo Downloading and Running CoreFX Test Binaries
+echo call "%_dotnet%" "%_CoreFXTestUtilitiesOutputPath%\%_CoreFXTestSetupUtilityName%.dll" --clean --outputDirectory "%_CoreFXTestBinariesPath%" --testListJsonPath "%__CoreFXTestList%" --testUrl "!_CoreFXTestRemoteURL!" %_CoreFX_RunCommand% --dotnetPath "%_CoreFXTestHost%\dotnet.exe" --executable %_CoreFXTestExecutable% --logPath %_CoreFXLogsDir%  %_CoreFXTestExecutableArgs% 


All "echo" to the screen should be prefixed by %__MsgPrefix%

BruceForstall · 2018-06-27T01:44:34Z

tests/runtest.cmd

+if not exist %_CoreFXTestHost%\dotnet.exe echo CoreFX test host not found, please run runtest.cmd again && exit /b 1
+
+set /p _CoreFXTestRemoteURL=< "%__ProjectFilesDir%\CoreFX\CoreFXTestListURL.txt"
+if not defined __CoreFXTestList ( set __CoreFXTestList=%__ProjectFilesDir%\CoreFX\TopN.CoreFX.Windows.issues.json )


TopN.CoreFX.Windows.issues.json [](start = 82, length = 31)

regarding "TopN.CoreFX.Windows.issues.json":

should there instead be a single file, with per-OS, per-arch, per-configuration (e.g., stress mode) exclusions (kind of like smarty's .lst file Categories), OR,

should this filename also have arch (e.g., x86, x64), and maybe config (e.g., stress mode) in it?

I would hope we could do #1, because if a test fails in all architectures/OS/stress combinations, I wouldn't want to add it to 15+ different files. Conversely, I hope we can disable on a relatively fine granularity.

1 is completely reasonable (and ideal) - the helper projects are fairly simple and the schema is easily extensible. For the time being I've renamed the files to include architecture.

If this proves to be a viable approach and the python script is fully deprecated, architectures/OS/stress combinations should definitely be included as well.

BruceForstall · 2018-06-27T01:45:30Z

tests/CoreFX/TopN.CoreFX.Unix.issues.json

@@ -0,0 +1,47 @@
+[


Can these files have either (1) a full description at the top of the file format, as a comment, or (2) a link to documentation for this format?

BruceForstall · 2018-06-27T01:47:41Z

tests/src/Common/CoreFX/TestFileSetup/Program.cs

+using System;
+using System.Collections.Generic;
+using System.CommandLine;
+using System.IO;


this (and other source files) need license headers.

And also comments (a file header comment describing the purpose of the code, minimally)

BruceForstall · 2018-06-27T01:52:20Z

tests/src/Common/CoreFX/TestFileSetup/Helpers/TestRunHelper.cs

+            arguments.Append(Path.Combine(testDirectory, Path.GetFileName(testDirectory)));
+            arguments.Append(".rsp");
+            arguments.Append("\"");
+            arguments.Append(" ");


Is this a place where C# interpolated strings would make it easier, e.g.:

arguments.Append($"@\"{Path.Combine(testDirectory, Path.GetFileName(testDirectory))}.rsp\" ");

A-And · 2018-06-27T03:14:07Z

@BruceForstall Thanks for taking a look!
Yes this will be extended to Unix as well. The platform-specific code is very minimal and the Unix implementation mirrors this one closely.

jkotas · 2018-06-27T03:21:37Z

Why only x64 testing, not x86?

I agree that Checked x64 + Checked x86 is a better mix to trigger by default. It provides more coverage than Checked x64 + Release x64 for the same cost.

A-And · 2018-06-27T03:35:34Z

That might add a bit of complexity - we would have to rely on our own xunit console runner, so as to not maintain two versions of CoreFX assemblies in blob storage.
I'm conscious of this PR becoming bigger. Should I add this or add it as a separate PR once this is merged?

jkotas · 2018-06-27T03:43:25Z

Yes, that's fine.

* Add test list CL switch * End-To-End Test Run on Windows * Cleanup * MAX_PATH Workaround * Set Execution directory for CoreFX tests * Add All CoreFX PR Tests * Add test dependencies * Add extra dependencies * Add parallel test execution * Disable OuterLoop tests and System.Data.SqlClient.* tests * Initialize maximum degree of parallelization to Environment.ProcessCount * Remove unnecessary cli option * Update Dependencies * Add "enabled" property to tests * Remove exclusions due to TestUtilities mismatch * Add capability to run all tests for running Helix test lists directly * Refactor build script to build testhost when skipping managed tests * Disable failing System.Threading.Tests.EventWaitHandleTests.Ctor_InvalidMode * Add switch to skip native test build * Add testing documentation * Don't run tests marked as "disabled" when running all available tests * Add switch to build only testhost and remove Core_Root_Stage * Clean up TopN.CoreFX.Windows.issues.json * Refactor build-test.cmd * PR feedback - build pipeline and documentation * PR Feedback - Test Helper headers and comments * Fix buildtesthost option for only building CoreFX test dependencies * Disable intermittently failing test DrawBezier_PointFs Commit migrated from dotnet/coreclr@d3772d9

A-And added 2 commits June 7, 2018 16:37

End-To-End Test Run on Windows

0c6ffa6

Add test list CL switch

6ebf1fe

A-And force-pushed the FXCI branch from 190bebf to 6b0f893 Compare June 8, 2018 00:12

A-And force-pushed the FXCI branch 2 times, most recently from c971e60 to 2bab039 Compare June 8, 2018 03:13

A-And mentioned this pull request Jun 8, 2018

Windows CoreFX CI Staging Commit #18388

Merged

A-And changed the title ~~[WIP] Windows CoreFX CI~~ Windows CoreFX CI Jun 9, 2018

A-And requested a review from BruceForstall June 9, 2018 02:41

A-And changed the title ~~Windows CoreFX CI~~ [WIP] Windows CoreFX CI Jun 9, 2018

A-And added 4 commits June 11, 2018 20:27

Cleanup

8711b78

MAX_PATH Workaround

c8ffd48

Set Execution directory for CoreFX tests

4bc4188

Add All CoreFX PR Tests

b7c1420

A-And force-pushed the FXCI branch 2 times, most recently from 57b6bf7 to 6480390 Compare June 12, 2018 03:31

A-And changed the title ~~[WIP] Windows CoreFX CI~~ Windows CoreFX CI Jun 12, 2018

A-And force-pushed the FXCI branch from 6480390 to f12bf26 Compare June 12, 2018 16:43

Add test dependencies

785f7d3

A-And force-pushed the FXCI branch from f12bf26 to 785f7d3 Compare June 12, 2018 16:59

A-And added 2 commits June 13, 2018 20:40

Add extra dependencies

122b987

Add parallel test execution

78243d8

A-And force-pushed the FXCI branch from 8a435fd to 78243d8 Compare June 14, 2018 16:22

BruceForstall reviewed Jun 27, 2018

View reviewed changes

A-And added 2 commits June 26, 2018 19:59

PR feedback - build pipeline and documentation

23055ff

PR Feedback - Test Helper headers and comments

da81d96

A-And force-pushed the FXCI branch from 6a933f7 to da81d96 Compare June 27, 2018 03:08

Fix buildtesthost option for only building CoreFX test dependencies

3e1fd30

A-And force-pushed the FXCI branch from c96a308 to 3e1fd30 Compare June 27, 2018 18:14

Disable intermittently failing test DrawBezier_PointFs

b9f5503

BruceForstall approved these changes Jun 27, 2018

View reviewed changes

A-And merged commit d3772d9 into dotnet:master Jun 27, 2018

A-And deleted the FXCI branch June 27, 2018 21:43

A-And mentioned this pull request Jun 27, 2018

Re-enable Test Logs and build only test host in CoreFX CI jobs #18677

Merged

BruceForstall mentioned this pull request Jun 27, 2018

Add TargetArch (or similar) as a xunit.netcore.extensions dotnet/buildtools#1979

Closed

This was referenced Jul 2, 2018

CoreFX CI Unix Staging commit #18750

Merged

CoreFX CI Unix #18753

Merged

A-And mentioned this pull request Jul 10, 2018

Exclude failing CoreFx test System.Drawing.Tests.FontTests #18844

Merged



		echo Downloading and Running CoreFX Test Binaries
		echo call "%_dotnet%" "%_CoreFXTestUtilitiesOutputPath%\%_CoreFXTestSetupUtilityName%.dll" --clean --outputDirectory "%_CoreFXTestBinariesPath%" --testListJsonPath "%__CoreFXTestList%" --testUrl "!_CoreFXTestRemoteURL!" %_CoreFX_RunCommand% --dotnetPath "%_CoreFXTestHost%\dotnet.exe" --executable %_CoreFXTestExecutable% --logPath %_CoreFXLogsDir% %_CoreFXTestExecutableArgs%

Conversation

A-And commented Jun 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

A-And commented Jun 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sergiy-k commented Jun 8, 2018

Uh oh!

A-And commented Jun 9, 2018

Uh oh!

A-And commented Jun 12, 2018

Uh oh!

jkotas commented Jun 12, 2018

Uh oh!

A-And commented Jun 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

A-And commented Jun 13, 2018

Uh oh!

A-And commented Jun 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

A-And Jun 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BruceForstall Jun 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

A-And commented Jun 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jkotas commented Jun 27, 2018

Uh oh!

A-And commented Jun 27, 2018

Uh oh!

jkotas commented Jun 27, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

A-And commented Jun 7, 2018 •

edited

Loading

A-And commented Jun 8, 2018 •

edited

Loading

A-And commented Jun 12, 2018 •

edited

Loading

A-And commented Jun 14, 2018 •

edited

Loading

A-And Jun 27, 2018 •

edited

Loading

BruceForstall Jun 27, 2018 •

edited

Loading

A-And commented Jun 27, 2018 •

edited

Loading