Skip to content

always partition tests assemblies#43539

Merged
jmarolf merged 1 commit intodotnet:masterfrom
jmarolf:infra/always-partition-tests
Apr 22, 2020
Merged

always partition tests assemblies#43539
jmarolf merged 1 commit intodotnet:masterfrom
jmarolf:infra/always-partition-tests

Conversation

@jmarolf
Copy link
Contributor

@jmarolf jmarolf commented Apr 21, 2020

back in the day there were only a few IDE tests that were worth partitioning. Then came nullable.

Now assemblies like Semantic unit tests are taking 28 minutes in a 52 minute test run.

(Tests that take over 5 mins)

Assemble Name State Execution Time
Microsoft.VisualStudio.LanguageServices.CSharp.UnitTests.dll PASSED 00:05:03.3564739
Microsoft.CodeAnalysis.Workspaces.UnitTests.dll PASSED 00:05:04.6994768
InteractiveHost.UnitTests.dll PASSED 00:05:12.7279281
Microsoft.CodeAnalysis.CSharp.Emit.UnitTests.dll.1 PASSED 00:05:33.0601148
Microsoft.CodeAnalysis.CSharp.CodeStyle.UnitTests.dll PASSED 00:06:13.1561937
Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests.dll.7 PASSED 00:06:19.2043116
Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests.dll.5 PASSED 00:06:41.8132374
Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests.dll.4 PASSED 00:07:03.5482936
Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests.dll.2 PASSED 00:07:14.0802639
Microsoft.CodeAnalysis.VisualBasic.EditorFeatures.UnitTests.dll.1 PASSED 00:07:41.8327451
Microsoft.CodeAnalysis.VisualBasic.EditorFeatures.UnitTests.dll.3 PASSED 00:07:44.4432539
Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests.dll.6 PASSED 00:07:48.4731964
Microsoft.CodeAnalysis.VisualBasic.Semantic.UnitTests.dll PASSED 00:10:52.6523791
Microsoft.CodeAnalysis.CSharp.EditorFeatures.UnitTests.dll.3 PASSED 00:12:24.2802953
Microsoft.CodeAnalysis.VisualBasic.Emit.UnitTests.dll PASSED 00:13:35.3311507
Microsoft.CodeAnalysis.EditorFeatures2.UnitTests.dll.1 PASSED 00:15:39.8984603
Microsoft.CodeAnalysis.CSharp.Emit.UnitTests.dll.2 PASSED 00:22:19.3518864
Microsoft.VisualStudio.LanguageServices.UnitTests.dll.1 PASSED 00:22:32.9940231
Microsoft.CodeAnalysis.CSharp.Semantic.UnitTests.dll PASSED 00:28:35.9769910

With this this change I see a noticeable improvement on my machine

branch Test execution time
master 00:13:38.9999106
this PR 00:10:36.2635440

I'll now verify if this makes a difference on the CI machines

@jmarolf jmarolf requested a review from a team as a code owner April 21, 2020 19:03
@RikkiGibson
Copy link
Member

RikkiGibson commented Apr 21, 2020

⏱️

@jaredpar
Copy link
Member

jaredpar commented Apr 21, 2020

I'll now verify if this makes a difference on the CI machines

Please make sure to do this. The currently partitioning rules are engineered around the processing power of our CI machines. They have generally 4 cores and last time I tried scaling up the partitioning the CI machines fell over and got slower.

@jmarolf
Copy link
Contributor Author

jmarolf commented Apr 21, 2020 •
edited
Loading

@jaredpar Comparing the run from CI on this PR to a random one from master

Test Leg this PR master % improvement
Windows_Desktop_Unit_Tests debug_64 53:17.4105959 56:08.6697293 5.08%
Windows_Desktop_Unit_Tests debug_64 56:15.1579141 58:20.5608026 3.58%
Windows_Desktop_Unit_Tests release_32 41:00.6242238 40:50.7759179 -0.40%
Windows_Desktop_Unit_Tests release_64 45:24.9074712 43:08.1130949 -5.29%
Windows_Desktop_Spanish_Unit_Tests 46:51.6156277 52:51.0471193 11.33%

this appears to be a positive change if you have 4 or more cores

@jmarolf jmarolf merged commit 50d0e54 into dotnet:master Apr 22, 2020
@jmarolf jmarolf deleted the infra/always-partition-tests branch April 22, 2020 02:25
@ghost ghost added this to the Next milestone Apr 22, 2020
@sharwell sharwell modified the milestones: Next, temp, 16.7.P1 Apr 28, 2020
@JoeRobich JoeRobich mentioned this pull request May 27, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jaredpar jaredpar jaredpar approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

16.7.P1

Development

Successfully merging this pull request may close these issues.

4 participants

@jmarolf @RikkiGibson @jaredpar @sharwell