Skip to content

Windows CI: Enable more integration tests#38469

Merged
thaJeztah merged 1 commit intomoby:masterfrom
olljanat:win-more-integration-tests
Sep 21, 2020
Merged

Windows CI: Enable more integration tests#38469
thaJeztah merged 1 commit intomoby:masterfrom
olljanat:win-more-integration-tests

Conversation

@olljanat
Copy link
Contributor

@olljanat olljanat commented Jan 1, 2019 •
edited
Loading

- What I did
After seeing too many times that any Docker and Windows updates cannot be installed without testing all features which we need I decided fix the root cause, the fact that most of the integration tests are disabled on Windows CI.

So on this PR I enabled those tests what can be enabled with minimal changes to code.

- How I did it
I started enabling those tests and figured out the fact that work was actually started already on #37715 but (source: #37715 (comment))

However, note that none of the integration tests are actually run with this PR, just the integration-cli tests.

so on #39240 added needed logic to Windows CI that those tests actually run.

Then year 2019 passed and on February I reopened this one and did find out that Windows CI was shown as passed even some tests actually failed so on #40599 I finally fixed that issue.

After getting Windows CI working correctly I noticed that there is acutally much less tests which can be enabled than I was originally thinking because many tests will need more modifications to working on Windows and because of the fact that there is open bugs on Windows #39191 #40998 #41354

- How to verify it
Look the CI log:

[2020-09-19T08:22:36.684Z] --- PASS: TestBuildWithRemoveAndForceRemove (0.02s)
[2020-09-19T08:22:36.684Z]     --- PASS: TestBuildWithRemoveAndForceRemove/successful_build_with_no_removal (12.98s)
[2020-09-19T08:22:36.684Z]     --- PASS: TestBuildWithRemoveAndForceRemove/failed_build_with_remove_and_force_remove (13.00s)
[2020-09-19T08:22:36.684Z]     --- PASS: TestBuildWithRemoveAndForceRemove/failed_build_with_no_removal (13.07s)
[2020-09-19T08:22:36.684Z]     --- PASS: TestBuildWithRemoveAndForceRemove/successful_build_with_remove_and_force_remove (16.16s)
[2020-09-19T08:22:36.684Z]     --- PASS: TestBuildWithRemoveAndForceRemove/successful_build_with_remove (8.19s)
[2020-09-19T08:22:36.684Z]     --- PASS: TestBuildWithRemoveAndForceRemove/failed_build_with_remove (8.25s)

[2020-09-19T08:22:46.658Z] === RUN   TestBuildMultiStageParentConfig
[2020-09-19T08:22:48.835Z] --- PASS: TestBuildMultiStageParentConfig (1.83s)

[2020-09-19T08:22:49.801Z] === RUN   TestBuildMultiStageOnBuild
[2020-09-19T08:23:00.120Z] --- PASS: TestBuildMultiStageOnBuild (10.61s)

[2020-09-19T08:23:00.120Z] === RUN   TestBuildMultiStageLayerLeak
[2020-09-19T08:23:10.445Z] --- PASS: TestBuildMultiStageLayerLeak (9.24s)

[2020-09-19T08:23:15.527Z] === RUN   TestCopyFromContainerPathDoesNotExist
[2020-09-19T08:23:15.985Z] --- PASS: TestCopyFromContainerPathDoesNotExist (0.20s)

[2020-09-19T08:23:15.986Z] === RUN   TestCopyToContainerPathDoesNotExist
[2020-09-19T08:23:15.986Z] --- PASS: TestCopyToContainerPathDoesNotExist (0.22s)

[2020-09-19T08:23:18.628Z] === RUN   TestExec
[2020-09-19T08:23:20.166Z] --- PASS: TestExec (1.76s)

[2020-09-19T08:23:23.242Z] === RUN   TestKillContainer/killing_signal
[2020-09-19T08:23:25.522Z] --- PASS: TestKillContainer (3.37s)
[2020-09-19T08:23:25.522Z]     --- PASS: TestKillContainer/no_signal (1.43s)
[2020-09-19T08:23:25.522Z]     --- PASS: TestKillContainer/non_killing_signal (0.00s)
[2020-09-19T08:23:25.522Z]     --- PASS: TestKillContainer/killing_signal (1.90s)

[2020-09-19T08:23:27.059Z] === RUN   TestNetworkLocalhostTCPNat
[2020-09-19T08:23:28.595Z] --- PASS: TestNetworkLocalhostTCPNat (1.85s)

[2020-09-19T08:23:45.081Z] === RUN   TestResize
[2020-09-19T08:23:46.623Z] --- PASS: TestResize (1.51s)

[2020-09-19T08:24:18.770Z] === RUN   TestRemoveImageOrphaning
[2020-09-19T08:24:19.229Z] --- PASS: TestRemoveImageOrphaning (0.68s)

[2020-09-19T08:24:19.688Z] === RUN   TestTagOfficialNames
[2020-09-19T08:24:19.688Z] --- PASS: TestTagOfficialNames (0.08s)

[2020-09-19T08:24:53.629Z] === RUN   TestVolumesCreateAndList
[2020-09-19T08:24:53.629Z] --- PASS: TestVolumesCreateAndList (0.05s)

[2020-09-19T08:24:53.629Z] === RUN   TestVolumesRemove
[2020-09-19T08:24:53.629Z] --- PASS: TestVolumesRemove (0.06s)

- A picture of a cute animal (not mandatory but encouraged)
image

@olljanat olljanat requested a review from vdemeester as a code owner January 1, 2019 12:24
@olljanat olljanat mentioned this pull request Jan 1, 2019
Closed
7 tasks
@olljanat olljanat closed this Jan 1, 2019
@olljanat olljanat reopened this Feb 27, 2020
@codecov
Copy link

codecov bot commented Feb 27, 2020 •
edited
Loading

Codecov Report

❗ No coverage uploaded for pull request base (master@3967999). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #38469   +/-   ##
=========================================
  Coverage          ?   36.62%           
=========================================
  Files             ?      608           
  Lines             ?    45040           
  Branches          ?        0           
=========================================
  Hits              ?    16495           
  Misses            ?    26265           
  Partials          ?     2280

@olljanat olljanat force-pushed the win-more-integration-tests branch from e212e6b to b145192 Compare February 27, 2020 16:47
@olljanat
Copy link
Contributor Author

olljanat commented Feb 27, 2020

Re-opened and rebased to be able to see if these works better with current versions CI and Windows.

@olljanat olljanat force-pushed the win-more-integration-tests branch 4 times, most recently from 98031c3 to d535ba4 Compare February 27, 2020 19:06
@olljanat
Copy link
Contributor Author

olljanat commented Feb 27, 2020

@thaJeztah FYI. There looks to be actually quite nasty bug on Windows CI as some tests actually fails on win-RS5 but it is still shown as green 😟

@thaJeztah
Copy link
Member

thaJeztah commented Feb 27, 2020

😞 Sounds like this issue I opened a while back; #39576

@olljanat olljanat closed this Apr 28, 2020
@olljanat olljanat reopened this May 28, 2020
@olljanat olljanat force-pushed the win-more-integration-tests branch from d535ba4 to 7633244 Compare May 28, 2020 16:08
@olljanat olljanat requested a review from tianon as a code owner May 28, 2020 17:17
@olljanat olljanat closed this May 29, 2020
@olljanat olljanat reopened this Sep 15, 2020
@olljanat olljanat force-pushed the win-more-integration-tests branch from 37e4039 to d0923cc Compare September 15, 2020 19:43
thaJeztah
thaJeztah reviewed Sep 15, 2020
View reviewed changes
@olljanat olljanat force-pushed the win-more-integration-tests branch 4 times, most recently from d640ce5 to ccdc043 Compare September 16, 2020 16:52
@olljanat olljanat mentioned this pull request Sep 16, 2020
Merged
@olljanat olljanat force-pushed the win-more-integration-tests branch 3 times, most recently from 385661c to dcd2a8a Compare September 17, 2020 13:38
@olljanat olljanat force-pushed the win-more-integration-tests branch 9 times, most recently from 61715ab to 9f20357 Compare September 19, 2020 07:57
@olljanat olljanat changed the title [WIP]: Windows: Enable more integration tests Windows CI: Enable more integration tests Sep 19, 2020
thaJeztah
thaJeztah approved these changes Sep 19, 2020
View reviewed changes
Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

thank you!

@thaJeztah
Copy link
Member

thaJeztah commented Sep 19, 2020

@AkihiroSuda @cpuguy83 ptal

@olljanat olljanat mentioned this pull request Sep 19, 2020
Merged
StefanScherer
StefanScherer approved these changes Sep 20, 2020
View reviewed changes
Copy link
Contributor

@StefanScherer StefanScherer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

cpuguy83
cpuguy83 requested changes Sep 20, 2020
View reviewed changes
@olljanat
Windows: Enable more integration tests
66a37b4 
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
@olljanat olljanat force-pushed the win-more-integration-tests branch from 9f20357 to 66a37b4 Compare September 20, 2020 19:10
cpuguy83
cpuguy83 approved these changes Sep 20, 2020
View reviewed changes
Copy link
Member

@cpuguy83 cpuguy83 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

thaJeztah
thaJeztah approved these changes Sep 21, 2020
View reviewed changes
Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@thaJeztah thaJeztah merged commit 4ce3d71 into moby:master Sep 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cpuguy83 cpuguy83 cpuguy83 approved these changes

@thaJeztah thaJeztah thaJeztah approved these changes

@vdemeester vdemeester Awaiting requested review from vdemeester

@tianon tianon Awaiting requested review from tianon

+1 more reviewer

@StefanScherer StefanScherer StefanScherer approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area/testing platform/windows status/2-code-review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@olljanat @thaJeztah @StefanScherer @cpuguy83 @GordonTheTurtle