[Carry 39595] Tailor CI for ARM, skip legacy integration test. by thaJeztah · Pull Request #39678 · moby/moby

thaJeztah · 2019-08-06T10:48:46Z

carry of #39595 Tailor CI for ARM, skip legacy integration test
closes #39595 Tailor CI for ARM, skip legacy integration test
fixes #39479 Jenkins CI for arm64

- What I did
This PR is trying to make a minimal test set (without legacy integration test cases) for ARM.
Introduce a new environment variable SKIP_LEGACY_INTEGRATION_TEST to control whether legacy integration test cases are ignored or not.

- How I did it

Introduce a new environment variable SKIP_LEGACY_INTEGRATION_TEST, default value is "no".
If it's set to "yes" or anything rather than "no", legacy integration suites will be skipped.
In arm CI, SKIP_LEGACY_INTEGRATION_TEST is set to "yes".
CI for other architecture's are not impacted.

- How to verify it
Run "make test-integration" on an ARM machine.

- Description for the changelog

See the discussion in issue: #39479
Backgroud: CI on arm (32 and 64 bit) architecture took too long time to finish. So the CI for it was stopped. Now I am trying to bring it back, and try to set a minimal set in it to save time.

- A picture of a cute animal (not mandatory but encouraged)

thaJeztah · 2019-08-06T13:43:29Z

Whoop! That worked;

thaJeztah · 2019-08-06T13:44:52Z

Looks like performance on that machine isn't terrible (see it has lots of ram as well); let me try how long running everything takes (just for fun)

From the docker info output;

Kernel Version: 4.10.0-38-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: aarch64
CPUs: 96
Total Memory: 125.9GiB

We should probably decide if it should run on every PR (if we have a limited number of machines on ARM currently, might want to only run it on master by default)

AWS machines:

Kernel Version: 4.15.0-1044-aws
Operating System: Ubuntu 16.04.6 LTS
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 7.524GiB

thaJeztah · 2019-08-06T13:46:46Z

I'm still a bit on the fence if we should name this arm64 instead of aarch64 (arm64 might be slightly clearer)

michael2012z · 2019-08-06T13:59:16Z

I vote for arm64, it sounds more understandable than aarch64.

thaJeztah · 2019-08-06T14:38:56Z

I vote for arm64, it sounds more understandable than aarch64.

Agreed; I think I'll change it (I'll let this one running for now, but can push an updated version after it has completed)

thaJeztah · 2019-08-06T15:58:23Z

PowerPC here failed on https://ci.docker.com/public/job/moby/job/PR-39678/6/execution/node/208/log/

Which is tracked as a flaky test; #23516

[2019-08-06T15:03:11.839Z] ----------------------------------------------------------------------
[2019-08-06T15:03:11.839Z] FAIL: docker_api_swarm_node_test.go:75: DockerSwarmSuite.TestAPISwarmNodeDrainPause

I posted details in #23516 (comment)

thaJeztah · 2019-08-06T18:04:49Z

OK, so running everything is indeed a bit too long

thaJeztah · 2019-08-07T20:46:36Z

No idea what this Janky failure was; https://ci.docker.com/public/blue/organizations/jenkins/moby/detail/PR-39678/7/pipeline


[2019-08-06T22:30:11.181Z] ----------------------------------------------------------------------
[2019-08-06T22:30:11.181Z] FAIL: docker_cli_run_test.go:2459: DockerSuite.TestRunModeUTSHost
[2019-08-06T22:30:11.181Z] 
[2019-08-06T22:30:11.181Z] assertion failed: 
[2019-08-06T22:30:11.181Z] Command:  /usr/local/cli/docker run busybox readlink /proc/self/ns/uts
[2019-08-06T22:30:11.181Z] ExitCode: 125
[2019-08-06T22:30:11.181Z] Error:    exit status 125
[2019-08-06T22:30:11.181Z] Stdout:   
[2019-08-06T22:30:11.181Z] Stderr:   /usr/local/cli/docker: Error response from daemon: ttrpc: client shutting down: ttrpc: closed: unknown.
[2019-08-06T22:30:11.181Z] time="2019-08-06T22:30:10Z" level=error msg="error waiting for container: context canceled" 
[2019-08-06T22:30:11.181Z] 
[2019-08-06T22:30:11.181Z] 
[2019-08-06T22:30:11.181Z] Failures:
[2019-08-06T22:30:11.181Z] ExitCode was 125 expected 0
[2019-08-06T22:30:11.181Z] Expected no error

thaJeztah · 2019-08-07T21:22:45Z

Removed the restriction to run on packet machines only; now will pick whatever aarch64 worker is available (packet, or auto-scaling
a1.xlarge arm64 machines on AWS)

michael2012z · 2019-08-08T07:53:30Z

I can't manage to reproduce the failure in Janky:
[2019-08-07T22:32:20.683Z] FAIL: docker_cli_swarm_test.go:1333: DockerSwarmSuite.TestSwarmClusterRotateUnlockKey

Similar failure was seen long ago in #34384 and #36696.

thaJeztah · 2019-08-08T12:21:37Z

Yes, that test is still known to be flaky (TestSwarmClusterRotateUnlockKey)

michael2012z · 2019-08-08T14:17:25Z

Great, now everything looks good.

thaJeztah · 2019-08-08T14:26:15Z

@psftw @andrewhsu PTAL

michael2012z · 2019-08-14T10:07:14Z

Hi, all
It's been silent for days on this PR. :)
Is it going to be merged?

thaJeztah · 2019-08-14T19:51:04Z

I'll do one more rebase (and I think there were some changes in the other stages/platforms I'll have to sync)

I'll check for people to review this 👍

thaJeztah · 2019-09-21T14:30:57Z

rebased

michael2012z · 2019-10-08T12:25:02Z

Hi, @thaJeztah and other maintainers,
Regarding the test case TestTemplatedConfig failure, it passed with the early commits made on Aug/6, it was also OK with the rebase on Aug/8. It began to appear stably since the rebase on Aug/15. I am thinking if it was triggered by some PR merged during Aug/8 and Aug/15.
Maybe identifying the commit that triggered the problem (by "binary-searching" all the PRs during the 7 days) is an easier way to move forward.
I am willing to fix the problem, but it was not reproducible in my local tests. May I get some temporary permission to apply Jenkinsfile in my PR, then I can help?

Signed-off-by: Michael Zhao <michael.zhao@arm.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Also switch aarch64 to use overlay2 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Pick whatever is available; packet worker, or auto-scaling a1.xlarge arm64 machines on AWS Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

michael2012z · 2019-10-09T02:12:52Z

:), interesting, all test cases passed in this run.
Thank you for rebasing the code.

thaJeztah · 2019-10-09T13:26:09Z

oh! I see I forgot to leave a comment (sorry!)

interesting; all I did was indeed rebase this PR (and sync some minor changes that were added in other stages). So either something was fixed in our codebase, or the arm machines were updated, and it was an issues with the machines themselves 🤔

Either way; happy to see this green. Let's see if I can get people to review this

@tiborvass @andrewhsu @kolyshkin PTAL

thaJeztah · 2019-10-09T13:29:58Z

One thing I notice is that these machines are running Ubuntu 16.04 (other stages are running 18.04); something we may want to update in a follow up (I would need to check if there's 18.04 arm64 machines)

andrewhsu

LGTM

cpuguy83

LGTM

thaJeztah · 2019-10-10T19:12:13Z

Whoop, merged! Thank you for getting the ball rolling on this one, and for your patience @michael2012z

michael2012z · 2019-10-10T23:28:14Z

Thank you @thaJeztah for carrying my original PR, making a lot of improvement, and patient rebasing again and again.
And thanks to all other maintainers for reviewing and merging.

thaJeztah added status/2-code-review area/testing process/cherry-pick platform/arm labels Aug 6, 2019

thaJeztah requested a review from tianon as a code owner August 6, 2019 10:48

GordonTheTurtle added the status/0-triage label Aug 6, 2019

thaJeztah force-pushed the carry_39595_tailor_arm_ci branch 2 times, most recently from 37ca01e to 1097180 Compare August 6, 2019 10:55

thaJeztah removed the status/0-triage label Aug 6, 2019

thaJeztah mentioned this pull request Aug 6, 2019

Tailor CI for ARM, skip legacy integration test. #39595

Closed

thaJeztah force-pushed the carry_39595_tailor_arm_ci branch from 1097180 to 5d00eec Compare August 6, 2019 11:05

thaJeztah changed the title ~~Tailor CI for ARM, skip legacy integration test.~~ [Carry 39595] Tailor CI for ARM, skip legacy integration test. Aug 6, 2019

thaJeztah mentioned this pull request Aug 6, 2019

Flaky test: TestAPISwarmNodeDrainPause #23516

Open

thaJeztah force-pushed the carry_39595_tailor_arm_ci branch from 571fec7 to 13f3032 Compare August 6, 2019 21:52

thaJeztah force-pushed the carry_39595_tailor_arm_ci branch from 13f3032 to bf4feab Compare August 7, 2019 21:21

thaJeztah force-pushed the carry_39595_tailor_arm_ci branch from 4ecc247 to 4be066f Compare September 21, 2019 14:30

michael2012z and others added 9 commits October 8, 2019 18:37

Test to enable CI on aarch64.

af86580

Signed-off-by: Michael Zhao <michael.zhao@arm.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Tailor CI for ARM, skip legacy integration test.

48b06a2

Signed-off-by: Michael Zhao <michael.zhao@arm.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Jenkinsfile: aarch64: move stage inside parallel group

402c7b1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Jenkinsfile: aarch64: sync stage with other stages

86e0c5a

Also switch aarch64 to use overlay2 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Jenkinsfile: aarch64: split into stages, add "print info" unit-tests

58d57c7

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Jenkinsfile: rename aarch64 to arm64

9d5361d

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Jenkinsfile: aarch64: don't restrict to packet workers only

14ea1f6

Pick whatever is available; packet worker, or auto-scaling a1.xlarge arm64 machines on AWS Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Jenkinsfile: aarch64: sync with latest changes

a0d670e

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Jenkinsfile: aarch64: use new labels to select agents

eda98ad

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah force-pushed the carry_39595_tailor_arm_ci branch from 4be066f to eda98ad Compare October 8, 2019 16:49

andrewhsu approved these changes Oct 9, 2019

View reviewed changes

cpuguy83 approved these changes Oct 9, 2019

View reviewed changes

tiborvass approved these changes Oct 10, 2019

View reviewed changes

tiborvass merged commit d31633b into moby:master Oct 10, 2019

thaJeztah deleted the carry_39595_tailor_arm_ci branch October 10, 2019 18:22

This was referenced Oct 22, 2019

[18.09 backport] Jenkinsfile updates, and add junit.xml for "integration" docker-archive/engine#395

Merged

[19.03 backport][Carry 39595] Tailor CI for ARM, skip legacy integration test docker-archive/engine#413

Closed

thaJeztah added process/cherry-picked and removed process/cherry-pick labels Oct 22, 2019

thaJeztah mentioned this pull request Oct 22, 2019

[WIP][18.09 backport][Carry 39595] Tailor CI for ARM, skip legacy integration test docker-archive/engine#414

Closed

thaJeztah mentioned this pull request Jan 25, 2020

[19.03 backport][Carry 39595] Tailor CI for ARM, skip legacy integration test #40414

Closed

thaJeztah added this to the 20.03.0 milestone Apr 2, 2020

Conversation

thaJeztah commented Aug 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah commented Aug 6, 2019

Uh oh!

thaJeztah commented Aug 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah commented Aug 6, 2019

Uh oh!

michael2012z commented Aug 6, 2019

Uh oh!

thaJeztah commented Aug 6, 2019

Uh oh!

thaJeztah commented Aug 6, 2019

Uh oh!

thaJeztah commented Aug 6, 2019

Uh oh!

thaJeztah commented Aug 7, 2019

Uh oh!

thaJeztah commented Aug 7, 2019

Uh oh!

michael2012z commented Aug 8, 2019

Uh oh!

thaJeztah commented Aug 8, 2019

Uh oh!

michael2012z commented Aug 8, 2019

Uh oh!

thaJeztah commented Aug 8, 2019

Uh oh!

michael2012z commented Aug 14, 2019

Uh oh!

thaJeztah commented Aug 14, 2019

Uh oh!

thaJeztah commented Sep 21, 2019

Uh oh!

michael2012z commented Oct 8, 2019

Uh oh!

michael2012z commented Oct 9, 2019

Uh oh!

thaJeztah commented Oct 9, 2019

Uh oh!

thaJeztah commented Oct 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewhsu left a comment

Choose a reason for hiding this comment

Uh oh!

cpuguy83 left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Oct 10, 2019

Uh oh!

michael2012z commented Oct 10, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

thaJeztah commented Aug 6, 2019 •

edited

Loading

thaJeztah commented Aug 6, 2019 •

edited

Loading

thaJeztah commented Oct 9, 2019 •

edited

Loading