Jenkinsfile: add cgroup2 by AkihiroSuda · Pull Request #41925 · moby/moby

AkihiroSuda · 2021-01-23T14:38:53Z

Thanks to @StefanScherer for setting up the Jenkins nodes: #41218 (comment)

Replaces #41918 (Vagrant on GitHub Actions)

TestInspectOomKilledTrue is skipped due to its flakiness on v2 (Flaky test: cgroup2: TestInspectOomKilledTrue fails intermittently #41929)

StefanScherer · 2021-01-23T17:18:34Z

Thanks @AkihiroSuda, the new pipeline stages give us the first feedback. LGTM!

It found some failing tests, eg. https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-41925/6/pipeline/224/#step-227-log-705 is failing.

AkihiroSuda · 2021-01-23T17:27:33Z

=== FAIL: amd64.integration.container TestUpdateMemory (0.26s)

[2021-01-23T16:46:56.634Z]     update_linux_test.go:30: assertion failed: error is not nil: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: process_linux.go:422: setting cgroup config for procHooks process caused: failed to write "209715200" to "/sys/fs/cgroup/system.slice/docker-8068b7b7baf1aba88c19c024a4328536d123b0f175d245d8a3366264810dd36f.scope/memory.swap.max": open /sys/fs/cgroup/system.slice/docker-8068b7b7baf1aba88c19c024a4328536d123b0f175d245d8a3366264810dd36f.scope/memory.swap.max: permission denied: unknown

unable to repro on my Ubuntu 20.10 host. Maybe kernel cmdline needs swapaccount=1? (containers/podman#6365)

StefanScherer · 2021-01-23T17:34:21Z

OK, I'll add swapaccount=1 to the cmdline, rebuild new AMI and rerun this PR.

StefanScherer · 2021-01-23T20:21:06Z

Yes, that did it. The cgroup2 test is green 💚 @AkihiroSuda

AkihiroSuda · 2021-01-23T20:47:01Z

Thanks 🎉

thaJeztah

LGTM

thaJeztah · 2021-01-24T19:49:59Z

s390x and ppc64 failures are unrelated

integration/container/health_test.go

thaJeztah · 2021-01-24T19:53:04Z

@tiborvass ptal

thaJeztah · 2021-01-26T12:54:07Z

Still odd why this one is failing; and why it's loading all frozen images;


[2021-01-26T09:36:20.268Z] === FAIL: ppc64le.integration.build TestBuildUserNamespaceValidateCapabilitiesAreV2 (16.80s)
[2021-01-26T09:36:20.268Z] Loaded image: buildpack-deps:buster
[2021-01-26T09:36:20.268Z] Loaded image: busybox:latest
[2021-01-26T09:36:20.268Z] Loaded image: busybox:glibc
[2021-01-26T09:36:20.268Z] Loaded image: debian:bullseye
[2021-01-26T09:36:20.268Z] Loaded image: hello-world:latest
[2021-01-26T09:36:20.268Z] Loaded image: arm32v7/hello-world:latest
[2021-01-26T09:36:20.268Z]     build_userns_linux_test.go:82: assertion failed: error is not nil: Error response from daemon: reference does not exist: failed to download capabilities image

thaJeztah · 2021-01-26T12:54:48Z

AH!! 🤦

moby/testutil/fixtures/load/frozen.go

Lines 24 to 26 in 46cdcd2

    
           // TODO: This loads whatever is in the frozen image dir, regardless of what 
        
           // images were passed in. If the images need to be downloaded, then it will respect 
        
           // the passed in images

thaJeztah · 2021-01-26T14:31:00Z

Still no idea what exactly is failing there on ppc64;

time="2021-01-26T09:36:19.568390575Z" level=debug msg="Calling POST /v1.41/build?buildargs=null&cachefrom=null&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=&labels=null&memory=0&memswap=0&networkmode=&rm=0&shmsize=0&t=capabilities%3A1.0&target=&ulimits=null&version="
time="2021-01-26T09:36:19.651203419Z" level=debug msg="[BUILDER] Cache miss: [/bin/sh -c setcap CAP_NET_BIND_SERVICE=+eip /bin/sleep]"
time="2021-01-26T09:36:19.651265625Z" level=debug msg="[BUILDER] Command to be executed: [/bin/sh -c setcap CAP_NET_BIND_SERVICE=+eip /bin/sleep]"
time="2021-01-26T09:36:19.677731123Z" level=debug msg="container mounted via layerStore: &{/go/src/github.com/docker/docker/bundles/test-integration/TestBuildUserNamespaceValidateCapabilitiesAreV2/d37f8809d44bf/root/165536.165536/overlay2/f26502707c91bcff3698df15c17b69c762dad5bef9e1332b1f640704a23d1a05/merged 0x106cee640 0x106cee640}" container=7bc2e748d27997e7c6e870f789183000caec0f8e3d3ac2748287d3a8e2ee75e9
time="2021-01-26T09:36:19.687581863Z" level=debug msg="attach: stdout: begin"
time="2021-01-26T09:36:19.687650497Z" level=debug msg="attach: stderr: begin"
time="2021-01-26T09:36:19.688495019Z" level=debug msg="container mounted via layerStore: &{/go/src/github.com/docker/docker/bundles/test-integration/TestBuildUserNamespaceValidateCapabilitiesAreV2/d37f8809d44bf/root/165536.165536/overlay2/f26502707c91bcff3698df15c17b69c762dad5bef9e1332b1f640704a23d1a05/merged 0x106cee640 0x106cee640}" container=7bc2e748d27997e7c6e870f789183000caec0f8e3d3ac2748287d3a8e2ee75e9
time="2021-01-26T09:36:19.688915533Z" level=debug msg="Assigning addresses for endpoint confident_clarke's interface on network bridge"
time="2021-01-26T09:36:19.688939907Z" level=debug msg="RequestAddress(LocalDefault/172.18.0.0/16, <nil>, map[])"
time="2021-01-26T09:36:19.688966926Z" level=debug msg="Request address PoolID:172.18.0.0/16 App: ipam/default/data, ID: LocalDefault/172.18.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:false PrefAddress:<nil> "
time="2021-01-26T09:36:19.691786309Z" level=debug msg="Assigning addresses for endpoint confident_clarke's interface on network bridge"
time="2021-01-26T09:36:19.694883376Z" level=debug msg="Programming external connectivity on endpoint confident_clarke (9c8dd46dcdb58edc28e25881f6301602074c951d190cf5e157dc38d6441b12a3)"
time="2021-01-26T09:36:19.695548587Z" level=debug msg="EnableService 7bc2e748d27997e7c6e870f789183000caec0f8e3d3ac2748287d3a8e2ee75e9 START"
time="2021-01-26T09:36:19.695567762Z" level=debug msg="EnableService 7bc2e748d27997e7c6e870f789183000caec0f8e3d3ac2748287d3a8e2ee75e9 DONE"
time="2021-01-26T09:36:19.697528615Z" level=debug msg="bundle dir created" bundle=/tmp/dxr/d37f8809d44bf/containerd/7bc2e748d27997e7c6e870f789183000caec0f8e3d3ac2748287d3a8e2ee75e9 module=libcontainerd namespace=d37f8809d44bf root=/go/src/github.com/docker/docker/bundles/test-integration/TestBuildUserNamespaceValidateCapabilitiesAreV2/d37f8809d44bf/root/165536.165536/overlay2/f26502707c91bcff3698df15c17b69c762dad5bef9e1332b1f640704a23d1a05/merged
time="2021-01-26T09:36:19.891662161Z" level=debug msg="sandbox set key processing took 85.374579ms for container 7bc2e748d27997e7c6e870f789183000caec0f8e3d3ac2748287d3a8e2ee75e9"
time="2021-01-26T09:36:19.951245790Z" level=debug msg=event module=libcontainerd namespace=d37f8809d44bf topic=/tasks/create
time="2021-01-26T09:36:19.960840533Z" level=debug msg=event module=libcontainerd namespace=d37f8809d44bf topic=/tasks/start
time="2021-01-26T09:36:19.993301192Z" level=debug msg=event module=libcontainerd namespace=d37f8809d44bf topic=/tasks/exit
time="2021-01-26T09:36:20.004716086Z" level=debug msg=event module=libcontainerd namespace=d37f8809d44bf topic=/tasks/delete
time="2021-01-26T09:36:20.004762193Z" level=info msg="ignoring event" container=7bc2e748d27997e7c6e870f789183000caec0f8e3d3ac2748287d3a8e2ee75e9 module=libcontainerd namespace=d37f8809d44bf topic=/tasks/delete type="*events.TaskDelete"
time="2021-01-26T09:36:20.005388506Z" level=debug msg="attach: stdout: end"
time="2021-01-26T09:36:20.005416492Z" level=debug msg="attach: stderr: end"
time="2021-01-26T09:36:20.005493021Z" level=debug msg="attach done"
time="2021-01-26T09:36:20.005990773Z" level=debug msg="Revoking external connectivity on endpoint confident_clarke (9c8dd46dcdb58edc28e25881f6301602074c951d190cf5e157dc38d6441b12a3)"
time="2021-01-26T09:36:20.009290768Z" level=debug msg="DeleteConntrackEntries purged ipv4:0, ipv6:0"
time="2021-01-26T09:36:20.098863807Z" level=debug msg="Releasing addresses for endpoint confident_clarke's interface on network bridge"
time="2021-01-26T09:36:20.098899018Z" level=debug msg="ReleaseAddress(LocalDefault/172.18.0.0/16, 172.18.0.2)"
time="2021-01-26T09:36:20.098930402Z" level=debug msg="Released address PoolID:LocalDefault/172.18.0.0/16, Address:172.18.0.2 Sequence:App: ipam/default/data, ID: LocalDefault/172.18.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65532, Sequence: (0xe0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:3"
time="2021-01-26T09:36:20.135073617Z" level=debug msg="Calling GET /v1.41/images/get?names=capabilities%3A1.0"
time="2021-01-26T09:36:20.135619017Z" level=info msg="Processing signal 'interrupt'"
time="2021-01-26T09:36:20.135773024Z" level=debug msg="daemon configured with a 15 seconds minimum shutdown timeout"
time="2021-01-26T09:36:20.135826484Z" level=debug msg="start clean shutdown of all containers with a 15 seconds timeout..."
time="2021-01-26T09:36:20.136012876Z" level=debug msg="found 0 orphan layers"
time="2021-01-26T09:36:20.136534425Z" level=debug msg="Unix socket /tmp/dxr/d37f8809d44bf/libnetwork/8d71b95ce9b5.sock doesn't exist. cannot accept client connections"
time="2021-01-26T09:36:20.136567593Z" level=debug msg="Cleaning up old mountid : start."
time="2021-01-26T09:36:20.136634002Z" level=error msg="Failed to get event" error="rpc error: code = Unavailable desc = transport is closing" module=libcontainerd namespace=d37f8809d44bf
time="2021-01-26T09:36:20.136686737Z" level=info msg="Waiting for containerd to be ready to restart event processing" module=libcontainerd namespace=d37f8809d44bf
time="2021-01-26T09:36:20.136742232Z" level=warning msg="Error while testing if containerd API is ready" error="rpc error: code = Canceled desc = grpc: the client connection is closing"
time="2021-01-26T09:36:20.136951293Z" level=debug msg="Cleaning up old mountid : done."
time="2021-01-26T09:36:20.137157734Z" level=debug msg="Clean shutdown succeeded"
time="2021-01-26T09:36:20.137179614Z" level=info msg="Daemon shutdown complete"

Possibly the docker build failed, but we don't check for Succesfully built in the output; let me try; #41936

The test fails intermittently on cgroup v2. ``` === FAIL: amd64.integration.container TestInspectOomKilledTrue (0.53s) kill_test.go:171: assertion failed: true (true bool) != false (inspect.State.OOMKilled bool) ``` Tracked in issue 41929 Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

thaJeztah

LGTM

cpuguy83 · 2021-01-29T17:10:21Z

=== RUN   TestBuildUserNamespaceValidateCapabilitiesAreV2
Loaded image: buildpack-deps:buster
Loaded image: busybox:latest
Loaded image: busybox:glibc
Loaded image: debian:bullseye
Loaded image: hello-world:latest
Loaded image: arm32v7/hello-world:latest
--- FAIL: TestBuildUserNamespaceValidateCapabilitiesAreV2 (17.29s)
    build_userns_linux_test.go:82: assertion failed: error is not nil: Error response from daemon: reference does not exist: failed to download capabilities image

thaJeztah · 2021-01-29T17:49:10Z

@cpuguy83 trying to fix that one in #41936

cpuguy83 · 2021-01-29T17:10:21Z

=== RUN   TestBuildUserNamespaceValidateCapabilitiesAreV2
Loaded image: buildpack-deps:buster
Loaded image: busybox:latest
Loaded image: busybox:glibc
Loaded image: debian:bullseye
Loaded image: hello-world:latest
Loaded image: arm32v7/hello-world:latest
--- FAIL: TestBuildUserNamespaceValidateCapabilitiesAreV2 (17.29s)
    build_userns_linux_test.go:82: assertion failed: error is not nil: Error response from daemon: reference does not exist: failed to download capabilities image

thaJeztah · 2021-01-29T17:49:10Z

@cpuguy83 trying to fix that one in #41936

Thanks to Stefan Scherer for setting up the Jenkins nodes. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda · 2021-02-16T06:07:20Z

Can we merge this? Failures on ppc64le and s390x are unrelated.

thaJeztah

still LGTM

thaJeztah · 2021-02-16T08:20:57Z

Yes, let's merge this one

This was referenced Jan 23, 2021

Setup cgroups v2 CI #41218

Closed

add cgroup2 CI (Vagrant on GitHub Actions) #41918

Closed

AkihiroSuda force-pushed the cgroup2ci-jenkins branch 2 times, most recently from b5afd3f to 5c8b685 Compare January 23, 2021 14:48

StefanScherer approved these changes Jan 23, 2021

View reviewed changes

AkihiroSuda added the area/cgroup2 cgroup v2 label Jan 23, 2021

thaJeztah added area/testing status/2-code-review labels Jan 24, 2021

thaJeztah approved these changes Jan 24, 2021

View reviewed changes

thaJeztah reviewed Jan 24, 2021

View reviewed changes

integration/container/health_test.go Outdated Show resolved Hide resolved

alivx approved these changes Jan 25, 2021

View reviewed changes

AkihiroSuda force-pushed the cgroup2ci-jenkins branch from 5c8b685 to 54afede Compare January 26, 2021 09:16

thaJeztah mentioned this pull request Jan 26, 2021

[testing] fix TestBuildUserNamespaceValidateCapabilitiesAreV2 failures #41936

Merged

AkihiroSuda force-pushed the cgroup2ci-jenkins branch from 54afede to 1d4adda Compare January 29, 2021 07:05

thaJeztah approved these changes Jan 29, 2021

View reviewed changes

thaJeztah added status/4-merge and removed status/2-code-review labels Jan 29, 2021

Jenkinsfile: add cgroup2

c23b99f

Thanks to Stefan Scherer for setting up the Jenkins nodes. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda force-pushed the cgroup2ci-jenkins branch from 1d4adda to c23b99f Compare February 1, 2021 05:53

AkihiroSuda added the process/cherry-pick label Feb 2, 2021

thaJeztah added this to the 21.xx milestone Feb 16, 2021

thaJeztah approved these changes Feb 16, 2021

View reviewed changes

thaJeztah merged commit 2834afe into moby:master Feb 16, 2021

thaJeztah mentioned this pull request Feb 16, 2021

[20.10 backport] Jenkinsfile: add cgroup2 #42030

Merged

thaJeztah added process/cherry-picked and removed process/cherry-pick labels Feb 16, 2021

AkihiroSuda mentioned this pull request Feb 16, 2021

cgroup2 check list #40360

Closed

8 tasks

Conversation

AkihiroSuda commented Jan 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StefanScherer commented Jan 23, 2021

Uh oh!

AkihiroSuda commented Jan 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StefanScherer commented Jan 23, 2021

Uh oh!

StefanScherer commented Jan 23, 2021

Uh oh!

AkihiroSuda commented Jan 23, 2021

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Jan 24, 2021

Uh oh!

Uh oh!

thaJeztah commented Jan 24, 2021

Uh oh!

thaJeztah commented Jan 26, 2021

Uh oh!

thaJeztah commented Jan 26, 2021

Uh oh!

thaJeztah commented Jan 26, 2021

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

cpuguy83 commented Jan 29, 2021

Uh oh!

thaJeztah commented Jan 29, 2021

Uh oh!

cpuguy83 commented Jan 29, 2021

Uh oh!

thaJeztah commented Jan 29, 2021

Uh oh!

AkihiroSuda commented Feb 16, 2021

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Feb 16, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

AkihiroSuda commented Jan 23, 2021 •

edited

Loading

AkihiroSuda commented Jan 23, 2021 •

edited

Loading