runc: `git bisect start v1.2.0-rc.2 v1.1.0` (manually) by AkihiroSuda · Pull Request #48366 · moby/moby

AkihiroSuda · 2024-08-23T11:07:41Z

Trying to find out the regression commit that is causing the compatibility issue of runc v1.2 RC.

Dockerfile: update to runc v1.2.2 #47666 (thaJeztah)
Update runc to 1.2.0 #48160 (kolyshkin)
[DNM] Test CI with runc 1.2 #48336 (rata)

Full(er) list of test failures:

TestBuildUserNamespaceValidateCapabilitiesAreV2

TestNetworkNat

TestNetworkLocalhostTCPNat

TestBuildSquashParent

TestDiff

TestUsernsCommit

TestReadPluginNoRead/explicitly_enabled_caching

Most failures are related to not getting output from a container run. As I can't repro this locally I'm not sure how to bisect it.

Originally posted by @kolyshkin in #48160 (comment)

🔴 v1.2.0-rc.2
🔴 v1.1.0-974-g1950892f (The (merge) commit to disable dmz by default, although it still compiles dmz by default)
🔴 v1.1.0-941-g7c004d8e w/ runc_nodmz
🔴 v1.1.0-931-gee73091a w/ runc_nodmz
🔴 v1.1.0-929-gf2f16213 w/ runc_nodmz 🔥 causing stdio regression
⚪ v1.1.0-928-g8e1cd2f5 w/ runc_nodmz (failures are similar to v1.1.0-753-g321aa20c)
⚪ v1.1.0-924-g4baaf18c w/ runc_nodmz (failures are similar to v1.1.0-753-g321aa20c)
⚪ v1.1.0-909-gab146f23 w/ runc_nodmz (failures are similar to v1.1.0-753-g321aa20c)
⚪ v1.1.0-843-g823636c3 w/ runc_nodmz (failures are similar to v1.1.0-753-g321aa20c)
⚪ v1.1.0-753-g321aa20c (the last commit before the introduction of dmz; failures are different from OP runc: git bisect start v1.2.0-rc.2 v1.1.0 (manually) #48366 (comment))
🟢 v1.1.0-562-gf8ad20f5
🟢 v1.1.0

(🟢=Good, 🟡=Pending, 🔴=Bad for the same reason as @kolyshkin reported, ⚪=CI fails due to another reason)

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda · 2024-08-23T15:34:23Z

Failure in v1.1.0-753-g321aa20c is different from what was reported in #48160 (comment)

=== RUN   TestNoNewPrivileges/CapabilityRequested=false
    capabilities_linux_test.go:104: test produced invalid error: "writing sync \"procError\": write pipe: file already closed\nexec /bin/cat: operation not permitted", expected "exec /bin/cat: operation not permitted". Stdout:""
--- FAIL: TestNoNewPrivileges (9.93s)
    --- PASS: TestNoNewPrivileges/CapabilityRequested=true (0.33s)
    --- FAIL: TestNoNewPrivileges/CapabilityRequested=false (0.31s)
FAIL

https://github.com/moby/moby/actions/runs/10527465798/job/29171779480?pr=48366

=== FAIL: amd64.integration-cli TestDockerCLIRunSuite/TestRunOOMExitCode (0.21s)
    docker_cli_run_unix_test.go:605: assertion failed: error is not nil: wrong exit code for OOM container: expected 137, got 125 (output: "/usr/local/cli-integration/docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: failed to write \"16777216\": write /sys/fs/cgroup/memory/docker/7dbb71c83095125594eb6892e13cafd6fbac60e0e45c85c2893fbcb7c6071c55/memory.memsw.limit_in_bytes: device or resource busy: unknown.\ntime=\"2024-08-23T15:11:18Z\" level=error msg=\"error waiting for container: context canceled\" \n")
    --- FAIL: TestDockerCLIRunSuite/TestRunOOMExitCode (0.21s)

https://github.com/moby/moby/actions/runs/10527465798/job/29171799694?pr=48366

Co-authored-by: Kir Kolyshkin <kolyshkin@gmail.com> Co-authored-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

rata · 2024-08-23T15:59:44Z

I think you can work-around failures in runc_dmz using EXTRA_BUILDTAGS="runc_nodmz". This will never compile it (it is disabled by default, so you can just avoid compiling it too).

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda · 2024-08-23T16:43:08Z

v1.1.0-843-g823636c3 w/ runc_nodmz fails with the same reasons as v1.1.0-753-g321aa20c, which is different from the OP

=== RUN   TestNoNewPrivileges/CapabilityRequested=false
    capabilities_linux_test.go:104: test produced invalid error: "writing sync procError: write sync: file already closed\nexec /bin/cat: operation not permitted", expected "exec /bin/cat: operation not permitted". Stdout:""
--- FAIL: TestNoNewPrivileges (8.89s)
    --- PASS: TestNoNewPrivileges/CapabilityRequested=true (0.30s)
    --- FAIL: TestNoNewPrivileges/CapabilityRequested=false (0.29s)

https://github.com/moby/moby/actions/runs/10528984279/job/29175849263?pr=48366

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda · 2024-08-24T16:17:42Z

v1.1.0-929-gf2f16213 (opencontainers/runc@f2f1621 init: close internal fds before execve) seems the first commit that is causing the issue reported by @kolyshkin ("Most failures are related to not getting output from a container run.").

test (graphdriver) / integration (ubuntu-20.04)

=== Failed
=== FAIL: amd64.integration.container TestCreateWithCDIDevices (0.87s)
    cdi_test.go:68: assertion failed: [] does not contain FOO=injected

=== FAIL: amd64.integration.container TestDiff (0.34s)
    diff_test.go:28: assertion failed: 
        --- expected
        +++ items
          []container.FilesystemChange(
        - 	{{Kind: s"A", Path: "/foo"}, {Kind: s"A", Path: "/foo/bar"}},
        + 	nil,
          )
        

=== FAIL: amd64.integration.container TestNetworkNat (0.36s)
    nat_test.go:33: assertion failed: error is not nil: dial tcp 172.17.0.3:8080: connect: connection refused

=== FAIL: amd64.integration.plugin.logging TestReadPluginNoRead/explicitly_enabled_caching (1.00s)
    read_test.go:90: assertion failed: strings.TrimSpace(buf.String()) is not "hello world": []
    --- FAIL: TestReadPluginNoRead/explicitly_enabled_caching (1.00s)

=== FAIL: amd64.integration.plugin.logging TestReadPluginNoRead/default (1.03s)
    read_test.go:90: assertion failed: strings.TrimSpace(buf.String()) is not "hello world": []
    --- FAIL: TestReadPluginNoRead/default (1.03s)

=== FAIL: amd64.integration.plugin.logging TestReadPluginNoRead (5.20s)
    read_test.go:93: [d8b21c48ed581] daemon is not started

...

Looks like stdout is closed too early on some condition.

cc @cyphar @kolyshkin @rata @thaJeztah

Its parent commit v1.1.0-928-g8e1cd2f5 (opencontainers/runc@8e1cd2f, init: verify after chdir that cwd is inside the container) still fails due to other issues.

test (graphdriver) / integration (ubuntu-20.04)

=== FAIL: amd64.integration.capabilities TestNoNewPrivileges/CapabilityRequested=false (0.32s)
    capabilities_linux_test.go:104: test produced invalid error: "writing sync procError: write sync: file already closed\nexec /bin/cat: operation not permitted", expected "exec /bin/cat: operation not permitted". Stdout:""
    --- FAIL: TestNoNewPrivileges/CapabilityRequested=false (0.32s)

test (graphdriver) / integration (ubuntu-20.04, rootless)

=== FAIL: amd64.integration.container TestUpdateRestartPolicy (0.35s)
    update_test.go:31: assertion failed: error is not nil: Error response from daemon: Cannot update container a33223f3221da201c20e58724db3bc9b04586d80b305096181f8b46b4cd95fd3: runc did not terminate successfully: exit status 1: cannot set cpu limit: container could not join or create cgroup
        : unknown

...

thaJeztah · 2024-08-24T16:22:12Z

Interesting that commit was part of a CVE fix that also went to the v1.1 branch (reading from my phone, so didn't check if the patch was the same there)

AkihiroSuda · 2024-08-24T16:24:42Z

Let's continue the discussion in:

Moby CI is failing due to stdio issue introduced in f2f16213 init: close internal fds before execve opencontainers/runc#4384

runc v1.1.0-562-gf8ad20f5

dc1c41f

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda changed the title ~~git bisect start v1.2.0-rc.2 v1.1.0 (manually)~~ runc: git bisect start v1.2.0-rc.2 v1.1.0 (manually) Aug 23, 2024

AkihiroSuda force-pushed the runc-bisect branch from 69afe3a to b9b4d81 Compare August 23, 2024 14:18

runc v1.1.0-753-g321aa20c

ad38300

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda force-pushed the runc-bisect branch from b9b4d81 to ad38300 Compare August 23, 2024 14:28

runc v1.1.0-974-g1950892f

ac6f641

Co-authored-by: Kir Kolyshkin <kolyshkin@gmail.com> Co-authored-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda mentioned this pull request Aug 23, 2024

Update runc to 1.2.0 #48160

Draft

runc v1.1.0-843-g823636c3

6c34f5e

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda force-pushed the runc-bisect branch from 9c07e6e to 6c34f5e Compare August 23, 2024 16:20

AkihiroSuda added 2 commits August 24, 2024 01:44

runc v1.1.0-909-gab146f23

fffeadd

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

runc v1.1.0-941-g7c004d8e

9489e1f

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda force-pushed the runc-bisect branch from c020b44 to 9489e1f Compare August 23, 2024 17:36

runc v1.1.0-924-g4baaf18c

dbc4602

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda force-pushed the runc-bisect branch from b203238 to dbc4602 Compare August 23, 2024 18:46

AkihiroSuda added 4 commits August 24, 2024 03:56

runc v1.1.0-932-g8454bbb6

1beb0cb

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

runc v1.1.0-931-gee73091a

93e931f

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

runc v1.1.0-929-gf2f16213

647cdea

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

runc v1.1.0-928-g8e1cd2f5

cf3c867

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda mentioned this pull request Aug 24, 2024

Blockers for v1.2.0 opencontainers/runc#4114

Closed

21 tasks

AkihiroSuda mentioned this pull request Aug 24, 2024

Moby CI is failing due to stdio issue introduced in f2f16213 init: close internal fds before execve opencontainers/runc#4384

Closed

AkihiroSuda closed this Aug 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runc: `git bisect start v1.2.0-rc.2 v1.1.0` (manually)#48366

runc: `git bisect start v1.2.0-rc.2 v1.1.0` (manually)#48366
AkihiroSuda wants to merge 11 commits intomoby:masterfrom
AkihiroSuda:runc-bisect

AkihiroSuda commented Aug 23, 2024 •

edited

Loading

Uh oh!

AkihiroSuda commented Aug 23, 2024

Uh oh!

rata commented Aug 23, 2024

Uh oh!

AkihiroSuda commented Aug 23, 2024

Uh oh!

AkihiroSuda commented Aug 24, 2024

Uh oh!

thaJeztah commented Aug 24, 2024

Uh oh!

AkihiroSuda commented Aug 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

AkihiroSuda commented Aug 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AkihiroSuda commented Aug 23, 2024

Uh oh!

rata commented Aug 23, 2024

Uh oh!

AkihiroSuda commented Aug 23, 2024

Uh oh!

AkihiroSuda commented Aug 24, 2024

Uh oh!

thaJeztah commented Aug 24, 2024

Uh oh!

AkihiroSuda commented Aug 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AkihiroSuda commented Aug 23, 2024 •

edited

Loading