[20.10 backport] seccomp: add support for "clone3" syscall in default policy by tianon · Pull Request #42836 · moby/moby

tianon · 2021-09-09T18:35:27Z

This is a backport of 9f6b562 (#42681, see also #42680), adapted to avoid the refactoring that happened in d927397 (#42005).

fixes #42963
fixes #42876

tianon · 2021-09-09T20:00:09Z

Doh, build_userns_linux_test.go:73: assertion failed: error is not nil: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall clone3: requested action matches default action of filter: unknown looks very related. 😞

thaJeztah · 2021-09-09T20:41:00Z

hmmmm yes. ISTR there was a recent change in runc related to that (at least, I recall a PR about ignoring defaults)

thaJeztah · 2021-09-09T20:43:53Z

Yup; fix for that is in v1.0.2. But that seems to indicate we may not need this patch / backport? opencontainers/runc#3173

tianon · 2021-09-10T23:28:51Z

Now that I'm reading that error message with a fresh eye, I realize it's complaining that SCMP_ACT_ERRNO is the default action, which ... yep, what of it? The errnoRet is different?? 😕

tianon · 2021-09-10T23:29:29Z

~~That makes me wonder if the "fix" in opencontainers/runc#3129 is actually correct? Skipping rules like this doesn't feel like the right answer.~~

Scratch that; looking at the implementation there, I see it does take errnoRet into account.

tianon · 2021-09-10T23:48:54Z

I think I found what was wrong. 🤞

tianon · 2021-09-11T00:47:31Z

Got it -- the only failure on the last run was gofmt, which I've corrected, so this one should be good.

tianon · 2021-09-13T14:49:11Z

FWIW, @lucaskanashiro tested this and reported in https://bugs.launchpad.net/cloud-images/+bug/1943049/comments/25 that it does fix the problem 😄

This is a backport of 9f6b562, adapted to avoid the refactoring that happened in d927397. Original commit message is as follows: > If no seccomp policy is requested, then the built-in default policy in > dockerd applies. This has no rule for "clone3" defined, nor any default > errno defined. So when runc receives the config it attempts to determine > a default errno, using logic defined in its commit: > > opencontainers/runc@7a8d716 > > As explained in the above commit message, runc uses a heuristic to > decide which errno to return by default: > > [quote] > The solution applied here is to prepend a "stub" filter which returns > -ENOSYS if the requested syscall has a larger syscall number than any > syscall mentioned in the filter. The reason for this specific rule is > that syscall numbers are (roughly) allocated sequentially and thus newer > syscalls will (usually) have a larger syscall number -- thus causing our > filters to produce -ENOSYS if the filter was written before the syscall > existed. > [/quote] > > Unfortunately clone3 appears to one of the edge cases that does not > result in use of ENOSYS, instead ending up with the historical EPERM > errno. > > Latest glibc (2.33.9000, in Fedora 35 rawhide) will attempt to use > clone3 by default. If it sees ENOSYS then it will automatically > fallback to using clone. Any other errno is treated as a fatal > error. Thus when docker seccomp policy triggers EPERM from clone3, > no fallback occurs and programs are thus unable to spawn threads. > > The clone3 syscall is much more complicated than clone, most notably its > flags are not exposed as a directly argument any more. Instead they are > hidden inside a struct. This means that seccomp filters are unable to > apply policy based on values seen in flags. Thus we can't directly > replicate the current "clone" filtering for "clone3". We can at least > ensure "clone3" returns ENOSYS errno, to trigger fallback to "clone" > at which point we can filter on flags. Signed-off-by: Tianon Gravi <admwiggin@gmail.com> Co-authored-by: Daniel P. Berrangé <berrange@redhat.com>

tianon · 2021-09-13T15:58:40Z

Re-reviewing this, it felt silly to still have the createSpecsSyscall function which is only invoked once (and with an unused argument to boot, because I didn't notice in my refactoring 😂), so I've updated again to remove that completely.

thaJeztah

LGTM

gotmax23 · 2021-09-27T06:09:21Z

Hi,

Can you please consider merging this change and expediting a release containing the fix? This clone3 issue has caused a lot of problems over the past couple months.

Thanks you for your work. I apologize if I'm being too pushy.

-- Maxwell 😀

Following commit migrates CI to latest stable `f35`. * Make sure we are using latest docker: Since support for `clone3` was added in default seccomp profile was backported via moby/moby#42836. * Bump podman to `v3.4.0` for `f35` * New versions of `clang-check` adds a check for `-Wunused-but-set-variable` so mark variables with `__attribute__ ((unused))` where its needed. Signed-off-by: Aditya Rajan <arajan@redhat.com>

Following commit migrates CI to latest stable `f35`. * Make sure we are using latest docker: Since support for `clone3` was added in default seccomp profile was backported via moby/moby#42836. * Bump podman to `v3.4.0` for `f35` and skip newly added tests which are not supported in this env. * New versions of `clang-check` adds a check for `-Wunused-but-set-variable` so mark variables with `__attribute__ ((unused))` where its needed. Signed-off-by: Aditya Rajan <arajan@redhat.com>

Following commit migrates CI to latest stable `f35`. * Make sure we are using latest docker: Since support for `clone3` was added in default seccomp profile was backported via moby/moby#42836. * Bump podman to `v3.4.0` for `f35` and skip newly added tests which are not supported in this env. Signed-off-by: Aditya Rajan <arajan@redhat.com>

tianon · 2021-11-08T18:13:12Z

@zhsj brought https://buildd.debian.org/status/fetch.php?pkg=docker.io&arch=mips64el&ver=20.10.10%2Bdfsg1-1&stamp=1636015943&raw=0 to my attention in which TestUnmarshalDefaultProfile fails on mips64le because ENOSYS has a different numeric value (because of course it does, right? we can't possibly have nice things)

=== FAIL: profiles/seccomp TestUnmarshalDefaultProfile (0.19s)
    seccomp_test.go:68: assertion failed: 
        --- expected.Syscalls
        +++ profile.Syscalls
          []*seccomp.Syscall{
          	... // 14 identical elements
          	&{Names: {"clone"}, Action: "SCMP_ACT_ALLOW", Args: {&{Value: 2114060288, Op: "SCMP_CMP_MASKED_EQ"}}, Excludes: {Caps: {"CAP_SYS_ADMIN"}, Arches: {"s390", "s390x"}}},
          	&{Names: {"clone"}, Action: "SCMP_ACT_ALLOW", Args: {&{Index: 1, Value: 2114060288, Op: "SCMP_CMP_MASKED_EQ"}}, Comment: "s390 parameter ordering for clone is different", ...},
          	&{
          		Name:     "",
          		Names:    {"clone3"},
          		Action:   "SCMP_ACT_ERRNO",
        - 		ErrnoRet: &89,
        + 		ErrnoRet: &38,
          		Args:     {},
          		Comment:  "",
          		... // 2 identical fields
          	},
          	&{Names: {"reboot"}, Action: "SCMP_ACT_ALLOW", Args: {}, Includes: {Caps: {"CAP_SYS_BOOT"}}, ...},
          	&{Names: {"chroot"}, Action: "SCMP_ACT_ALLOW", Args: {}, Includes: {Caps: {"CAP_SYS_CHROOT"}}, ...},
          	... // 8 identical elements
          }

I think this test will probably fail on MIPS arches on the version that made it into master, too (for the same reason).

Edit: I've raised this further up the chain at opencontainers/runtime-spec#1087 (comment) now

tianon · 2021-11-08T18:15:16Z

On the plus side, at least this only affects MIPS* and sparc64? 😬

$ git grep -inE 'ENOSYS[[:space:]]+=' | grep linux | column -t
unix/zerrors_linux_386.go:588:       ENOSYS  =  syscall.Errno(0x26)
unix/zerrors_linux_amd64.go:588:     ENOSYS  =  syscall.Errno(0x26)
unix/zerrors_linux_arm.go:594:       ENOSYS  =  syscall.Errno(0x26)
unix/zerrors_linux_arm64.go:585:     ENOSYS  =  syscall.Errno(0x26)
unix/zerrors_linux_mips.go:591:      ENOSYS  =  syscall.Errno(0x59)
unix/zerrors_linux_mips64.go:591:    ENOSYS  =  syscall.Errno(0x59)
unix/zerrors_linux_mips64le.go:591:  ENOSYS  =  syscall.Errno(0x59)
unix/zerrors_linux_mipsle.go:591:    ENOSYS  =  syscall.Errno(0x59)
unix/zerrors_linux_ppc.go:646:       ENOSYS  =  syscall.Errno(0x26)
unix/zerrors_linux_ppc64.go:650:     ENOSYS  =  syscall.Errno(0x26)
unix/zerrors_linux_ppc64le.go:650:   ENOSYS  =  syscall.Errno(0x26)
unix/zerrors_linux_riscv64.go:575:   ENOSYS  =  syscall.Errno(0x26)
unix/zerrors_linux_s390x.go:650:     ENOSYS  =  syscall.Errno(0x26)
unix/zerrors_linux_sparc64.go:639:   ENOSYS  =  syscall.Errno(0x5a)

Edit: I've raised this further up the chain at opencontainers/runtime-spec#1087 (comment) now

thaJeztah · 2021-11-09T10:20:58Z

Ahm.. yes, so we could adjust the test and use a different default.json for that architecture, or (bit of a hack), make the rule conditional, and used a hard-coded value for mips and sparc64. Looks like for clone, we also had to use a conditional rule;

moby/profiles/seccomp/default_linux.go

Lines 613 to 619 in 2b70006

    
           Comment: "s390 parameter ordering for clone is different", 
        
           Includes: &Filter{ 
        
           	Arches: []string{"s390", "s390x"}, 
        
           }, 
        
           Excludes: &Filter{ 
        
           	Caps: []string{"CAP_SYS_ADMIN"}, 
        
           },

thaJeztah · 2021-11-09T10:21:49Z

But I guess the "portable" way would be to use a string for this, and have the conversion done by the runtime that sets the actual profile

…MIPS The value of ENOSYS differs between MIPS and non-MIPS architectures. While this is not problematic for the embedded seccomp profile, it prevents the profile from being saved as a portable JSON file that can be used for both architectures. To work around this situation, we include conditional rules for both arches. and hard-code the value for ENOSYS in both. For more details, refer to moby#42836 (comment) and opencontainers/runtime-spec#1087 (comment) A test was added, which can be run with: go test --tags=seccomp -run TestLoadConditionalClone3 -v ./profiles/seccomp/ === RUN TestLoadConditionalClone3 === RUN TestLoadConditionalClone3/clone3_default_amd64 === RUN TestLoadConditionalClone3/clone3_default_mips === RUN TestLoadConditionalClone3/clone3_cap_sys_admin_amd64 === RUN TestLoadConditionalClone3/clone3_cap_sys_admin_mips --- PASS: TestLoadConditionalClone3 (0.01s) --- PASS: TestLoadConditionalClone3/clone3_default_amd64 (0.00s) --- PASS: TestLoadConditionalClone3/clone3_default_mips (0.00s) --- PASS: TestLoadConditionalClone3/clone3_cap_sys_admin_amd64 (0.00s) --- PASS: TestLoadConditionalClone3/clone3_cap_sys_admin_mips (0.00s) PASS ok github.com/docker/docker/profiles/seccomp 0.015s Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Fedora containers fail with QEMU 6.1.0 and Docker 20.10.9. It can be worked around with option '--security-opt seccomp=unconfined', but that is not allowed on GitHub Actions. Anyway, it is fixed in Docker 20.10.10. * moby/moby#42963 * moby/moby#42836

glibc 2.34 uses the clone3 syscall, which is not part of the seccomp filters that moby ships on older versions. While as a workaround you might be able to run containers with `--privileged`, it's the better call to just run a more recent Docker runtime. References: - docker/buildx#772 - moby/buildkit#2379 - moby/moby#42836 - NixOS/nixpkgs#170900

tianon mentioned this pull request Sep 9, 2021

seccomp: add support for "clone3" syscall in default policy #42681

Merged

tianon force-pushed the 20.10-clone3 branch from 16c40ac to 9cd6936 Compare September 9, 2021 18:39

tianon changed the title ~~seccomp: add support for "clone3" syscall in default policy~~ [20.10 backport] seccomp: add support for "clone3" syscall in default policy Sep 9, 2021

thaJeztah added this to the 20.10.9 milestone Sep 9, 2021

thaJeztah added area/security/seccomp status/2-code-review labels Sep 9, 2021

tianon marked this pull request as draft September 10, 2021 23:33

tianon force-pushed the 20.10-clone3 branch from eee326b to 0a709f7 Compare September 10, 2021 23:48

tianon force-pushed the 20.10-clone3 branch from 0a709f7 to df18e28 Compare September 11, 2021 00:45

tianon marked this pull request as ready for review September 11, 2021 00:47

AkihiroSuda approved these changes Sep 11, 2021

View reviewed changes

tianon force-pushed the 20.10-clone3 branch from df18e28 to 567c01f Compare September 13, 2021 15:57

thaJeztah approved these changes Sep 16, 2021

View reviewed changes

AkihiroSuda mentioned this pull request Sep 27, 2021

Zypper Core Dump on OpenSUSE Tumbleweed (clone3 EPERM) #42876

Closed

thaJeztah mentioned this pull request Sep 28, 2021

[v0.9] vendor: update docker to support seccomp clone3 moby/buildkit#2379

Merged

icy mentioned this pull request Sep 29, 2021

Update dpkg tests for Debian 11 (bullseye) icy/pacapt#207

Merged

lslezak mentioned this pull request Sep 30, 2021

Docker incompatibility with glibc-2.34 actions/runner-images#4193

Closed

8 tasks

ldorau mentioned this pull request Oct 1, 2021

common: move OpenSUSE_Tumbleweed build to Nightly_Experimental pmem/rpma#1301

Merged

amezin mentioned this pull request Oct 1, 2021

Run tests on Fedora 35 ddterm/gnome-shell-extension-ddterm#85

Closed

flouthoc mentioned this pull request Nov 5, 2021

tests: Migrate to f35 | Use latest docker with seccomp profiles supporting clone3. containers/crun#771

Merged

tianon mentioned this pull request Nov 8, 2021

seccomp: allow to override default errno return code opencontainers/runtime-spec#1087

Merged

thaJeztah mentioned this pull request Nov 9, 2021

profiles/seccomp: use conditional (hard-coded) errNoRet for MIPS/non-MIPS #43005

Closed

thaJeztah mentioned this pull request Nov 10, 2021

[Bug] MacOs overlay/overlay2 sendfile syscall in Alpine Linux #42998

Closed

nmeyerhans mentioned this pull request Jan 20, 2022

[Bug] - Couldn't resolve host name amazonlinux/amazon-linux-2023#80

Closed

benesch mentioned this pull request Apr 26, 2022

deps: require Python 3.9+ and "dev" dependency classes MaterializeInc/materialize#12017

Merged

1 task

blitz mentioned this pull request Apr 29, 2022

[21.11] docker: 2.10.9 -> 20.10.14 (for glibc 2.34 compatiblity!) NixOS/nixpkgs#170900

Merged

13 tasks

thespad mentioned this pull request May 6, 2022

Java errors in latest version v4.3.1-ls52 linuxserver/docker-nzbhydra2#20

Closed

flokli mentioned this pull request Jun 8, 2022

docs/src/how-to/install/dependencies.rst: require Docker >= 20.10.14 wireapp/wire-server#2462

Merged

13 tasks

dbarrosop mentioned this pull request Jun 11, 2022

Startup exception in 0.2.x releases nhost/hasura-storage#93

Closed

jet-go mentioned this pull request Jul 13, 2022

Upgrade prometheus version tkestack/kvass#102

Open

rodnymolina mentioned this pull request Nov 30, 2022

Installing nix package manager in sandbox - Operation not permitted error nestybox/sysbox#607

Open

streamer45 mentioned this pull request Jan 13, 2023

[MM-49667] Run recording job with security opts mattermost/calls-offloader#15

Merged

avsm mentioned this pull request Mar 24, 2023

eio_linux: add subprocess support ocaml-multicore/eio#472

Merged

2 tasks

vasiliy-ul mentioned this pull request Mar 31, 2023

virt-operator pod crash becasue of securityContext.seccompProfile: type: RuntimeDefault kubevirt/kubevirt#9540

Closed

azat mentioned this pull request May 2, 2023

🐳 Update Ubuntu Image ClickHouse/ClickHouse#46784

Merged

jannic mentioned this pull request May 17, 2023

thread 'main' panicked at 'called Result::unwrap() on an Err value : Operation not permitted fanout/condure#12

Closed

Felixoid mentioned this pull request Nov 6, 2024

Upgrade clickhouse-server and keeper base images ClickHouse/ClickHouse#71505

Merged

21 tasks

Conversation

tianon commented Sep 9, 2021 • edited by thaJeztah Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tianon commented Sep 9, 2021

Uh oh!

thaJeztah commented Sep 9, 2021

Uh oh!

thaJeztah commented Sep 9, 2021

Uh oh!

tianon commented Sep 10, 2021

Uh oh!

tianon commented Sep 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tianon commented Sep 10, 2021

Uh oh!

tianon commented Sep 11, 2021

Uh oh!

tianon commented Sep 13, 2021

Uh oh!

tianon commented Sep 13, 2021

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

gotmax23 commented Sep 27, 2021

Uh oh!

tianon commented Nov 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tianon commented Nov 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah commented Nov 9, 2021

Uh oh!

thaJeztah commented Nov 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

tianon commented Sep 9, 2021 •

edited by thaJeztah

Loading

tianon commented Sep 10, 2021 •

edited

Loading

tianon commented Nov 8, 2021 •

edited

Loading

tianon commented Nov 8, 2021 •

edited

Loading