support DisableCgroup, DisableApparmor, RestrictOOMScoreAdj #970

AkihiroSuda · 2018-11-09T11:01:43Z

Add following config for supporting "rootless" mode

DisableCgroup: disable cgroup
DisableApparmor: disable Apparmor
RestrictOOMScoreAdj: restrict the lower bound of OOMScoreAdj

Signed-off-by: Akihiro Suda suda.akihiro@lab.ntt.co.jp

A quick way to test this PR is to use Usernetes v20181109.0 ( https://github.com/rootless-containers/usernetes )

$ ./run.sh default-containerd
$ ./kubectl.sh run -it --rm --image alpine foo

cc @giuseppe for consistency with CRI-O implementation (https://github.com/kubernetes-sigs/cri-o/blob/2accad9fab352c445bb4c414d9d3c8cdf351c524/server/rootless.go could be simplified as in this PR with the latest runc IIUC)

mikebrow

See comments, I missed containerd/containerd#2766 going in, sry. If we want this back ported to earlier releases we'll want to keep these changes you made and walk them back as deprecated.. then do a follow up commit putting them in as runc options vs cri root options.

pkg/config/config.go

Random-Liu · 2018-11-12T18:46:19Z

In OCI spec, both AppArmor and OOMScoreAdj is a Process level configuration. I think we can match that, so I'm fine with a daemon level NoAppArmor and RestrictOOMScoreAdj option.

As for NoCgroup, although it is Linux specific, but it is still different from the existing runtime options, e.g. "NoPivot":

Options like "NoPivot" or "SystemdCgroup" are actually runc specific (they are runc flags), and they are actually changing runc's behavior.
However, NoCgroup (NoAppArmor and RestrictOOMScoreAdj as well) is changing the cri plugin's behavior - it changes how cri should config the OCI spec. Given so, I think it is a cri daemon level configuration. As for the fact that it is Linux specific, we need to have different groups of daemon level configurations for Windows and Linux in the future anyway.

@AkihiroSuda Are these options only useful for rootless? If that is the case, can we add a Rootless section in the config? So that it is clear to the user that this is for rootless.

pkg/server/container_create.go

mikebrow · 2018-11-12T22:52:46Z

@AkihiroSuda Are these options only useful for rootless? If that is the case, can we add a Rootless section in the config? So that it is clear to the user that this is for rootless.

I was sort of thinking these are runtime config options required for filtering oci spec generation to make sure it will run rootless. But I don't see why we would want to filter/force all pods on the host to be rootless. Is that the goal here? Push rootless up the stack or to support running the shims/runc/containerd as rootless? Switching to rootless at the host/containerd/cri level seems like a pretty big switch.

It might be better to return error if an apparmor profile is specified and is disabled, than to filter the profile option.

How will these changes effect kata support?

Random-Liu · 2018-11-13T01:16:28Z

Switching to rootless at the host/containerd/cri level seems like a pretty big switch.

If I understand correctly, "Rootless" means run containerd as a non-root user. If that is the case, it is a daemon level thing to me.

It is "rootless" containerd, not running a non-root container.

AkihiroSuda · 2018-11-13T02:54:21Z

In this context I mean running everything (runc, containerd, kubelet) as a non-root user. I can explain the details in person and show you demo during KubeCon Shanghai

mikebrow · 2018-11-13T05:16:58Z

ok. Even kubelet, interesting.

mikebrow · 2018-11-13T05:28:34Z

Maybe what's needed here is a single check to see if we are running as root or not then use that as flag to filter out any attempt to do something that is known to fail when not running as root. For example, if !runningAsRoot && apparmorProfile != "" return err.."apparmor is not supported when running as non-root..."

AkihiroSuda · 2018-11-13T05:52:41Z

It might be complicating, if we want to support automatic detection, maybe we should change the variable type from bool to string, and reserve "auto" for the future implementation (Either way, probably we should allow setting NoCgroups=false even in rootless mode, because cgroups can be used when the fs is chowned)

AkihiroSuda · 2018-11-22T05:32:12Z

@Random-Liu @mikebrow any thought?

mikebrow · 2018-11-23T17:53:04Z

Suggest: using a string for a rootless option, "auto" should be default. False would make sure we are running as root otherwise throw error on init, true would make sure we are not running as root otherwise error on init. Agree to implementing NoCgroup, NoAppArmor, and RestrictOOMScoreAdj as daemon level options. See @Random-Liu's comment above regarding:

move implementation of these options to the place the option was originally configured, e.g. OOMScoreAdj is configured in setOCILinuxResource

Thoughts?

AkihiroSuda · 2018-12-03T08:23:56Z

thanks, updated

mikebrow

/LGTM

pkg/server/container_create.go

pkg/server/helpers.go

pkg/config/config.go

pkg/server/container_create.go

Random-Liu · 2018-12-19T19:06:54Z

pkg/server/container_create.go

 	g.SetProcessNoNewPrivileges(securityContext.GetNoNewPrivs())

 	// TODO(random-liu): [P1] Set selinux options (privileged or not).



Add unit test cases for this logic.

Can we cover this in an integrated test (in a follow-up PR)?

It is current hard to write an integration test with different containerd configurations.

I think a unit test would be easier. You can check existing unit tests, it should be simple to add a similar one. :)

pkg/server/helpers.go

AkihiroSuda · 2018-12-26T06:53:29Z

updated

pkg/server/helpers.go

pkg/server/container_create.go

pkg/server/helpers.go

Random-Liu · 2019-01-02T18:23:53Z

pkg/server/container_create.go

 	g.SetProcessNoNewPrivileges(securityContext.GetNoNewPrivs())

 	// TODO(random-liu): [P1] Set selinux options (privileged or not).



It is current hard to write an integration test with different containerd configurations.

I think a unit test would be easier. You can check existing unit tests, it should be simple to add a similar one. :)

Random-Liu · 2019-01-02T18:25:33Z

LGTM other than the comments.

AkihiroSuda · 2019-01-02T20:03:07Z

thanks, updated

Add following config for supporting "rootless" mode * DisableCgroup: disable cgroup * DisableApparmor: disable Apparmor * RestrictOOMScoreAdj: restrict the lower bound of OOMScoreAdj Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>

Random-Liu

/lgtm

k8s-ci-robot added the size/M label Nov 9, 2018

This was referenced Nov 9, 2018

[UMBRELLA] patch submission status rootless-containers/usernetes#42

Closed

Rootless mode containerd/containerd#2766

Merged

mikebrow suggested changes Nov 9, 2018

View reviewed changes

pkg/config/config.go Outdated Show resolved Hide resolved

pkg/config/config.go Show resolved Hide resolved

Random-Liu assigned Random-Liu and mikebrow Nov 9, 2018

Random-Liu reviewed Nov 12, 2018

View reviewed changes

pkg/server/container_create.go Show resolved Hide resolved

AkihiroSuda force-pushed the rootless branch from 15f0196 to db254b4 Compare December 3, 2018 07:37

k8s-ci-robot added size/L and removed size/M labels Dec 3, 2018

AkihiroSuda force-pushed the rootless branch 2 times, most recently from 4d8509c to 21f1d7a Compare December 3, 2018 08:05

k8s-ci-robot added the lgtm label Dec 10, 2018

mikebrow approved these changes Dec 10, 2018

View reviewed changes

Random-Liu reviewed Dec 19, 2018

View reviewed changes

Random-Liu reviewed Dec 20, 2018

View reviewed changes

pkg/server/helpers.go Outdated Show resolved Hide resolved

AkihiroSuda force-pushed the rootless branch from 21f1d7a to 2c75686 Compare December 26, 2018 06:48

k8s-ci-robot removed the lgtm label Dec 26, 2018

AkihiroSuda changed the title ~~support NoCgroup,NoApparmor, RestrictOOMScoreAdj~~ support DisableCgroup, DisableApparmor, RestrictOOMScoreAdj Dec 26, 2018

AkihiroSuda force-pushed the rootless branch from 2c75686 to 9b6f2fb Compare December 26, 2018 06:57

Random-Liu reviewed Jan 2, 2019

View reviewed changes

AkihiroSuda force-pushed the rootless branch from 9b6f2fb to db25f26 Compare January 2, 2019 20:02

AkihiroSuda force-pushed the rootless branch from db25f26 to cd8231a Compare January 2, 2019 20:12

k8s-ci-robot added the lgtm label Jan 3, 2019

Random-Liu reviewed Jan 3, 2019

View reviewed changes

Random-Liu merged commit 0fa8668 into containerd:master Jan 3, 2019

AkihiroSuda mentioned this pull request Jun 4, 2019

KEP: New cgroup driver: "none" (for Rootless) kubernetes/enhancements#1084

Closed

AkihiroSuda mentioned this pull request Aug 14, 2024

Remove disable_cgroup from CRI config containerd/containerd#10594

Merged

		g.SetProcessNoNewPrivileges(securityContext.GetNoNewPrivs())

		// TODO(random-liu): [P1] Set selinux options (privileged or not).

support DisableCgroup, DisableApparmor, RestrictOOMScoreAdj #970

support DisableCgroup, DisableApparmor, RestrictOOMScoreAdj #970

Uh oh!

Conversation

AkihiroSuda commented Nov 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mikebrow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Random-Liu commented Nov 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

mikebrow commented Nov 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Random-Liu commented Nov 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AkihiroSuda commented Nov 13, 2018

Uh oh!

mikebrow commented Nov 13, 2018

Uh oh!

mikebrow commented Nov 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AkihiroSuda commented Nov 13, 2018

Uh oh!

AkihiroSuda commented Nov 22, 2018

Uh oh!

mikebrow commented Nov 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AkihiroSuda commented Dec 3, 2018

Uh oh!

mikebrow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Random-Liu Dec 19, 2018

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda Dec 26, 2018

Choose a reason for hiding this comment

Uh oh!

Random-Liu Jan 2, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AkihiroSuda commented Dec 26, 2018

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Random-Liu Jan 2, 2019

Choose a reason for hiding this comment

Uh oh!

Random-Liu commented Jan 2, 2019

Uh oh!

AkihiroSuda commented Jan 2, 2019

Uh oh!

Random-Liu left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

AkihiroSuda commented Nov 9, 2018 •

edited

Loading

Random-Liu commented Nov 12, 2018 •

edited

Loading

mikebrow commented Nov 12, 2018 •

edited

Loading

Random-Liu commented Nov 13, 2018 •

edited

Loading

mikebrow commented Nov 13, 2018 •

edited

Loading

mikebrow commented Nov 23, 2018 •

edited

Loading