[1.1] Fixes inability to use /dev/null when inside a container by evanphx · Pull Request #3620 · opencontainers/runc

evanphx · 2022-10-02T23:32:24Z

The original code depended on the origin filesystem to have /dev/{block,char} populated. This is done by udev normally and while is very common non-containerized systemd installs, it's very easy to start systemd in a container created by runc itself and not have /dev/{block,char} populated. When this occurs, the following error output is observed:

$ docker run hello-world
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error reopening /dev/null inside container: open /dev/null: operation not permitted: unknown.

/dev/null can't be opened because it was not added to the deviceAllowList, as there was no /dev/char directory. The change here utilizes the fact that when sysfs in in use, there is a /sys/dev/{block,char} that is kernel maintained that we can check.

I'd also recommend having a base allow list that includes /dev/null (1:3) since runc implicitly depends on it.

AkihiroSuda · 2022-10-03T04:43:29Z

What distro needs this fix?

libcontainer/cgroups/systemd/common.go

evanphx · 2022-10-03T17:41:52Z

It's not a specific distro that needs the fix, it was more of a one-off. I was experimenting with running systemd inside a containerd managed container, with docker installed inside the that container (so, nested containerds). The installation of systemd didn't pull in udev (it was ubuntu) and thusly /dev/{block,char} were not created.

thaJeztah · 2022-10-03T18:10:53Z

Looks like this PR was opened against the 1.1 branch; the standard flow is to open against main, and then cherry-pick backports; could you change the PR to tarter "main"?

evanphx · 2022-10-03T18:12:29Z

@thaJeztah The code on main has been reorganized so the patch wouldn't be easily backported. I'm happy to open an additional PR with the fix for main.

This is a forward port of opencontainers#3620 The original code depended on the origin filesystem to have /dev/{block,char} populated. This is done by udev normally and while is very common non-containerized systemd installs, it's very easy to start systemd in a container created by runc itself and not have /dev/{block,char} populated. When this occurs, the following error output is observed: $ docker run hello-world docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error reopening /dev/null inside container: open /dev/null: operation not permitted: unknown. /dev/null can't be opened because it was not added to the deviceAllowList, as there was no /dev/char directory. The change here utilizes the fact that when sysfs in in use, there is a /sys/dev/{block,char} that is kernel maintained that we can check. Signed-off-by: Evan Phoenix <evan@phx.io>

evanphx · 2022-10-03T18:19:06Z

@thaJeztah Went ahead and did the forward port: #3623

thaJeztah · 2022-10-04T13:08:51Z

Thanks! Just making sure that we have all fixes in main (otherwise we may forget porting, and end up with regressions if a release is cut from the main branch 😅)

kolyshkin · 2022-10-05T22:27:44Z

I suggest we discuss this in #3623 first.

This is a forward port of opencontainers#3620 The original code depended on the origin filesystem to have /dev/{block,char} populated. This is done by udev normally and while is very common non-containerized systemd installs, it's very easy to start systemd in a container created by runc itself and not have /dev/{block,char} populated. When this occurs, the following error output is observed: $ docker run hello-world docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error reopening /dev/null inside container: open /dev/null: operation not permitted: unknown. /dev/null can't be opened because it was not added to the deviceAllowList, as there was no /dev/char directory. The change here utilizes the fact that when sysfs in in use, there is a /sys/dev/{block,char} that is kernel maintained that we can check. Signed-off-by: Evan Phoenix <evan@phx.io>

kolyshkin · 2022-10-12T20:42:38Z

#3623 is merged, please backport

evanphx · 2022-10-13T03:20:04Z

@kolyshkin ready to go!

kolyshkin · 2022-10-17T17:23:56Z

CI centos-stream-8 timed out; restarted.

kolyshkin

LGTM

AkihiroSuda

Please use git cherry-pick -x for tracking the main commit

This is a forward port of opencontainers#3620 The original code depended on the origin filesystem to have /dev/{block,char} populated. This is done by udev normally and while is very common non-containerized systemd installs, it's very easy to start systemd in a container created by runc itself and not have /dev/{block,char} populated. When this occurs, the following error output is observed: $ docker run hello-world docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error reopening /dev/null inside container: open /dev/null: operation not permitted: unknown. /dev/null can't be opened because it was not added to the deviceAllowList, as there was no /dev/char directory. The change here utilizes the fact that when sysfs in in use, there is a /sys/dev/{block,char} that is kernel maintained that we can check. Signed-off-by: Evan Phoenix <evan@phx.io> (cherry picked from commit 462e719)

evanphx · 2022-10-20T02:18:13Z

@AkihiroSuda No problem, done!

kolyshkin

LGTM

mprasil · 2023-03-30T08:53:50Z

It's not a specific distro that needs the fix, it was more of a one-off.

Just a follow up on this. @evanphx's fix is actually useful for systems running on top of LXD. (possibly in certain configuration, although I hit this with very standard setup IMO) So this is probably more common than you'd expect.

I am certainly very grateful for this fix. 🙏

This is a forward port of opencontainers/runc#3620 The original code depended on the origin filesystem to have /dev/{block,char} populated. This is done by udev normally and while is very common non-containerized systemd installs, it's very easy to start systemd in a container created by runc itself and not have /dev/{block,char} populated. When this occurs, the following error output is observed: $ docker run hello-world docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error reopening /dev/null inside container: open /dev/null: operation not permitted: unknown. /dev/null can't be opened because it was not added to the deviceAllowList, as there was no /dev/char directory. The change here utilizes the fact that when sysfs in in use, there is a /sys/dev/{block,char} that is kernel maintained that we can check. Signed-off-by: Evan Phoenix <evan@phx.io>

This is a forward port of opencontainers/runc#3620 The original code depended on the origin filesystem to have /dev/{block,char} populated. This is done by udev normally and while is very common non-containerized systemd installs, it's very easy to start systemd in a container created by runc itself and not have /dev/{block,char} populated. When this occurs, the following error output is observed: $ docker run hello-world docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error reopening /dev/null inside container: open /dev/null: operation not permitted: unknown. /dev/null can't be opened because it was not added to the deviceAllowList, as there was no /dev/char directory. The change here utilizes the fact that when sysfs in in use, there is a /sys/dev/{block,char} that is kernel maintained that we can check. Signed-off-by: Evan Phoenix <evan@phx.io> (cherry picked from commit 462e719)

This is a forward port of opencontainers/runc#3620 The original code depended on the origin filesystem to have /dev/{block,char} populated. This is done by udev normally and while is very common non-containerized systemd installs, it's very easy to start systemd in a container created by runc itself and not have /dev/{block,char} populated. When this occurs, the following error output is observed: $ docker run hello-world docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error reopening /dev/null inside container: open /dev/null: operation not permitted: unknown. /dev/null can't be opened because it was not added to the deviceAllowList, as there was no /dev/char directory. The change here utilizes the fact that when sysfs in in use, there is a /sys/dev/{block,char} that is kernel maintained that we can check. Signed-off-by: Evan Phoenix <evan@phx.io> (cherry picked from commit 98b5f8b28cf4d34ff612d69c31692b1a11bb65e7)

This is a forward port of opencontainers/runc#3620 The original code depended on the origin filesystem to have /dev/{block,char} populated. This is done by udev normally and while is very common non-containerized systemd installs, it's very easy to start systemd in a container created by runc itself and not have /dev/{block,char} populated. When this occurs, the following error output is observed: $ docker run hello-world docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error reopening /dev/null inside container: open /dev/null: operation not permitted: unknown. /dev/null can't be opened because it was not added to the deviceAllowList, as there was no /dev/char directory. The change here utilizes the fact that when sysfs in in use, there is a /sys/dev/{block,char} that is kernel maintained that we can check. Signed-off-by: Evan Phoenix <evan@phx.io>

evanphx force-pushed the b-remove-udev-dep branch from fea204e to 4932cf4 Compare October 2, 2022 23:45

AkihiroSuda reviewed Oct 3, 2022

View reviewed changes

libcontainer/cgroups/systemd/common.go Show resolved Hide resolved

evanphx force-pushed the b-remove-udev-dep branch 3 times, most recently from f3a00d3 to a3e87da Compare October 3, 2022 17:47

evanphx mentioned this pull request Oct 3, 2022

Fixes inability to use /dev/null when inside a container #3623

Merged

AkihiroSuda changed the title ~~Fixes inability to use /dev/null when inside a container~~ [1.1] Fixes inability to use /dev/null when inside a container Oct 3, 2022

AkihiroSuda marked this pull request as draft October 3, 2022 23:21

kolyshkin added this to the 1.1.5 milestone Oct 12, 2022

evanphx force-pushed the b-remove-udev-dep branch from a3e87da to 7fa15e0 Compare October 13, 2022 02:58

evanphx marked this pull request as ready for review October 13, 2022 02:59

kolyshkin previously approved these changes Oct 17, 2022

View reviewed changes

AkihiroSuda requested changes Oct 18, 2022

View reviewed changes

evanphx dismissed kolyshkin’s stale review via 3b95828 October 19, 2022 00:06

evanphx force-pushed the b-remove-udev-dep branch from 7fa15e0 to 3b95828 Compare October 19, 2022 00:06

AkihiroSuda approved these changes Oct 20, 2022

View reviewed changes

kolyshkin approved these changes Oct 21, 2022

View reviewed changes

kolyshkin merged commit 974efd2 into opencontainers:release-1.1 Oct 21, 2022

kolyshkin mentioned this pull request Jan 23, 2023

Nvidia GPU devices in systemd will be ignored after 1.1.3 #3671

Closed

ScheererJ mentioned this pull request Mar 17, 2023

Workaround new runc version causing problems. gardener-attic/machine-controller-manager-provider-local#15

Merged

shoenig mentioned this pull request Mar 29, 2023

build(deps): bump github.com/opencontainers/runc from 1.1.4 to 1.1.5 hashicorp/nomad#16712

Merged

Luap99 mentioned this pull request Apr 11, 2023

chore(deps): update dependency containers/automation_images to v20230405 containers/podman#18124

Merged

1 task

This was referenced Apr 25, 2023

[1.1] libct/cg/sd: use systemd version when generating dev props #3845

Merged

libct/cg/sd/v2: rely on systemd to set device access rules #3847

Closed

Conversation

evanphx commented Oct 2, 2022

Uh oh!

AkihiroSuda commented Oct 3, 2022

Uh oh!

Uh oh!

evanphx commented Oct 3, 2022

Uh oh!

thaJeztah commented Oct 3, 2022

Uh oh!

evanphx commented Oct 3, 2022

Uh oh!

evanphx commented Oct 3, 2022

Uh oh!

thaJeztah commented Oct 4, 2022

Uh oh!

kolyshkin commented Oct 5, 2022

Uh oh!

kolyshkin commented Oct 12, 2022

Uh oh!

evanphx commented Oct 13, 2022

Uh oh!

kolyshkin commented Oct 17, 2022

Uh oh!

kolyshkin left a comment

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda left a comment

Choose a reason for hiding this comment

Uh oh!

evanphx commented Oct 20, 2022

Uh oh!

kolyshkin left a comment

Choose a reason for hiding this comment

Uh oh!

mprasil commented Mar 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants