Allow for a read-only "/proc/sys/net". #47769

robmry · 2024-04-26T18:58:23Z

- What I did

If dockerd runs on a host with a read-only /proc/sys/net filesystem, it isn't able to enable or disable IPv6 on network interfaces when attaching a container to a network (including initial networks during container creation).

In release 26.0.2, a read-only /proc/sys/net meant container creation failed in all cases. Now, the disable_ipv6 setting is only modified if necessary and, if IPv6 can't be disabled when it needs to be, an environment variable can be used to tell dockerd to proceed anyway.

Fixes #47751

The plan is to remove the environment variable once it's easier to enable IPv6, probably in release 27.0 - at that point, on a system where IPv6 can't be disabled on an interface, IPv6 will have to be explicitly allowed in the network configuration. #47773.

- How I did it

Don't attempt to enable/disable IPv6 on an interface if it's already set appropriately.

If it's not possible to enable IPv6 when it's needed, just log (because that's what libnetwork has always done if IPv6 is disabled in the kernel).

If it's not possible to disable IPv6 when it needs to be disabled, refuse to create the container and raise an error that suggests setting environment variable "DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1", to tell the daemon it's ok to ignore the problem.

- How to verify it

New regression test - environment variable DOCKER_TEST_RO_DISABLE_IPV6 is used to simulate failure to modify the IPv6 setting.

- Description for the changelog

- Resolve an issue preventing container creation on hosts with a read-only `/proc/sys/net` filesystem. If IPv6 cannot be disabled on an interface due to this, either disable IPv6 by default on the host or ensure `/proc/sys/net` is read-write. Otherwise, start dockerd with `DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1` to bypass the error.
> [!NOTE]
> The `DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE` is added as a temporary fix and will be phased out in a future major release after simplifying the IPv6 enablement process.

thaJeztah · 2024-04-29T12:36:02Z

libnetwork/osl/namespace_linux.go

+				log.G(context.TODO()).WithError(err).Infof(
+					"Cannot disable IPv6 on container interface %s but DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1, continuing.", iface)


Would it be useful to use a field for the interface for these logs? e.g. something like;

Suggested change

log.G(context.TODO()).WithError(err).Infof(

"Cannot disable IPv6 on container interface %s but DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1, continuing.", iface)

log.G(context.TODO()).WithFields(log.Fields{

"error": err,

"interface": iface,

}).Info("Cannot disable IPv6 on container interface, but DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1, continuing")

(same for the other logs changed in this PR)

Done - I'm not sure how useful it is to include the interface name really, it'll probably be in err and it's a name we invented rather than something that'll be meaningful to the user. But, it was there before.

Yeah, maybe it's not. My thinking of making it a structured field was that (imo) it's a good practice, and if we're consistent, it could allow for logs to be correlated ("here's logs related to interface=xxxxxx").

thaJeztah · 2024-04-29T14:18:05Z

libnetwork/osl/namespace_linux.go

+				logger.Warnf("Cannot enable IPv6 on container interface, continuing.")
+			} else if os.Getenv("DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE") == "1" {
+				// TODO(robmry) - remove this escape hatch for https://github.com/moby/moby/issues/47751
+				//   If the "/proc" file exists but isn't writable, we can't disable IPv6, which is
+				//   https://github.com/moby/moby/security/advisories/GHSA-x84c-p2g9-rqv9 ... so,
+				//   the user is required to override the error (or configure IPv6, or disable IPv6
+				//   by default in the OS, or make the "/proc" file writable). Once it's possible
+				//   to enable IPv6 without having to configure IPAM etc, the env var should be
+				//   removed. Then the user will have to explicitly enable IPv6 if it can't be
+				//   disabled on the interface.
+				logger.Infof("Cannot disable IPv6 on container interface but DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1, continuing.")
+			} else {
+				logger.Errorf("Cannot disable IPv6 on container interface. Set env var DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1 to ignore.")


nit (not a blocker, but linters could start to warn about this); looks like we're no longer doing formatting so these could all be without f;

Suggested change

logger.Warnf("Cannot enable IPv6 on container interface, continuing.")

} else if os.Getenv("DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE") == "1" {

// TODO(robmry) - remove this escape hatch for https://github.com/moby/moby/issues/47751

// If the "/proc" file exists but isn't writable, we can't disable IPv6, which is

// https://github.com/moby/moby/security/advisories/GHSA-x84c-p2g9-rqv9 ... so,

// the user is required to override the error (or configure IPv6, or disable IPv6

// by default in the OS, or make the "/proc" file writable). Once it's possible

// to enable IPv6 without having to configure IPAM etc, the env var should be

// removed. Then the user will have to explicitly enable IPv6 if it can't be

// disabled on the interface.

logger.Infof("Cannot disable IPv6 on container interface but DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1, continuing.")

} else {

logger.Errorf("Cannot disable IPv6 on container interface. Set env var DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1 to ignore.")

logger.Warn("Cannot enable IPv6 on container interface, continuing.")

} else if os.Getenv("DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE") == "1" {

// TODO(robmry) - remove this escape hatch for https://github.com/moby/moby/issues/47751

// If the "/proc" file exists but isn't writable, we can't disable IPv6, which is

// https://github.com/moby/moby/security/advisories/GHSA-x84c-p2g9-rqv9 ... so,

// the user is required to override the error (or configure IPv6, or disable IPv6

// by default in the OS, or make the "/proc" file writable). Once it's possible

// to enable IPv6 without having to configure IPAM etc, the env var should be

// removed. Then the user will have to explicitly enable IPv6 if it can't be

// disabled on the interface.

logger.Info("Cannot disable IPv6 on container interface but DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1, continuing.")

} else {

logger.Error("Cannot disable IPv6 on container interface. Set env var DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1 to ignore.")

Oh, good point, thank you. I've fixed it.

thaJeztah

LGTM

left one nit, but not a blocker (so let me know if you want to update the PR before merging)

If dockerd runs on a host with a read-only /proc/sys/net filesystem, it isn't able to enable or disable IPv6 on network interfaces when attaching a container to a network (including initial networks during container creation). In release 26.0.2, a read-only /proc/sys/net meant container creation failed in all cases. So, don't attempt to enable/disable IPv6 on an interface if it's already set appropriately. If it's not possible to enable IPv6 when it's needed, just log (because that's what libnetwork has always done if IPv6 is disabled in the kernel). If it's not possible to disable IPv6 when it needs to be disabled, refuse to create the container and raise an error that suggests setting environment variable "DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1", to tell the daemon it's ok to ignore the problem. Signed-off-by: Rob Murray <rob.murray@docker.com>

thaJeztah

LGTM

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [docker/docker](https://togithub.com/docker/docker) | patch | `26.1.0` -> `26.1.1` | --- ### Release Notes <details> <summary>docker/docker (docker/docker)</summary> ### [`v26.1.1`](https://togithub.com/moby/moby/releases/tag/v26.1.1) [Compare Source](https://togithub.com/docker/docker/compare/v26.1.0...v26.1.1) #### 26.1.1 For a full list of pull requests and changes in this release, refer to the relevant GitHub milestones: - [docker/cli, 26.1.1 milestone](https://togithub.com/docker/cli/issues?q=is%3Aclosed+milestone%3A26.1.1) - [moby/moby, 26.1.1 milestone](https://togithub.com/moby/moby/issues?q=is%3Aclosed+milestone%3A26.1.1) - Deprecated and removed features, see [Deprecated Features](https://togithub.com/docker/cli/blob/v26.1.1/docs/deprecated.md). - Changes to the Engine API, see [API version history](https://togithub.com/moby/moby/blob/v26.1.1/docs/api/version-history.md). ##### Bug fixes and enhancements - Fix `docker run -d` printing an `context canceled` spurious error when OTEL is configured. [docker/cli#5044](https://togithub.com/docker/cli/pull/5044) - Experimental environment variable `DOCKER_BRIDGE_PRESERVE_KERNEL_LL=1` will prevent the daemon from removing the kernel-assigned link local address on a Linux bridge. [moby/moby#47775](https://togithub.com/moby/moby/pull/47775) - Resolve an issue preventing container creation on hosts with a read-only `/proc/sys/net` filesystem. If IPv6 cannot be disabled on an interface due to this, either disable IPv6 by default on the host or ensure `/proc/sys/net` is read-write. Otherwise, start dockerd with `DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1` to bypass the error. [moby/moby#47769](https://togithub.com/moby/moby/pull/47769) > \[!NOTE] > The `DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE` is added as a temporary fix and will be phased out in a future major release after simplifying the IPv6 enablement process. ##### Packaging updates - Update BuildKit to [v0.13.2](https://togithub.com/moby/buildkit/releases/tag/v0.13.2). [moby/moby#47762](https://togithub.com/moby/moby/pull/47762) - Update Compose to [v2.27.0](https://togithub.com/docker/compose/releases/tag/v2.27.0). [docker/docker-ce-packages#1017](https://togithub.com/docker/docker-ce-packaging/pull/1017) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6am on monday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/earthly/dind).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

Works around failure of Docker to create containers (moby/moby#47769)— level=error msg="Cannot disable IPv6 on container interface." error="open /proc/sys/net/ipv6/conf/eth0/disable_ipv6: read-only file system" interface=eth0 —as suggested in the Docker Engine 27.0.1 release notes.

robmry self-assigned this Apr 26, 2024

robmry added status/2-code-review area/networking Networking kind/bugfix PR's that fix bugs area/networking/ipv6 Networking labels Apr 26, 2024

robmry added this to the 26.0.3 milestone Apr 26, 2024

robmry force-pushed the 47751_readonly_procsysnet branch 2 times, most recently from d4b786d to 949486c Compare April 26, 2024 19:18

robmry added the impact/changelog label Apr 26, 2024

robmry force-pushed the 47751_readonly_procsysnet branch from 949486c to 6a26fbe Compare April 29, 2024 08:22

robmry marked this pull request as ready for review April 29, 2024 08:23

robmry requested review from akerouanton and corhere April 29, 2024 08:23

thaJeztah added process/cherry-pick process/cherry-pick/26.0 and removed process/cherry-pick/26.0 labels Apr 29, 2024

thaJeztah reviewed Apr 29, 2024

View reviewed changes

robmry force-pushed the 47751_readonly_procsysnet branch from 6a26fbe to 69f4dd2 Compare April 29, 2024 13:36

akerouanton approved these changes Apr 29, 2024

View reviewed changes

thaJeztah reviewed Apr 29, 2024

View reviewed changes

thaJeztah approved these changes Apr 29, 2024

View reviewed changes

robmry force-pushed the 47751_readonly_procsysnet branch from 69f4dd2 to 01ea18f Compare April 29, 2024 15:17

robmry modified the milestones: 26.0.3, 26.1.1 Apr 29, 2024

robmry mentioned this pull request Apr 29, 2024

Remove env-var that allows IPv6 to be left enabled on an IPv4-only interface #47773

Closed

thaJeztah approved these changes Apr 29, 2024

View reviewed changes

thaJeztah merged commit 8cbd202 into moby:master Apr 29, 2024

vvoland mentioned this pull request Apr 29, 2024

[26.1 backport] Allow for a read-only "/proc/sys/net". #47774

Merged

vvoland added process/cherry-picked and removed process/cherry-pick process/cherry-pick/26.1 labels Apr 29, 2024

robmry deleted the 47751_readonly_procsysnet branch May 1, 2024 09:09

chayuto mentioned this pull request May 2, 2024

<listen_host>::</listen_host> doesnt work with docker engine 26.1.1 ClickHouse/ClickHouse#63256

Open

robmry mentioned this pull request Jun 12, 2024

Remove env-var that allowed startup with IPv6 enabled on an IPv4 interface #47963

Merged

AndrewKvalheim mentioned this pull request Oct 11, 2024

Ensure that /proc/sys/net is writable SeaGL/ubuntu-server-dev#17

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allow for a read-only "/proc/sys/net". #47769

Allow for a read-only "/proc/sys/net". #47769

Uh oh!

robmry commented Apr 26, 2024 •

edited by vvoland

Loading

Uh oh!

thaJeztah Apr 29, 2024

Uh oh!

robmry Apr 29, 2024

Uh oh!

thaJeztah Apr 29, 2024

Uh oh!

thaJeztah Apr 29, 2024

Uh oh!

robmry Apr 29, 2024

Uh oh!

thaJeztah left a comment

Uh oh!

thaJeztah left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		log.G(context.TODO()).WithError(err).Infof(
		"Cannot disable IPv6 on container interface %s but DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1, continuing.", iface)

-				log.G(context.TODO()).WithError(err).Infof(
-					"Cannot disable IPv6 on container interface %s but DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1, continuing.", iface)
+				log.G(context.TODO()).WithFields(log.Fields{
+					"error": err,
+					"interface": iface,
+				}).Info("Cannot disable IPv6 on container interface, but DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE=1, continuing")

Allow for a read-only "/proc/sys/net". #47769

Allow for a read-only "/proc/sys/net". #47769

Uh oh!

Conversation

robmry commented Apr 26, 2024 • edited by vvoland Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah Apr 29, 2024

Choose a reason for hiding this comment

Uh oh!

robmry Apr 29, 2024

Choose a reason for hiding this comment

Uh oh!

thaJeztah Apr 29, 2024

Choose a reason for hiding this comment

Uh oh!

thaJeztah Apr 29, 2024

Choose a reason for hiding this comment

Uh oh!

robmry Apr 29, 2024

Choose a reason for hiding this comment

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

robmry commented Apr 26, 2024 •

edited by vvoland

Loading