Detect overlay2 support on pre-4.0 kernels by thaJeztah · Pull Request #35527 · moby/moby

thaJeztah · 2017-11-16T16:41:07Z

The overlay2 storage-driver requires multiple lower dir support for overlayFs. Support for this feature was added in kernel 4.x, but some distros (RHEL 7.4, CentOS 7.4) ship with an older kernel with this feature backported.

This patch adds feature-detection for multiple lower dirs, and will perform this feature-detection on pre-4.x kernels with overlayFS support.

With this patch applied, daemons running on a kernel with multiple lower dir support will now select "overlay2" as storage-driver, instead of falling back to "overlay".

- Description for the changelog

+ Add support for `overlay2` storage driver on CentOS/RHEL 7.4 [moby/moby#35527](https://github.com/moby/moby/pull/35527)

thaJeztah · 2017-11-16T16:42:55Z

@kolyshkin @friism @dmcgowan I think this os roughly what's needed, but would need testing, and can use some eyes; opened this so that we can discuss

dmcgowan · 2017-11-16T18:31:23Z

daemon/graphdriver/overlay2/overlay.go

Logic is reversed, should be if opts.overrideKernelCheck

Oh! Yes, last-minute change before I pushed, thought to keep the Warning in place so made it an if/else; updating

dmcgowan · 2017-11-16T18:33:36Z

daemon/graphdriver/overlay2/check.go

What is the error that gets returned when it is not supported?

I saw @kolyshkin tested this;

CentOS version Kernel version Compile date Error

7.1 3.10.0-229.1.2.el7.x86_64 ? failed to mount overlay: invalid argument

7.1 3.10.0-229.20.1.el7.x86_64 ? failed to mount overlay: invalid argument

7.2 3.10.0-327.el7.x86_64 Nov 19 2015 nil

7.2 3.10.0-327.36.3.el7.x86_64 Oct 24 2016 nil

7.3 3.10.0-514.26.2.el7.x86_64 Jul 4 2017 nil

kolyshkin · 2017-11-16T19:49:11Z

daemon/graphdriver/overlay2/check.go

nit: s/MkdirAll/Mkdir/

kolyshkin · 2017-11-16T19:49:19Z

daemon/graphdriver/overlay2/check.go

kolyshkin · 2017-11-16T19:53:57Z

daemon/graphdriver/overlay2/check.go

Maybe use a for loop here instead? Something like

for _, dir := range []string{"lower1", "lower2", "upper", "work", "merged"} { if err := os.Mkdir(filepath.Join(td, dir), 0755); err != nil { return err } }

Yes, I like that (I used the other check in this file as a starting point, but that looks better 👍)

kolyshkin · 2017-11-16T19:55:18Z

daemon/graphdriver/overlay2/check.go

No need to use defer here, I think it can just be unmounted right away

kolyshkin · 2017-11-16T19:56:15Z

daemon/graphdriver/overlay2/overlay.go

Perhaps this hunk need to be removed

dmcgowan

LGTM

andrewhsu · 2017-11-27T23:04:19Z

daemon/graphdriver/overlay2/check.go

@kolyshkin curious to know if this was tested with selinux enabled? just wanted to make sure the writing to tmp works.

kolyshkin · 2017-11-28T01:46:43Z

curious to know if this was tested with selinux enabled? just wanted to make sure the writing to tmp works.

@andrewhsu good question! I did my checks on the box with selinux disabled, but by looking at the code I see that directories are created under the graphdriver "home" dir which should succeed as long as docker+selinux is confgured properly.

The above is purely theoretical though; I'll do some real checks

kolyshkin · 2017-11-28T05:48:47Z

Oops

Nov 28 04:37:42 kir-ce71 dockerd[669]: time="2017-11-28T04:37:42.349701814Z" level=debug msg="Multiple lower dirs not supported: stat /var/lib/docker/overlay2: no such file or directory"

I guess we should either do something like

-                        if err := supportsMultipleLowerDir(home); err != nil {
+                        if err := supportsMultipleLowerDir(filepath.Dir(home)); err != nil {

or move the creation of the driver home to above this check (and make sure to remove the dir on error path). I like the former version more.

Update: in fact at this point even the driver home's parent dir (i.e. /var/lib/docker) might not be available, so maybe it's easy to create a dir and do the check

thaJeztah · 2017-11-28T21:43:00Z

@kolyshkin looking at the code, if /var/lib/docker (--data-root) isn't there yet, how does the check for fsmagic work in this case?

fsMagic, err := graphdriver.GetFSMagic(home)

thaJeztah · 2017-11-28T21:54:29Z

Oh, never mind that takes the parent directory

The overlay2 storage-driver requires multiple lower dir support for overlayFs. Support for this feature was added in kernel 4.x, but some distros (RHEL 7.4, CentOS 7.4) ship with an older kernel with this feature backported. This patch adds feature-detection for multiple lower dirs, and will perform this feature-detection on pre-4.x kernels with overlayFS support. With this patch applied, daemons running on a kernel with multiple lower dir support will now select "overlay2" as storage-driver, instead of falling back to "overlay". Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah · 2017-11-28T21:57:39Z

@kolyshkin updated; could you check with SELinux? (given that it now will check /var/lib instead of /var/lib/docker).

Alternatively, we should create /var/lib/docker before the graphdriver checks are performed

kolyshkin · 2017-11-29T02:46:53Z

(given that it now will check /var/lib instead of /var/lib/docker)

By looking at the code, it seems it is checking /var/lib/docker (as it is a parent dir of driver's home, /var/lib/docker/overlay2).

Anyway, it works!

kolyshkin · 2017-11-29T02:53:31Z

The test to start dockerd was done

on a CentOS 7.4 machine with kernel downgraded to 3.10.0-327.el7.x86_64
with selinux turned on
with /var/lib/docker removed

A relevant quote from the logs

Nov 29 02:46:02 kir-ce71 dockerd[7988]: time="2017-11-29T02:46:02.898726143Z" level=info msg="Loading containers: done."
Nov 29 02:46:02 kir-ce71 dockerd[7988]: time="2017-11-29T02:46:02.905776790Z" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: opaque flag erroneously copied up, consider update to kernel 4.8 or later to fix"
Nov 29 02:46:02 kir-ce71 dockerd[7988]: time="2017-11-29T02:46:02.915391003Z" level=info msg="Docker daemon" commit=bb370f5 graphdriver(s)=overlay2 version=dev
Nov 29 02:46:02 kir-ce71 dockerd[7988]: time="2017-11-29T02:46:02.915733174Z" level=info msg="Daemon has completed initialization"

Test sequence:

# systemctl stop docker
# mv /var/lib/docker{,-moved}
# systemctl start docker

System info:

# uname -a
Linux kir-ce71 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@kir-ce71 ~]# selinuxenabled ; echo $?
0

kolyshkin

LGTM (IANAM)

vieux

LGTM

thaJeztah requested a review from dmcgowan as a code owner November 16, 2017 16:41

GordonTheTurtle added the status/0-triage label Nov 16, 2017

thaJeztah added area/storage/overlay impact/changelog impact/documentation status/1-design-review and removed status/0-triage labels Nov 16, 2017

dmcgowan requested changes Nov 16, 2017

View reviewed changes

kolyshkin requested changes Nov 16, 2017

View reviewed changes

thaJeztah force-pushed the feature-detect-overlay2 branch 3 times, most recently from f320429 to de08f78 Compare November 16, 2017 20:35

thaJeztah added the rebuild/windowsRS1 label Nov 17, 2017

GordonTheTurtle removed the rebuild/windowsRS1 label Nov 17, 2017

dmcgowan approved these changes Nov 18, 2017

View reviewed changes

thaJeztah added status/2-code-review and removed status/1-design-review labels Nov 21, 2017

andrewhsu reviewed Nov 27, 2017

View reviewed changes

thaJeztah force-pushed the feature-detect-overlay2 branch from de08f78 to 955c1f8 Compare November 28, 2017 21:56

kolyshkin approved these changes Nov 29, 2017

View reviewed changes

vieux approved these changes Nov 29, 2017

View reviewed changes

vieux merged commit 11e07e7 into moby:master Nov 29, 2017

thaJeztah deleted the feature-detect-overlay2 branch November 29, 2017 21:58

thaJeztah mentioned this pull request Dec 7, 2017

devmapper: Log fstype and mount options during mount error #35732

Merged

thaJeztah mentioned this pull request Jan 26, 2018

Clarify overlay2 support in Docker EE RHEL / CentOS docker/docs#5817

Merged

kolyshkin mentioned this pull request Nov 7, 2019

Improve rootless Docker overlay support detection #40131

Closed

CentOS version	Kernel version	Compile date	Error
7.1	3.10.0-229.1.2.el7.x86_64	?	failed to mount overlay: invalid argument
7.1	3.10.0-229.20.1.el7.x86_64	?	failed to mount overlay: invalid argument
7.2	3.10.0-327.el7.x86_64	Nov 19 2015	nil
7.2	3.10.0-327.36.3.el7.x86_64	Oct 24 2016	nil
7.3	3.10.0-514.26.2.el7.x86_64	Jul 4 2017	nil

Conversation

thaJeztah commented Nov 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah commented Nov 16, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmcgowan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolyshkin commented Nov 28, 2017

Uh oh!

kolyshkin commented Nov 28, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah commented Nov 28, 2017

Uh oh!

thaJeztah commented Nov 28, 2017

Uh oh!

thaJeztah commented Nov 28, 2017

Uh oh!

kolyshkin commented Nov 29, 2017

Uh oh!

kolyshkin commented Nov 29, 2017

A relevant quote from the logs

Test sequence:

System info:

Uh oh!

kolyshkin left a comment

Choose a reason for hiding this comment

Uh oh!

vieux left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

thaJeztah commented Nov 16, 2017 •

edited

Loading

kolyshkin commented Nov 28, 2017 •

edited

Loading