Description

I am running into an error when using the latest version of kind, which just upgraded from containerd 1.6.21 to 1.7.1. For what it's worth, I'm also running with rootless podman on my host machine.

When attempting to import an image (e.g. ctr image import image.tar), I get an error.

Steps to reproduce the issue

1. Start a cluster using kind v0.20.0 using rootless podman on the host.
2. Attempt to load an image that does a recursive delete of files in an earlier layer, something as simple as:

FROM debian:bullseye-slim

RUN mkdir /a && touch /a/b
RUN rm -rf /a

Describe the results you received and expected

The error received from image import is something like:
mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3730886863: failed to convert whiteout file ".wh.a": unlinkat /var/lib/containerd/tmpmounts/containerd-mount3730886863/a: input/output error: unknown

I have traced this down to a commit in containerd that introduces the failure: fa4720f

I believe the ro option is not being passed to mount correctly here. I observe that the Mount() call fails with an "invalid argument" error, leading NeedsUserXAttr() to return false instead of true, which then causes later image operations to fail because userxattr is actually required.

Tracing syscalls, I see the mount call occur as:
mount("overlay", "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/userxattr-check/3845228009/merged", "overlay", 0, "ro,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/userxattr-check/3845228009/lower2:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/userxattr-check/3845228009/lower1,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/userxattr-check/3845228009/upper,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/userxattr-check/3845228009/work,userxattr")

Notably, notice that the mountflags is 0 and that the ro flag is captured in the data parameter.

Contrast that to a mount command-line with exact same arguments on the command line, see that the ro value is pulled from data and that the mountflags is MS_RDONLY:
mount("overlay", "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/userxattr-check/3845228009/merged", "overlay", MS_RDONLY, "lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/userxattr-check/3845228009/lower2:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/userxattr-check/3845228009/lower1,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/userxattr-check/3845228009/upper,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/userxattr-check/3845228009/work,userxattr")

What version of containerd are you using?

containerd github.com/containerd/containerd v1.7.1 1677a17

Any other relevant information

$ runc version 1.1.7
commit: v1.1.7-0-g860f061
spec: 1.0.2-dev
go: go1.20.4
libseccomp: 2.5.1

$ uname -a
Linux 0aa5d6e8cb1c 4.18.0-477.15.1.el8_8.x86_64 #1 SMP Wed Jun 28 15:04:18 UTC 2023 x86_64 GNU/Linu

Show configuration if it is related to CRI plugin.

version = 2

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    restrict_oom_score_adj = false
    sandbox_image = "registry.k8s.io/pause:3.7"
    tolerate_missing_hugepages_controller = true
    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      discard_unpacked_layers = true
      snapshotter = "overlayfs"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = "/etc/containerd/cri-base.json"
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            SystemdCgroup = false
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler]
          base_runtime_spec = "/etc/containerd/cri-base.json"
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.test-handler.options]
            SystemdCgroup = false

[proxy_plugins]
  [proxy_plugins.fuse-overlayfs]
    address = "/run/containerd-fuse-overlayfs.sock"
    type = "snapshot"