Skip to content

NemoClaw k3s Startup Fails - overlayfs Nested Mount Error #2481

Open
Bug
Open
NemoClaw k3s Startup Fails - overlayfs Nested Mount Error#2481
Bug
Assignees
ericksoa
Labels
area: e2eEnd-to-end tests, nightly failures, or validation infrastructureplatform: containerAffects Docker, containerd, Podman, or imagesplatform: k3sAffects K3s deploymentsplatform: ubuntuAffects Ubuntu Linux environments
@stay-pixel

Description

@stay-pixel
stay-pixel
opened on Apr 25, 2026

Description

Hello NVIDIA NemoClaw Support,

I'm experiencing a persistent issue with NemoClaw container startup on Ubuntu 24.04. The container fails to reach healthy status due to an overlayfs nested mount error.

Environment:

  • OS: Ubuntu 24.04 LTS
  • Docker: Latest (with overlayfs storage driver)
  • CPU: Ryzen 9 9950X3D
  • GPU: NVIDIA RTX 5060 Ti (16GB VRAM)
  • NemoClaw Image: ghcr.io/nvidia/openshell/cluster:0.0.36

Error:
failed to mount overlay: mount source: "overlay", target: "/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check.../merged", fstype: overlay, flags: 0, err: invalid argument

The error message suggests: "overlayfs" snapshotter cannot be enabled for "/var/lib/rancher/k3s/agent/containerd", try using "fuse-overlayfs" or "native"

What I've Verified:

  • Overlay kernel module is loaded: lsmod | grep overlay ✅
  • Docker storage driver: overlayfs (working fine for other containers) ✅
  • --privileged flag enabled ✅
  • Fresh container start (no corrupted state) ✅

Attempted Fixes:

  • Restarting container with --privileged flag
  • Clearing persisted volumes
  • Attempting --snapshotter=native flag (not recognized)

Question:
Is there a known workaround for nested overlay mount issues in Docker? Should I reconfigure the Docker storage driver, or is there a NemoClaw-specific configuration to avoid overlayfs?

Any guidance appreciated.

Thanks,
Ben H.

Reproduction Steps

Description

I'm unable to start the NemoClaw container due to a persistent overlayfs nested mount error. The container fails to reach healthy status during k3s initialization.

Environment

  • OS: Ubuntu 24.04 LTS
  • Docker: Latest (overlayfs storage driver)
  • Hardware: Ryzen 9 9950X3D, NVIDIA RTX 5060 Ti (16GB VRAM)
  • NemoClaw Image: ghcr.io/nvidia/openshell/cluster:0.0.36
  • Docker Command Used:
docker run -d \
  --name openshell-cluster-nemoclaw \
  --privileged \
  -p 8080:30051 \
  --add-host=host.docker.internal:192.168.50.100 \
  ghcr.io/nvidia/openshell/cluster:0.0.36

Error Message

failed to mount overlay: mount source: "overlay", target: "/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check.../merged", fstype: overlay, flags: 0, err: invalid argument
"overlayfs" snapshotter cannot be enabled for "/var/lib/rancher/k3s/agent/containerd", try using "fuse-overlayfs" or "native"

What I've Verified

  • ✅ Overlay kernel module is loaded: lsmod | grep overlay shows overlay 221184 4
  • ✅ Docker storage driver is overlayfs (working for all other containers)
  • ✅ Container runs with --privileged flag
  • ✅ Fresh container start (cleaned volumes, no corrupted state)
  • ✅ Attempted --snapshotter=native flag (not recognized by k3s server command)

Question

Is there a known workaround for nested overlay mount issues when running k3s inside Docker? Should I reconfigure Docker's storage driver, or is there a NemoClaw-specific configuration to avoid overlayfs?

Any guidance appreciated.

Environment

Ubuntu 24.04 LTS
Node.js version: v22.22.2
Docker: Docker Engine 27.x (overlayfs storage driver)
NemoClaw V. 24.04

Debug Output

Client:
 Version:    29.1.3
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc.)
    Version:  2.40.3+ds1-0ubuntu1~24.04.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  trust: Manage trust on Docker images (Docker Inc.)
    Version:  29.1.3
    Path:     /usr/libexec/docker/cli-plugins/docker-trust

Server:
 Containers: 7
  Running: 2
  Paused: 0
  Stopped: 5
 Images: 61
 Server Version: 29.1.3
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.17.0-22-generic
 Operating System: Ubuntu 24.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 91.7GiB
 Name: aineurogrid-MS-7E70
 ID: 36cab974-1d71-47e5-b252-bf99da053fb0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables

Logs

time="2026-04-25T20:53:08Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:53:08 +0000 UTC notAfter=2027-04-25 19:53:08 +0000 UTC"
time="2026-04-25T20:53:08Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:08 +0000 UTC notAfter=2027-04-25 19:53:08 +0000 UTC"
time="2026-04-25T20:53:08Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:08 +0000 UTC notAfter=2027-04-25 19:53:08 +0000 UTC"
time="2026-04-25T20:53:08Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:08 +0000 UTC notAfter=2027-04-25 19:53:08 +0000 UTC"
time="2026-04-25T20:53:08Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1326164430/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1326164430/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1326164430/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1326164430/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1326164430/work\", err: invalid argument"
time="2026-04-25T20:53:17Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:53:17 +0000 UTC notAfter=2027-04-25 19:53:17 +0000 UTC"
time="2026-04-25T20:53:17Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:17 +0000 UTC notAfter=2027-04-25 19:53:17 +0000 UTC"
time="2026-04-25T20:53:17Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:17 +0000 UTC notAfter=2027-04-25 19:53:17 +0000 UTC"
time="2026-04-25T20:53:17Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:17 +0000 UTC notAfter=2027-04-25 19:53:17 +0000 UTC"
time="2026-04-25T20:53:17Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check934964538/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check934964538/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check934964538/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check934964538/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check934964538/work\", err: invalid argument"
time="2026-04-25T20:53:26Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:53:26 +0000 UTC notAfter=2027-04-25 19:53:26 +0000 UTC"
time="2026-04-25T20:53:26Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:26 +0000 UTC notAfter=2027-04-25 19:53:26 +0000 UTC"
time="2026-04-25T20:53:26Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:26 +0000 UTC notAfter=2027-04-25 19:53:26 +0000 UTC"
time="2026-04-25T20:53:26Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:26 +0000 UTC notAfter=2027-04-25 19:53:26 +0000 UTC"
time="2026-04-25T20:53:26Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2056324134/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2056324134/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2056324134/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2056324134/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2056324134/work\", err: invalid argument"
time="2026-04-25T20:53:36Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:53:36 +0000 UTC notAfter=2027-04-25 19:53:36 +0000 UTC"
time="2026-04-25T20:53:36Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:36 +0000 UTC notAfter=2027-04-25 19:53:36 +0000 UTC"
time="2026-04-25T20:53:36Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:36 +0000 UTC notAfter=2027-04-25 19:53:36 +0000 UTC"
time="2026-04-25T20:53:36Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:36 +0000 UTC notAfter=2027-04-25 19:53:36 +0000 UTC"
time="2026-04-25T20:53:36Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2719447054/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2719447054/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2719447054/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2719447054/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2719447054/work\", err: invalid argument"
time="2026-04-25T20:53:45Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:53:45 +0000 UTC notAfter=2027-04-25 19:53:45 +0000 UTC"
time="2026-04-25T20:53:45Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:45 +0000 UTC notAfter=2027-04-25 19:53:45 +0000 UTC"
time="2026-04-25T20:53:45Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:45 +0000 UTC notAfter=2027-04-25 19:53:45 +0000 UTC"
time="2026-04-25T20:53:45Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:45 +0000 UTC notAfter=2027-04-25 19:53:45 +0000 UTC"
time="2026-04-25T20:53:45Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2185476076/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2185476076/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2185476076/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2185476076/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2185476076/work\", err: invalid argument"
time="2026-04-25T20:53:53Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:53:53 +0000 UTC notAfter=2027-04-25 19:53:53 +0000 UTC"
time="2026-04-25T20:53:53Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:53 +0000 UTC notAfter=2027-04-25 19:53:53 +0000 UTC"
time="2026-04-25T20:53:53Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:53 +0000 UTC notAfter=2027-04-25 19:53:53 +0000 UTC"
time="2026-04-25T20:53:53Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:53:53 +0000 UTC notAfter=2027-04-25 19:53:53 +0000 UTC"
time="2026-04-25T20:53:53Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2434392133/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2434392133/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2434392133/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2434392133/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2434392133/work\", err: invalid argument"
time="2026-04-25T20:54:03Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:54:03 +0000 UTC notAfter=2027-04-25 19:54:03 +0000 UTC"
time="2026-04-25T20:54:03Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:03 +0000 UTC notAfter=2027-04-25 19:54:03 +0000 UTC"
time="2026-04-25T20:54:03Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:03 +0000 UTC notAfter=2027-04-25 19:54:03 +0000 UTC"
time="2026-04-25T20:54:03Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:03 +0000 UTC notAfter=2027-04-25 19:54:03 +0000 UTC"
time="2026-04-25T20:54:03Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3978577899/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3978577899/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3978577899/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3978577899/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3978577899/work\", err: invalid argument"
time="2026-04-25T20:54:10Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:54:10 +0000 UTC notAfter=2027-04-25 19:54:10 +0000 UTC"
time="2026-04-25T20:54:10Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:10 +0000 UTC notAfter=2027-04-25 19:54:10 +0000 UTC"
time="2026-04-25T20:54:10Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:10 +0000 UTC notAfter=2027-04-25 19:54:10 +0000 UTC"
time="2026-04-25T20:54:10Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:10 +0000 UTC notAfter=2027-04-25 19:54:10 +0000 UTC"
time="2026-04-25T20:54:10Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check450061814/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check450061814/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check450061814/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check450061814/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check450061814/work\", err: invalid argument"
time="2026-04-25T20:54:18Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:54:18 +0000 UTC notAfter=2027-04-25 19:54:18 +0000 UTC"
time="2026-04-25T20:54:18Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:18 +0000 UTC notAfter=2027-04-25 19:54:18 +0000 UTC"
time="2026-04-25T20:54:18Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:18 +0000 UTC notAfter=2027-04-25 19:54:18 +0000 UTC"
time="2026-04-25T20:54:19Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:19 +0000 UTC notAfter=2027-04-25 19:54:19 +0000 UTC"
time="2026-04-25T20:54:19Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1055546733/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1055546733/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1055546733/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1055546733/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1055546733/work\", err: invalid argument"
time="2026-04-25T20:54:27Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:54:27 +0000 UTC notAfter=2027-04-25 19:54:27 +0000 UTC"
time="2026-04-25T20:54:27Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:27 +0000 UTC notAfter=2027-04-25 19:54:27 +0000 UTC"
time="2026-04-25T20:54:27Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:27 +0000 UTC notAfter=2027-04-25 19:54:27 +0000 UTC"
time="2026-04-25T20:54:27Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:27 +0000 UTC notAfter=2027-04-25 19:54:27 +0000 UTC"
time="2026-04-25T20:54:27Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2190037250/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2190037250/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2190037250/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2190037250/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2190037250/work\", err: invalid argument"
time="2026-04-25T20:54:34Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:54:34 +0000 UTC notAfter=2027-04-25 19:54:34 +0000 UTC"
time="2026-04-25T20:54:34Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:34 +0000 UTC notAfter=2027-04-25 19:54:34 +0000 UTC"
time="2026-04-25T20:54:34Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:34 +0000 UTC notAfter=2027-04-25 19:54:34 +0000 UTC"
time="2026-04-25T20:54:34Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:34 +0000 UTC notAfter=2027-04-25 19:54:34 +0000 UTC"
time="2026-04-25T20:54:34Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check580380705/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check580380705/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check580380705/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check580380705/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check580380705/work\", err: invalid argument"
time="2026-04-25T20:54:44Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:54:44 +0000 UTC notAfter=2027-04-25 19:54:44 +0000 UTC"
time="2026-04-25T20:54:44Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:44 +0000 UTC notAfter=2027-04-25 19:54:44 +0000 UTC"
time="2026-04-25T20:54:44Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:44 +0000 UTC notAfter=2027-04-25 19:54:44 +0000 UTC"
time="2026-04-25T20:54:44Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:44 +0000 UTC notAfter=2027-04-25 19:54:44 +0000 UTC"
time="2026-04-25T20:54:44Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3751655687/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3751655687/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3751655687/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3751655687/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3751655687/work\", err: invalid argument"
time="2026-04-25T20:54:53Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:54:53 +0000 UTC notAfter=2027-04-25 19:54:53 +0000 UTC"
time="2026-04-25T20:54:53Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:53 +0000 UTC notAfter=2027-04-25 19:54:53 +0000 UTC"
time="2026-04-25T20:54:53Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:53 +0000 UTC notAfter=2027-04-25 19:54:53 +0000 UTC"
time="2026-04-25T20:54:53Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:54:53 +0000 UTC notAfter=2027-04-25 19:54:53 +0000 UTC"
time="2026-04-25T20:54:53Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2224131658/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2224131658/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2224131658/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2224131658/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2224131658/work\", err: invalid argument"
time="2026-04-25T20:55:02Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:55:02 +0000 UTC notAfter=2027-04-25 19:55:02 +0000 UTC"
time="2026-04-25T20:55:02Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:02 +0000 UTC notAfter=2027-04-25 19:55:02 +0000 UTC"
time="2026-04-25T20:55:02Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:02 +0000 UTC notAfter=2027-04-25 19:55:02 +0000 UTC"
time="2026-04-25T20:55:02Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:02 +0000 UTC notAfter=2027-04-25 19:55:02 +0000 UTC"
time="2026-04-25T20:55:02Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2417992170/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2417992170/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2417992170/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2417992170/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check2417992170/work\", err: invalid argument"
time="2026-04-25T20:55:09Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:55:09 +0000 UTC notAfter=2027-04-25 19:55:09 +0000 UTC"
time="2026-04-25T20:55:09Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:09 +0000 UTC notAfter=2027-04-25 19:55:09 +0000 UTC"
time="2026-04-25T20:55:10Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:10 +0000 UTC notAfter=2027-04-25 19:55:10 +0000 UTC"
time="2026-04-25T20:55:10Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:10 +0000 UTC notAfter=2027-04-25 19:55:10 +0000 UTC"
time="2026-04-25T20:55:10Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3892354339/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3892354339/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3892354339/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3892354339/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3892354339/work\", err: invalid argument"
time="2026-04-25T20:55:19Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:55:19 +0000 UTC notAfter=2027-04-25 19:55:19 +0000 UTC"
time="2026-04-25T20:55:19Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:19 +0000 UTC notAfter=2027-04-25 19:55:19 +0000 UTC"
time="2026-04-25T20:55:19Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:19 +0000 UTC notAfter=2027-04-25 19:55:19 +0000 UTC"
time="2026-04-25T20:55:19Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:19 +0000 UTC notAfter=2027-04-25 19:55:19 +0000 UTC"
time="2026-04-25T20:55:19Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check4035095624/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check4035095624/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check4035095624/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check4035095624/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check4035095624/work\", err: invalid argument"
time="2026-04-25T20:55:28Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:55:28 +0000 UTC notAfter=2027-04-25 19:55:28 +0000 UTC"
time="2026-04-25T20:55:28Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:28 +0000 UTC notAfter=2027-04-25 19:55:28 +0000 UTC"
time="2026-04-25T20:55:28Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:28 +0000 UTC notAfter=2027-04-25 19:55:28 +0000 UTC"
time="2026-04-25T20:55:28Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:28 +0000 UTC notAfter=2027-04-25 19:55:28 +0000 UTC"
time="2026-04-25T20:55:28Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1278345218/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1278345218/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1278345218/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1278345218/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check1278345218/work\", err: invalid argument"
time="2026-04-25T20:55:37Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:55:37 +0000 UTC notAfter=2027-04-25 19:55:37 +0000 UTC"
time="2026-04-25T20:55:37Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:37 +0000 UTC notAfter=2027-04-25 19:55:37 +0000 UTC"
time="2026-04-25T20:55:37Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:37 +0000 UTC notAfter=2027-04-25 19:55:37 +0000 UTC"
time="2026-04-25T20:55:37Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:37 +0000 UTC notAfter=2027-04-25 19:55:37 +0000 UTC"
time="2026-04-25T20:55:37Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3610945350/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3610945350/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3610945350/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3610945350/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check3610945350/work\", err: invalid argument"
time="2026-04-25T20:55:46Z" level=info msg="certificate CN=60d0c2a78048 signed by CN=k3s-server-ca@1777150234: notBefore=2026-04-25 19:55:46 +0000 UTC notAfter=2027-04-25 19:55:46 +0000 UTC"
time="2026-04-25T20:55:46Z" level=info msg="certificate CN=system:node:60d0c2a78048,O=system:nodes signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:46 +0000 UTC notAfter=2027-04-25 19:55:46 +0000 UTC"
time="2026-04-25T20:55:46Z" level=info msg="certificate CN=system:kube-proxy signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:46 +0000 UTC notAfter=2027-04-25 19:55:46 +0000 UTC"
time="2026-04-25T20:55:46Z" level=info msg="certificate CN=system:k3s-controller signed by CN=k3s-client-ca@1777150234: notBefore=2026-04-25 19:55:46 +0000 UTC notAfter=2027-04-25 19:55:46 +0000 UTC"
time="2026-04-25T20:55:46Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check164393608/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check164393608/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check164393608/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check164393608/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check164393608/work\", err: invalid argument"
time="2026-04-25T20:55:48Z" level=info msg="Waiting up to 2s for graceful shutdown of Kine GRPC server..."
time="2026-04-25T20:55:48Z" level=info msg="TTL events watch channel closed"
time="2026-04-25T20:55:48Z" level=info msg="Closing database connections..."
time="2026-04-25T20:55:48Z" level=info msg="TTL events work queue has shut down"
time="2026-04-25T20:55:48Z" level=fatal msg="Error: failed to retrieve agent configuration: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: mount source: \"overlay\", target: \"/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check164393608/merged\", fstype: overlay, flags: 0, data: \"lowerdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check164393608/lower2:/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check164393608/lower1,upperdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check164393608/upper,workdir=/var/lib/rancher/k3s/agent/containerd/multiple-lowerdir-check164393608/work\", err: invalid argument"

Checklist

  • I confirmed this bug is reproducible
  • I searched existing issues and this is not a duplicate

Metadata

Metadata

Assignees

Labels

area: e2eEnd-to-end tests, nightly failures, or validation infrastructureplatform: containerAffects Docker, containerd, Podman, or imagesplatform: k3sAffects K3s deploymentsplatform: ubuntuAffects Ubuntu Linux environments

Type

Bug

Fields

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions