Skip to content

Segfault using kube driver with persistent volume #3850

@AythaE

Description

@AythaE

Contributing guidelines

I've found a bug and checked that ...

  • ... the documentation does not mention anything about my problem
  • ... there are no open or closed issues that are related to my problem

Description

I was testing the new feature persistent storage in kubernetes driver but everytime I create a builder with the persistent-volume-claim.requests.storage option i've received a segfault on further buildx commands.

Expected behaviour

The builder should work as expected. Similarly to how it works without persistent volumes.

Actual behaviour

I've tested with my local computer and with a remote server with the same results.

You can see the segmentation violation in the docker buildx ls log

I'm using Ubuntu 24.04.4 in both systems and trying to create a builder in an AWS EKS v1.34.6 cluster.

Buildx version

github.com/docker/buildx v0.34.0 3e73561

Docker info

Client: Docker Engine - Community
 Version:    29.4.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.34.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.1.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 182
  Running: 6
  Paused: 0
  Stopped: 176
 Images: 162
 Server Version: 29.4.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 77c84241c7cbdd9b4eca2591793e3d4f4317c590
 runc version: v1.3.5-0-g488fc13e
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.14.0-28-generic
 Operating System: Ubuntu 24.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.27GiB
 Name: aythae-desk-ubn2
 ID: 336f6b24-d446-4860-be97-023a9941bac7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables

Builders list

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x480 pc=0x1e29671]

goroutine 24 [running]:
github.com/docker/buildx/driver/kubernetes.(*Driver).Info(0x3b8847e762c0, {0x2b9c5f8, 0x3b8847a66bd0})
	github.com/docker/buildx/driver/kubernetes/driver.go:209 +0x111
github.com/docker/buildx/builder.(*Node).loadData(0x3b8848007c10, {0x2b9c5f8, 0x3b8847a66bd0}, {0x0, 0x0, 0x0})
	github.com/docker/buildx/builder/node.go:242 +0x7c
github.com/docker/buildx/builder.(*Builder).LoadNodes.(*Builder).LoadNodes.func2.func3()
	github.com/docker/buildx/builder/node.go:142 +0x56d
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 68
	golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

Configuration

# test.dockerfile
from alpine

run echo hello
docker buildx build --builder kube -t test -f test.dockerfile --load .

Build logs

docker buildx create -D  --bootstrap   --name=kube   --driver=kubernetes   --platform=linux/amd64   --node=docker-builder-amd64   --driver-opt=namespace=docker-buildkit,nodeselector="kubernetes.io/arch=amd64",requests.cpu=1,requests.memory=4G,limits.cpu=2,limits.memory=8G,persistent-volume-claim.requests.storage=50Gi

[+] Building 34.8s (1/1) FINISHED                                                                                                                                    
 => [internal] booting buildkit                                                                                                                                34.8s
 => => waiting for 1 pods to be ready, timeout: 2 minutes                                                                                                      34.7s
kube

Additional info

I've also tried to build an image with the created builder but a different segmentation violation happens. I guess the root cause should be the same.

docker buildx build --builder kube -t test -f test.dockerfile --load .
 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x480 pc=0x1e29671]

goroutine 98 [running]:
github.com/docker/buildx/driver/kubernetes.(*Driver).Info(0x12c3865c1ce0, {0x2b9c588, 0x12c3865559a0})
	github.com/docker/buildx/driver/kubernetes/driver.go:209 +0x111
github.com/docker/buildx/driver.Boot({0x2b9c588, 0x12c3865559a0}, {0x2b9c550, 0x12c3867a8f90}, 0x12c386555860, {0x2b9c748?, 0x12c386a122c0?})
	github.com/docker/buildx/driver/driver.go:96 +0x113
github.com/docker/buildx/build/resolver.(*nodeResolver).boot.func1.1({0x2b9c588, 0x12c3865559a0})
	github.com/docker/buildx/build/resolver/driver.go:282 +0x176
github.com/moby/buildkit/util/flightcontrol.(*call[...]).run(0x2bb25a0)
	github.com/moby/buildkit@v0.30.0/util/flightcontrol/flightcontrol.go:122 +0x14d
sync.(*Once).doSlow(0x0?, 0x12c386523f98?)
	sync/once.go:78 +0xac
sync.(*Once).Do(...)
	sync/once.go:69
created by github.com/moby/buildkit/util/flightcontrol.(*call[...]).wait in goroutine 14
	github.com/moby/buildkit@v0.30.0/util/flightcontrol/flightcontrol.go:166 +0x456

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't working

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions