Skip to content

DOCKER_BUILDKIT=1 crashes docker service #1836

Closed
Closed
DOCKER_BUILDKIT=1 crashes docker service#1836
moby/buildkit
#3896
 (+1)
Assignees
jedevc
Labels
kind/bugSomething isn't workingstatus/triage
@ap-wtioit

Description

@ap-wtioit
ap-wtioit
opened on May 24, 2023

Contributing guidelines

I've found a bug and checked that ...

  • ... the documentation does not mention anything about my problem
  • ... there are no open or closed issues that are related to my problem

Description

Using the following Dockerfile:

# syntax=docker/dockerfile:1
FROM debian:11 AS stage-base
FROM stage-base AS stage-final

and build with DOCKER_BUILDKIT=1 docker build . crashes the docker service on multiple of our servers and dev machines.

Expected behaviour

building images should not crash the docker service.

Actual behaviour

building this image should crashes the docker service.

Buildx version

github.com/docker/buildx v0.10.4 c513d34

Docker info

Client: Docker Engine - Community
 Version:    24.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 24
  Running: 10
  Paused: 0
  Stopped: 14
 Images: 59
 Server Version: 24.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.19.0-42-generic
 Operating System: Ubuntu 22.10
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 61.96GiB
 Name: wt-io-it-bigbear3001
 ID: WFOT:GN2I:KKU2:AHXI:QIJS:JHXO:5CGH:UYXZ:MADD:JK4W:K7Q3:ZAE7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true
 Default Address Pools:
   Base: 172.16.0.0/12, Size: 26
   Base: 192.168.0.0/16, Size: 26

Builders list

NAME/NODE            DRIVER/ENDPOINT             STATUS   BUILDKIT PLATFORMS
dreamy_shirley       docker-container                              
  dreamy_shirley0    unix:///var/run/docker.sock inactive          
ecstatic_ishizaka *  docker-container                              
  ecstatic_ishizaka0 unix:///var/run/docker.sock inactive          
default              docker                                        
  default            default                     running  v0.11.6  linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

Configuration

# syntax=docker/dockerfile:1
FROM debian:11 AS stage-base
FROM stage-base AS stage-final

Note: initially it happened for https://github.com/docker-mailserver/docker-mailserver/blob/master/Dockerfile but this minimal file reproduces the issue on our systems

Build logs

[+] Building 0.1s (6/6) FINISHED                                                                                                                                                                                                  
 => [internal] load .dockerignore                                                                                                                                                                                            0.0s
 => => transferring context: 59B                                                                                                                                                                                             0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                         0.0s
 => => transferring dockerfile: 126B                                                                                                                                                                                         0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                                                                                   0.0s
 => CACHED docker-image://docker.io/docker/dockerfile:1                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/debian:11                                                                                                                                                                 0.0s
 => CACHED [stage-base 1/1] FROM docker.io/library/debian:11                                                                                                                                                                 0.0s
ERROR: failed to solve: Unavailable: error reading from server: EOF

Additional info

Build with DOCKER_BUILDKIT=0 docker build . works without any issues (but has warnings for deprecated build method).

Output of journalctl -u docker.service -f

Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]: panic: no ':' separator in digest ""
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]: goroutine 4749 [running]:
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]: github.com/docker/docker/vendor/github.com/opencontainers/go-digest.Digest.sepIndex({0x0, 0x0})
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]:         /go/src/github.com/docker/docker/vendor/github.com/opencontainers/go-digest/digest.go:153 +0x9a
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]: github.com/docker/docker/vendor/github.com/opencontainers/go-digest.Digest.Algorithm(...)
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]:         /go/src/github.com/docker/docker/vendor/github.com/opencontainers/go-digest/digest.go:122
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/provenance.slsaMaterials({{0xc001df0500, 0x3, 0x3}, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0x0, ...}, ...})
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/provenance/predicate.go:70 +0x16b
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/provenance.NewPredicate(0xc001280c30)
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/provenance/predicate.go:137 +0xa5
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.NewProvenanceCreator({0x55ebdba328f8, 0xc0009b42d0}, 0x0?, {0x55ebdba36600, 0xc000ff5820}, 0x55ebda1d45dc?, 0xc00065a000)
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/provenance.go:397 +0x3d0
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.(*Solver).recordBuildHistory.func1.1({0x55ebdba36600?, 0xc000ff5820?}, 0x0?)
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:203 +0xb1
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]: github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver.(*Solver).recordBuildHistory.func1.2()
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]:         /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/llbsolver/solver.go:245 +0x54
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]: github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1()
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]:         /go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]: created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
Mai 24 11:10:47 wt-io-it-bigbear3001 dockerd[612359]:         /go/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
Mai 24 11:10:47 wt-io-it-bigbear3001 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mai 24 11:10:47 wt-io-it-bigbear3001 systemd[1]: docker.service: Failed with result 'exit-code'.
Mai 24 11:10:47 wt-io-it-bigbear3001 systemd[1]: docker.service: Consumed 2.063s CPU time.
Mai 24 11:10:49 wt-io-it-bigbear3001 systemd[1]: docker.service: Scheduled restart job, restart counter is at 13.
Mai 24 11:10:49 wt-io-it-bigbear3001 systemd[1]: Stopped Docker Application Container Engine.
Mai 24 11:10:49 wt-io-it-bigbear3001 systemd[1]: docker.service: Consumed 2.063s CPU time.
Mai 24 11:10:49 wt-io-it-bigbear3001 systemd[1]: Starting Docker Application Container Engine...
Mai 24 11:10:49 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:49.411606595+02:00" level=info msg="Starting up"
Mai 24 11:10:49 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:49.412074720+02:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Mai 24 11:10:49 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:49.422715216+02:00" level=info msg="[graphdriver] trying configured driver: overlay2"
Mai 24 11:10:49 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:49.447004239+02:00" level=info msg="Loading containers: start."
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.242672658+02:00" level=error msg="Resolver Start failed for container 26e559d6af80b7c181b078f350abf7e42094e64463a0480980cc583752d0c452, \"setting up IP table rules failed:  (iptables failed: iptables --wait -t nat -N DOCKER_OUTPUT: iptables: Chain already exists.\\n (exit status 1))\""
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.242688358+02:00" level=error msg="failed to populate fields for osl sandbox 8ed10e9e1bfd3534247b912af45814ab0d45f66ad8388df109821a6fb4d0c329"
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.244069480+02:00" level=error msg="Resolver Start failed for container 21e6ebc23b80443020402729e2fb83db1755aed722c6711a090c7713aa939f7f, \"setting up IP table rules failed:  (iptables failed: iptables --wait -t nat -N DOCKER_OUTPUT: iptables: Chain already exists.\\n (exit status 1))\""
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.244078290+02:00" level=error msg="failed to populate fields for osl sandbox 98ab43d12f211d61669b7f6b9ce8bf0a86cf4e0733873e5ed294ee8b8facb8d2"
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.245283145+02:00" level=error msg="Resolver Start failed for container 0a75f3c0a61d4ba61a976632f104a51b6ace820cdfcc21658f02d0af76187750, \"setting up IP table rules failed:  (iptables failed: iptables --wait -t nat -N DOCKER_OUTPUT: iptables: Chain already exists.\\n (exit status 1))\""
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.245291125+02:00" level=error msg="failed to populate fields for osl sandbox 9d37ca134ab77883d89592a6746dcf0f23b4fa188ed9a928f598e74eaff5aacd"
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.246381651+02:00" level=error msg="Resolver Start failed for container 966e664d24c8736f9290664be862345d03135a6ca2017c1cdbd1d2f168ac836f, \"setting up IP table rules failed:  (iptables failed: iptables --wait -t nat -N DOCKER_OUTPUT: iptables: Chain already exists.\\n (exit status 1))\""
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.246390161+02:00" level=error msg="failed to populate fields for osl sandbox 0aed137655f842afbbbac9da7808c2b238637879ab4017c07768f06ac557f072"
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.247495267+02:00" level=error msg="Resolver Start failed for container 9b177a6dfded45aaf848159f919f2420d57fa11b460d4b19cb6dcb77f98e476a, \"setting up IP table rules failed:  (iptables failed: iptables --wait -t nat -N DOCKER_OUTPUT: iptables: Chain already exists.\\n (exit status 1))\""
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.247503267+02:00" level=error msg="failed to populate fields for osl sandbox 2fbdca3f1bb1610b938f9859f97fea34f7605997b9a38d7bc13de37c829db430"
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.248747631+02:00" level=error msg="Resolver Start failed for container 010ff1f5951c3dd5c00ba5835b76501322e07e95b79f0c96430f373c2cdf4c0c, \"setting up IP table rules failed:  (iptables failed: iptables --wait -t nat -N DOCKER_OUTPUT: iptables: Chain already exists.\\n (exit status 1))\""
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.248756211+02:00" level=error msg="failed to populate fields for osl sandbox 7c4029c33e8e04466ffaaea536a590bcb71243df7a20ead07e52da7527e2232a"
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.250231843+02:00" level=error msg="Resolver Start failed for container 8fcb03019709742b533242837d7b9910063a143ebcf7f5d9a61e38a71448d4eb, \"setting up IP table rules failed:  (iptables failed: iptables --wait -t nat -N DOCKER_OUTPUT: iptables: Chain already exists.\\n (exit status 1))\""
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.250256362+02:00" level=error msg="failed to populate fields for osl sandbox f57d4b3ccaaf1798d5fc4ed8f318070671f85e8f97b5ccad231b000eff2b4610"
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.251365709+02:00" level=error msg="Resolver Start failed for container 309f1cf10ce46b06905bc264e2a830a7abfd5de47416c2ee8b86b2a2ba7d56ca, \"setting up IP table rules failed:  (iptables failed: iptables --wait -t nat -N DOCKER_OUTPUT: iptables: Chain already exists.\\n (exit status 1))\""
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.251374358+02:00" level=error msg="failed to populate fields for osl sandbox 44ee12dd45d972a636f46ec6481ca143456522c7bff49201efc4c2729bf46ef4"
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.252783011+02:00" level=error msg="Resolver Start failed for container a7271bd25baa8c9a6e8c67aca23a0a03016a6988243e3245a11e63ecd7464843, \"setting up IP table rules failed:  (iptables failed: iptables --wait -t nat -N DOCKER_OUTPUT: iptables: Chain already exists.\\n (exit status 1))\""
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.252792181+02:00" level=error msg="failed to populate fields for osl sandbox 5aff5b7174a72dfe704a3db1e499a9e182a35320a888aec5b8ade53e7dc43c1e"
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.254104404+02:00" level=error msg="Resolver Start failed for container bfde265ff6f9c1100adef435ea7160357e57f39ad0b7128adce571bd8c8274a6, \"setting up IP table rules failed:  (iptables failed: iptables --wait -t nat -N DOCKER_OUTPUT: iptables: Chain already exists.\\n (exit status 1))\""
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.254112954+02:00" level=error msg="failed to populate fields for osl sandbox aff49c7576aed883a5cf8dc287a92390d6a1d0a3b1261567950527850e4ed049"
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.254324041+02:00" level=info msg="there are running containers, updated network configuration will not take affect"
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.254864264+02:00" level=info msg="Loading containers: done."
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.261753168+02:00" level=info msg="Docker daemon" commit=463850e graphdriver=overlay2 version=24.0.1
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.261785987+02:00" level=info msg="Daemon has completed initialization"
Mai 24 11:10:50 wt-io-it-bigbear3001 dockerd[619284]: time="2023-05-24T11:10:50.486237053+02:00" level=info msg="API listen on /run/docker.sock"
Mai 24 11:10:50 wt-io-it-bigbear3001 systemd[1]: Started Docker Application Container Engine.

Errors of other builds that ran on the same server (while docker crashed) at the time:

#{N} ERROR: process "redacted_unrelated_cmd" did not complete successfully: exit code: 4294967295
ERROR: failed to solve: process "redacted_unrelated_cmd" did not complete successfully: exit code: 4294967295

When doing this often (or you have many CI jobs running in this issue) systemd stops restarting docker and you end up with:

ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

and need to restart docker with:

sudo systemctl start docker

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't workingstatus/triage

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions