concurrent map writes panic when building multiple images in parallel #40752
Description
BUG REPORT INFORMATION
I'm observing the docker daemon crash when I build multiple images in parallel. This started when we enabled Buildkit, so this is likely related. I first observed this on a system with 40 logical cores, but am able to reproduce this in under a minute on a 6 core vm.
The following script is the most concise repro I've found. It'll exit when the daemon panics.
#!/bin/bash
set -e
DOCKER_BUILDKIT=1; export DOCKER_BUILDKIT
mkdir -p /tmp/test_image
echo "FROM ubuntu:16.04" > /tmp/test_image/Dockerfile
echo "RUN echo 1 > out" >> /tmp/test_image/Dockerfile
docker build --build-arg BUILDKIT_INLINE_CACHE=1 -t foo:bar /tmp/test_image
while true; do
for i in $(seq 12)
do
docker build -t\
foo:derived_${i}\
--build-arg BUILDKIT_INLINE_CACHE=1\
--cache-from=foo:bar \
/tmp/test_image &
done
for job in `jobs -p`
do
wait $job
done
done
Describe the results you received:
The daemon exits with an error message fatal error: concurrent map writes
Describe the results you expected:
I expect the daemon to not have to restart.
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
docker version 1 ↵
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:25:55 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:24:26 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Output of docker info:
docker info
Client:
Debug Mode: false
Server:
Containers: 21
Running: 0
Paused: 0
Stopped: 21
Images: 7
Server Version: 19.03.8
Storage Driver: zfs
Zpool: rpool
Zpool Health: ONLINE
Parent Dataset: rpool/ROOT/ubuntu_2bts8c/var/lib
Space Used By Parent: 4867842048
Space Available: 43242509440
Parent Quota: no
Compression: lz4
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.3.0-40-generic
Operating System: Ubuntu 19.10
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 15.61GiB
Name: tsh-buster
ID: BEGE:NR2X:X2ZT:WZMW:XQSQ:3MPI:NS5M:HPPY:V6GE:ROLS:UF6T:TCDG
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.):
Ubuntu 19.10 vm running on a 2018 Macbook pro with 6 cores.
Originally this was seen on a CI system running with the 19-dind docker image on a 40 core server.
The following are logs of the backtraces that are associated with the panic.
docker-logs.txt