docker hang with container in created status #38064
Description
docker daemon do not reply do any command like:
docker logs testdocker inspect test
Steps to reproduce the issue:
Simple reproducer that I came with:
docker run --name test --rm -it alpine sh- hit
Ctrl-Cfast enough so that container created but not earlier (i.e. 0.5 mississippi)
Additional information you deem important (e.g. issue happens only occasionally):
So after this command if you are lucky enough (actually on my machine it triggers almost every time), docker will hang.
Even restarting the docker systemd service does not helps, since docker-containerd-shim hanged, and you will find next messages in journalctl after restart:
Oct 19 11:10:18 systemd[1]: docker.service: Found left-over process 4992 (docker-containe) in control group while starting unit. Ignoring.
Oct 19 11:10:18 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
I attached to the docker-containerd-shim with strace there is nothing interesting to me, and attaching to it with gdb does not helps me to understand what is going too (since I don't have full stacktraces).
But I noticed after this commands it has deleted pipes opened, i.e.:
root:/etc/udev (master)# pgrep -f shim | xargs -r lsof -np
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
docker-co 4092 root cwd DIR 0,21 120 130135851 /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/073656a3236e200c8c6454b1ee7744eefbd90547196f2e1008d7d21776f303cb
docker-co 4092 root rtd DIR 254,0 4096 2 /
docker-co 4092 root txt REG 254,0 4739264 2652881 /usr/bin/docker-containerd-shim
docker-co 4092 root 0r CHR 1,3 0t0 1028 /dev/null
docker-co 4092 root 1w CHR 1,3 0t0 1028 /dev/null
docker-co 4092 root 2w CHR 1,3 0t0 1028 /dev/null
docker-co 4092 root 3u unix 0x00000000ceb7d2aa 0t0 130135854 @/containerd-shim/moby/073656a3236e200c8c6454b1ee7744eefbd90547196f2e1008d7d21776f303cb/shim.sock@ type=STREAM
docker-co 4092 root 4u a_inode 0,13 0 11442 [eventpoll]
docker-co 4092 root 5u a_inode 0,13 0 11442 [eventpoll]
docker-co 4092 root 6u unix 0x00000000ceb7d2aa 0t0 130135854 @/containerd-shim/moby/073656a3236e200c8c6454b1ee7744eefbd90547196f2e1008d7d21776f303cb/shim.sock@ type=STREAM
docker-co 4092 root 7u unix 0x00000000f7fa6458 0t0 130132593 @/containerd-shim/moby/073656a3236e200c8c6454b1ee7744eefbd90547196f2e1008d7d21776f303cb/shim.sock@ type=STREAM
docker-co 4092 root 8r REG 0,21 105 130139168 /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/073656a3236e200c8c6454b1ee7744eefbd90547196f2e1008d7d21776f303cb/log.json
docker-co 4092 root 10u FIFO 0,21 0t0 130139164 /run/docker/containerd/073656a3236e200c8c6454b1ee7744eefbd90547196f2e1008d7d21776f303cb/init-stdin (deleted)
docker-co 4092 root 11u CHR 5,2 0t0 2 /ptmx
docker-co 4092 root 13r FIFO 0,21 0t0 130139164 /run/docker/containerd/073656a3236e200c8c6454b1ee7744eefbd90547196f2e1008d7d21776f303cb/init-stdin (deleted)
docker-co 4092 root 14u FIFO 0,21 0t0 130139165 /run/docker/containerd/073656a3236e200c8c6454b1ee7744eefbd90547196f2e1008d7d21776f303cb/init-stdout (deleted)
docker-co 4092 root 15w FIFO 0,21 0t0 130139165 /run/docker/containerd/073656a3236e200c8c6454b1ee7744eefbd90547196f2e1008d7d21776f303cb/init-stdout (deleted)
docker-co 4092 root 16u FIFO 0,21 0t0 130139165 /run/docker/containerd/073656a3236e200c8c6454b1ee7744eefbd90547196f2e1008d7d21776f303cb/init-stdout (deleted)
docker-co 4092 root 17r FIFO 0,21 0t0 130139165 /run/docker/containerd/073656a3236e200c8c6454b1ee7744eefbd90547196f2e1008d7d21776f303cb/init-stdout (deleted)
Anyway I think it will be better for you to just reproduce it and look by yourself.
But if you need some additional information, feel free to ask.
Output of docker version:
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.11
Git commit: e68fc7a215
Built: Fri Sep 7 11:26:59 2018
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.1-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.11
Git commit: e68fc7a215
Built: Fri Sep 7 11:26:11 2018
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 33
Server Version: 18.06.1-ce
Storage Driver: overlay
Backing Filesystem: extfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.18.1-arch1-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.43GiB
Name: azat
ID: TC2Z:EQI5:EUIS:ILMM:AGLC:4UBX:WBFT:5FSV:RKLC:FLB7:IOEQ:2N7Q
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false