Errror id already in use when trying to (re-)start a container #40613
Description
Description
Steps to reproduce the issue:
- Sometimes on updates1 (I'm not sure about that upgrades caused this but it looks like the last time it did) containers are killed leaving them in
Exited (137)status. They aren't even automatically restarted although restart policy isunless-stopped. - Trying to (re-)start the container via CLI or Ansible always leads to
id already in use
Describe the results you received:
id already in use
Describe the results you expected:
- Container would restart itself
- Restart via CLI or Ansible would be possible
Additional information you deem important (e.g. issue happens only occasionally):
I usually "fix" this by removing and recreating the affected containers.
Output of docker version: Docker version 19.03.6, build 369ce74a3c
Output of `docker info`: (click to expand)
Client:
Debug Mode: false
Server:
Containers: 27
Running: 22
Paused: 0
Stopped: 5
Images: 89
Server Version: 19.03.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 35bd7a5f69c13e1563af8a93431411cd9ecf5021
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-165-generic
Operating System: Ubuntu 16.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 5.824GiB
Name: some.hostname.com
ID: ULG6:F6GP:JJIN:2RM6:N53H:XIEJ:C2OL:Y57I:BZ72:JIRY:HDHQ:CJ3U
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.): physical
I'm not quite sure whether it could be helpful but this is an example of docker inspect for one of my containers that's is in this "unrestartable" state. (click to expand):
[
{
"Id": "7501b308911f45d04174a5eb91238812efde87ebf38bc0e678c580f98ddde6af",
"Created": "2019-11-07T12:57:41.29487253Z",
"Path": "npm",
"Args": [
"--silent",
"start"
],
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "id already in use",
"StartedAt": "2020-03-02T15:25:34.820136975Z",
"FinishedAt": "2020-03-02T15:22:02.880455838Z"
},
"Image": "sha256:d6df727d12c13468f1252462ca1722e2edca62d156113fd8d1b3e5d0da0f3664",
"ResolvConfPath": "/var/lib/docker/containers/7501b308911f45d04174a5eb91238812efde87ebf38bc0e678c580f98ddde6af/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/7501b308911f45d04174a5eb91238812efde87ebf38bc0e678c580f98ddde6af/hostname",
"HostsPath": "/var/lib/docker/containers/7501b308911f45d04174a5eb91238812efde87ebf38bc0e678c580f98ddde6af/hosts",
"LogPath": "/var/lib/docker/containers/7501b308911f45d04174a5eb91238812efde87ebf38bc0e678c580f98ddde6af/7501b308911f45d04174a5eb91238812efde87ebf38bc0e678c580f98ddde6af-json.log",
"Name": "/smtp-server",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": [],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": null,
"RestartPolicy": {
"Name": "unless-stopped",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Capabilities": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"ReadonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/f392785d570aa1c1cef7ef04c3c86e7fcbbd23319ba8895b30e98b3e33696224-init/diff:/var/lib/docker/overlay2/5f6c44a72282cb406cd50bf83294186424e0643f0feed20c0b41e5f7c9155379/diff:/var/lib/docker/overlay2/811a70aa7cbb7f2c0974f59b5054468386d6a0e714c42a674be30426323362e6/diff:/var/lib/docker/overlay2/be453c159fde682a7e9b240b21cc91a5bf223203e22656ae94f1a0a9cddd3e0f/diff:/var/lib/docker/overlay2/330cc92ef3c8b85a91080bf4bfb63825d49f823ae370fc43f07d82b473ed3c32/diff:/var/lib/docker/overlay2/5da3161a11aae0eb48bda0b1bf941bd8fe1ad1f86f4053a68c79446484399e01/diff:/var/lib/docker/overlay2/acdc7f921080730e434048391b8b38d9be1cdc56fd5ccb6f29f662e16397671b/diff:/var/lib/docker/overlay2/a8864376eee17b4dcdcaf4394ebd07752ecbdc093770ee58a9e0b4433b340d31/diff:/var/lib/docker/overlay2/b901903113000bf7ec9a4e04663e0eb58dae368a77ae7904f8caf5d4db1e6f65/diff:/var/lib/docker/overlay2/1130d4e4ab92338c30068ac6aefdf616065cdba3a8487c9c4c56f0efecec1d33/diff:/var/lib/docker/overlay2/fde3198bf2e6e65a34e8b44f69a5cc1f258e66e29d5d6bf902c2e5abecd8b7c4/diff:/var/lib/docker/overlay2/f3b3213394e467924e3d9641ed75e2c83f8d97883c66281fb3abf8af5f2ef3ff/diff:/var/lib/docker/overlay2/369e628360e8019c5a942055fc59031900388e51a70e2def5641de5cc19e024c/diff:/var/lib/docker/overlay2/217e8f1161f69891b8edb5ef9dae64e22b11e9d8f1a7a0fc331b6a9d7ee17f60/diff:/var/lib/docker/overlay2/8bfcc4c8478be7249197688bfc9417d85b30785d1d3e3190940d3308ea61fc75/diff",
"MergedDir": "/var/lib/docker/overlay2/f392785d570aa1c1cef7ef04c3c86e7fcbbd23319ba8895b30e98b3e33696224/merged",
"UpperDir": "/var/lib/docker/overlay2/f392785d570aa1c1cef7ef04c3c86e7fcbbd23319ba8895b30e98b3e33696224/diff",
"WorkDir": "/var/lib/docker/overlay2/f392785d570aa1c1cef7ef04c3c86e7fcbbd23319ba8895b30e98b3e33696224/work"
},
"Name": "overlay2"
},
"Mounts": [],
"Config": {
"Hostname": "smtp",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"25/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"SENDGRID_API=xx.xxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"NODE_VERSION=8.12.0",
"YARN_VERSION=1.9.4"
],
"Cmd": [
"npm",
"--silent",
"start"
],
"Image": "fgribreau/smtp-to-sendgrid-gateway",
"Volumes": null,
"WorkingDir": "/app",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "0a7aa60763bba1c312ea7e6b874a41e714c548cc961d458e001a5bab34c95bf9",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/0a7aa60763bb",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "d9e8ca412710a86e7ea7906d63b804c37eab5aefcac78b65d59568250804ba72",
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"DriverOpts": null
},
"smtp": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"7501b308911f"
],
"NetworkID": "9800dd9c0e8d110a4fb1747ceef26cd9bf11026694926a9e86858c2e1f397ca6",
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"DriverOpts": null
},
"internet": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"7501b308911f"
],
"NetworkID": "da00599f50e37ae0ca112a313fcb46610695464f6cf9621ff4ceded86e2373ac",
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"DriverOpts": null
}
}
}
}
]1) I installed some updates yesterday and containerd.io, docker-ce and docker-ce-cli were amongst the packages. ↩
Click for a list of the upgrades.
bsdutils:amd64 1:2.27.1-6ubuntu3.9 1:2.27.1-6ubuntu3.10
util-linux:amd64 2.27.1-6ubuntu3.9 2.27.1-6ubuntu3.10
mount:amd64 2.27.1-6ubuntu3.9 2.27.1-6ubuntu3.10
uuid-runtime:amd64 2.27.1-6ubuntu3.9 2.27.1-6ubuntu3.10
libuuid1:amd64 2.27.1-6ubuntu3.9 2.27.1-6ubuntu3.10
libblkid1:amd64 2.27.1-6ubuntu3.9 2.27.1-6ubuntu3.10
libfdisk1:amd64 2.27.1-6ubuntu3.9 2.27.1-6ubuntu3.10
libmount1:amd64 2.27.1-6ubuntu3.9 2.27.1-6ubuntu3.10
libsmartcols1:amd64 2.27.1-6ubuntu3.9 2.27.1-6ubuntu3.10
dbus:amd64 1.10.6-1ubuntu3.4 1.10.6-1ubuntu3.5
libdbus-1-3:amd64 1.10.6-1ubuntu3.4 1.10.6-1ubuntu3.5
dmidecode:amd64 3.0-2ubuntu0.1 3.0-2ubuntu0.2
ubuntu-release-upgrader-core:all 1:16.04.27 1:16.04.29
python3-distupgrade:all 1:16.04.27 1:16.04.29
python3-update-manager:all 1:16.04.16 1:16.04.17
update-manager-core:all 1:16.04.16 1:16.04.17
containerd.io:amd64 1.2.10-3 1.2.12-1
docker-ce-cli:amd64 5:19.03.4~3-0~ubuntu-xenial 5:19.03.6~3-0~ubuntu-xenial
docker-ce:amd64 5:19.03.4~3-0~ubuntu-xenial 5:19.03.6~3-0~ubuntu-xenial
unattended-upgrades:all 1.1ubuntu1.18.04.7~16.04.4 1.1ubuntu1.18.04.7~16.04.6
PS: I know that there are many similar issues (i.e. #20570, #38346, #38249) but they seem to be fixed/had other causes. Otherwise please tell me if I missed something! 🙏