dockerd crashes on restart with live-restart enabled and specific syslog setup #45002
Description
Description
dockerd crashes upon restart when it is configured with the live-restart option and a container is running that logs to an rsyslog tcp port forwarded by docker.
Reproduce
- configure dockerd with
"live-restart":truein /etc/docker/daemon.json - create a container that logs via syslog to a tcp port itself provides. e.g.
docker run -d -p 514:514/tcp --log-driver "syslog" --log-opt "syslog-address=tcp://localhost:514" rsyslog/syslog_appliance_alpine - restart dockerd 2 times
- dockerd crashes upon 2nd restart
- it's not possible to get docker running while the previously created container is still up
Expected behavior
dockerd (re)starts.
docker version
Client: Docker Engine - Community
Version: 23.0.1
API version: 1.42
Go version: go1.19.5
Git commit: a5ee5b1
Built: Thu Feb 9 19:46:54 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 23.0.1
API version: 1.42 (minimum version 1.12)
Go version: go1.19.5
Git commit: bc3805a
Built: Thu Feb 9 19:46:54 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.16
GitCommit: 31aa4358a36870b21a992d3ad2bef29e1d693bec
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
Client:
Context: default
Debug Mode: false
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 1
Server Version: 23.0.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
Kernel Version: 5.10.0-21-amd64
Operating System: Devuan GNU/Linux 4 (chimaera)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 972.7MiB
Name: oread
ID: 41bfc782-c37d-463f-84d9-d47339f0303a
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: trueAdditional Info
Relevant log section including stack trace
time="2023-02-15T13:45:50.754543940Z" level=info msg="[core] [Channel #7] Resolver state updated: {\n \"Addresses\
": [\n {\n \"Addr\": \"/var/run/docker/containerd/containerd.sock\",\n \"ServerName\": \"\",\n \"
Attributes\": {},\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n }\n ],\n
\"ServiceConfig\": null,\n \"Attributes\": null\n} (resolver returned new addresses)" module=grpc
time="2023-02-15T13:45:50.754605944Z" level=info msg="[core] [Channel #7] Channel switches to new LB policy \"pick_
first\"" module=grpc
time="2023-02-15T13:45:50.754660234Z" level=info msg="[core] [Channel #7 SubChannel #8] Subchannel created" module=
grpc
time="2023-02-15T13:45:50.754719230Z" level=info msg="[core] [Channel #7 SubChannel #8] Subchannel Connectivity cha
nge to CONNECTING" module=grpc
time="2023-02-15T13:45:50.754778561Z" level=info msg="[core] [Channel #7 SubChannel #8] Subchannel picks a new addr
ess \"/var/run/docker/containerd/containerd.sock\" to connect" module=grpc
time="2023-02-15T13:45:50.754976134Z" level=info msg="[core] [Channel #7] Channel Connectivity change to CONNECTING
" module=grpc
time="2023-02-15T13:45:50.755768320Z" level=info msg="[core] [Channel #7 SubChannel #8] Subchannel Connectivity cha
nge to READY" module=grpc
time="2023-02-15T13:45:50.755820296Z" level=info msg="[core] [Channel #7] Channel Connectivity change to READY" mod
ule=grpc
time="2023-02-15T13:45:50.765705314Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
time="2023-02-15T13:45:50.772673317Z" level=info msg="Loading containers: start."
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x55abca6561b4]
goroutine 87 [running]:
github.com/docker/docker/vendor/github.com/containerd/fifo.(*fifo).Close(0x0)
/go/src/github.com/docker/docker/vendor/github.com/containerd/fifo/fifo.go:210 +0x34
github.com/docker/docker/vendor/github.com/containerd/containerd/cio.(*cio).Close(0x55abcbda6c54?)
/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/cio/io.go:203 +0x73
github.com/docker/docker/libcontainerd/remote.(*client).Restore.func1()
/go/src/github.com/docker/docker/libcontainerd/remote/client.go:85 +0x65
github.com/docker/docker/libcontainerd/remote.(*client).Restore(0xc000a4e7e0, {0x55abcc859b70?, 0xc0000560a0}, {0xc
0006ca1c0, 0x40}, 0xc0006c03b0)
/go/src/github.com/docker/docker/libcontainerd/remote/client.go:106 +0x8d4
github.com/docker/docker/daemon.(*Daemon).restore.func3(0xc000bb5b80)
/go/src/github.com/docker/docker/daemon/daemon.go:328 +0x3f1
created by github.com/docker/docker/daemon.(*Daemon).restore
/go/src/github.com/docker/docker/daemon/daemon.go:297 +0x365