docker restarts containers before containerd socket is opened #203
Description
Hi,
Description
For a few days, my flatcar server doesn't reboot containers (set to always).
Impact
Since updates triggers reboot, I often have all my services down :-((
Environment and steps to reproduce
- Create a container with restart policy as always
- reboot
Expected behavior
I expect the containers to successfuly restart.
Additional information
My server is migrated from coreos. I disabled docker.socket and enabled docker.service according to #175, using systemctl enable --now docker.service and systemctal disable --now docker.socket.
# cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=2605.6.0
VERSION_ID=2605.6.0
BUILD_ID=2020-09-28-2140
PRETTY_NAME="Flatcar Container Linux by Kinvolk 2605.6.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org"
FLATCAR_BOARD="amd64-usr"journalctl --boot -u docker
-- Logs begin at Tue 2020-10-06 15:49:25 UTC, end at Tue 2020-10-06 17:44:38 UTC. --
Oct 06 17:38:09 mantienne systemd[1]: Starting Docker Application Container Engine...
Oct 06 17:38:09 mantienne dockerd[682]: time="2020-10-06T17:38:09.644756869Z" level=info msg="Starting up"
Oct 06 17:38:09 mantienne dockerd[682]: time="2020-10-06T17:38:09.699907778Z" level=info msg="libcontainerd: started new containerd process" pid=738
Oct 06 17:38:09 mantienne dockerd[682]: time="2020-10-06T17:38:09.701025860Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Oct 06 17:38:09 mantienne dockerd[682]: time="2020-10-06T17:38:09.701144665Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Oct 06 17:38:09 mantienne dockerd[682]: time="2020-10-06T17:38:09.701262568Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 }] }" modul>
Oct 06 17:38:09 mantienne dockerd[682]: time="2020-10-06T17:38:09.701387933Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Oct 06 17:38:09 mantienne dockerd[738]: time="2020-10-06T17:38:09.828401077Z" level=info msg="starting containerd" revision=40779f9760e207feb7ff24cf21236bf5e63b2b17 version=1.3.7
...
Oct 06 17:39:49 mantienne dockerd[682]: time="2020-10-06T17:39:42.841452604Z" level=error msg="Failed to start container 4ed4990aae075c3f833eb6e333c6ab36baee711ba44717a7489cf2ccb20a4a39: OCI runtime create failed> ...
...
Oct 06 17:40:16 mantienne dockerd[682]: time="2020-10-06T17:40:15.945040333Z" level=error msg="stream copy error: reading from a closed fifo"
Oct 06 17:40:16 mantienne dockerd[682]: time="2020-10-06T17:40:15.951512208Z" level=error msg="stream copy error: reading from a closed fifo"
Oct 06 17:40:16 mantienne dockerd[682]: time="2020-10-06T17:40:15.951589580Z" level=error msg="stream copy error: reading from a closed fifo"
Oct 06 17:40:16 mantienne dockerd[682]: time="2020-10-06T17:40:15.951642001Z" level=error msg="stream copy error: reading from a closed fifo"
Oct 06 17:40:16 mantienne dockerd[682]: time="2020-10-06T17:40:15.951676107Z" level=error msg="stream copy error: reading from a closed fifo"
Oct 06 17:40:16 mantienne dockerd[682]: time="2020-10-06T17:40:16.012045326Z" level=error msg="stream copy error: reading from a closed fifo"
Oct 06 17:40:16 mantienne dockerd[682]: time="2020-10-06T17:40:16.015254719Z" level=error msg="stream copy error: reading from a closed fifo"
Oct 06 17:40:16 mantienne dockerd[682]: time="2020-10-06T17:40:16.019335322Z" level=error msg="stream copy error: reading from a closed fifo"
...
Oct 06 17:40:21 mantienne dockerd[682]: time="2020-10-06T17:40:21.834153935Z" level=info msg="Daemon has completed initialization"
Oct 06 17:40:21 mantienne systemd[1]: Started Docker Application Container Engine.
Oct 06 17:40:21 mantienne dockerd[682]: time="2020-10-06T17:40:21.890268502Z" level=info msg="API listen on [::]:2376"
Oct 06 17:40:21 mantienne dockerd[682]: time="2020-10-06T17:40:21.890467617Z" level=info msg="API listen on /var/run/docker.sock"
Do you have some clue about this ?