From time to time, in Docker Desktop, when we start our VM (which include Docker 18.09.1), dockerd fails to start with a containerd timeout error. Is it possible to change the timeout ? Any ideas how to prevent this ?
On our CI, we have this error twice a day.
time="2019-01-25T07:04:44.977151687Z" level=debug msg="Listener created for HTTP on unix (/var/run/docker.sock)"
time="2019-01-25T07:04:45.162571112Z" level=info msg="libcontainerd: started new containerd process" pid=1674
time="2019-01-25T07:04:45.162840398Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2019-01-25T07:04:45.162856961Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2019-01-25T07:04:45.410986427Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/containerd.sock 0 <nil>}]" module=grpc
time="2019-01-25T07:04:45.411073113Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2019-01-25T07:04:45.411704138Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420206230, CONNECTING" module=grpc
time="2019-01-25T07:04:54.569090623Z" level=info msg="starting containerd" revision=9754871865f7fe2f4e74d43e2fc7ccd237edcbce version=v1.2.2
time="2019-01-25T07:04:54.569234870Z" level=debug msg="changing OOM score to -500"
time="2019-01-25T07:04:54.782698053Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
time="2019-01-25T07:04:54.784545823Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
time="2019-01-25T07:04:54.786567883Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
time="2019-01-25T07:04:54.786612170Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
time="2019-01-25T07:04:54.885911502Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "modprobe: module aufs not found in modules.dep\n": exit status 1"
time="2019-01-25T07:04:54.885964794Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
time="2019-01-25T07:04:54.889901357Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
time="2019-01-25T07:04:54.891825699Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
time="2019-01-25T07:04:55.021518729Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
time="2019-01-25T07:04:55.021582732Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
time="2019-01-25T07:04:55.025851643Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
time="2019-01-25T07:04:55.025888221Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
time="2019-01-25T07:04:55.025895800Z" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "modprobe: module aufs not found in modules.dep\n": exit status 1"
time="2019-01-25T07:04:55.166231705Z" level=error msg="failed connecting to containerd" error="failed to dial \"/var/run/docker/containerd/containerd.sock\": context deadline exceeded" module=libcontainerd
time="2019-01-25T07:04:55.266561864Z" level=info msg="killing and restarting containerd" module=libcontainerd pid=1674
time="2019-01-25T07:04:55.267250447Z" level=debug msg="received signal" signal=user defined signal 1
time="2019-01-25T07:04:55.268086716Z" level=info msg="=== BEGIN goroutine stack dump ===
goroutine 14 [running]:
github.com/containerd/containerd/cmd/containerd/command.dumpStacks()
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/cmd/containerd/command/main_unix.go:78 +0x8c
github.com/containerd/containerd/cmd/containerd/command.handleSignals.func1(0xc4200e9920, 0xc4200e98c0, 0x190aba0, 0xc4200b0010, 0xc420094660)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/cmd/containerd/command/main_unix.go:53 +0x274
created by github.com/containerd/containerd/cmd/containerd/command.handleSignals
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/cmd/containerd/command/main_unix.go:43 +0x8b
goroutine 1 [syscall]:
syscall.Syscall(0x4b, 0x3, 0x0, 0x0, 0x67d3e4, 0xc4200b4370, 0xc420364000)
/usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.Fdatasync(0x3, 0x3, 0x0)
/usr/local/go/src/syscall/zsyscall_linux_amd64.go:446 +0x42
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.fdatasync(0xc4203f6d20, 0x4000, 0x4000)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bolt_linux.go:9 +0x3f
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*DB).init(0xc4203f6d20, 0x0, 0xc4200845b0)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:438 +0x1b4
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.Open(0xc4201035e0, 0x48, 0x1a4, 0x20cf680, 0xc420422ce8, 0x1, 0x0)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:218 +0x240
github.com/containerd/containerd/services/server.LoadPlugins.func2(0xc420357ab0, 0xc42023e720, 0x21, 0xc42032e620, 0x1e)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/services/server/server.go:264 +0x49d
github.com/containerd/containerd/plugin.(*Registration).Init(0xc4200b7900, 0xc420357ab0, 0xc4200b7900)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/plugin/plugin.go:100 +0x3a
github.com/containerd/containerd/services/server.New(0x190aba0, 0xc4200b0010, 0xc4204307e0, 0x1, 0xc4204dbc80, 0x0)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/services/server/server.go:120 +0x557
github.com/containerd/containerd/cmd/containerd/command.App.func1(0xc4201351e0, 0xc4201351e0, 0xc4204dbd07)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/cmd/containerd/command/main.go:141 +0x67e
github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0x16f0900, 0x18e7090, 0xc4201351e0, 0xc4200e9860, 0x0)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:502 +0xca
github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0xc4203cee00, 0xc4200b60f0, 0x5, 0x5, 0x0, 0x0)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:268 +0x60e
main.main()
github.com/containerd/containerd/cmd/containerd/main.go:33 +0x51
goroutine 19 [syscall]:
os/signal.signal_recv(0x18fa520)
/usr/local/go/src/runtime/sigqueue.go:139 +0xa8
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x24
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:28 +0x43
goroutine 20 [chan receive]:
github.com/containerd/containerd/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x20aa9a0)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/golang/glog/glog.go:879 +0x8d
created by github.com/containerd/containerd/vendor/github.com/golang/glog.init.0
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/golang/glog/glog.go:410 +0x205
goroutine 15 [select, locked to thread]:
runtime.gopark(0x18e9ed0, 0x0, 0x115f5ff, 0x6, 0x18, 0x1)
/usr/local/go/src/runtime/proc.go:291 +0x120
runtime.selectgo(0xc42005a750, 0xc420094720)
/usr/local/go/src/runtime/select.go:392 +0xe56
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal_unix.go:549 +0x1f6
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1
goroutine 16 [select]:
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Broadcaster).run(0xc4200b7950)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:117 +0x3c4
created by github.com/containerd/containerd/vendor/github.com/docker/go-events.NewBroadcaster
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:39 +0x1b1
=== END goroutine stack dump ==="
time="2019-01-25T07:04:55.567850463Z" level=error msg="containerd did not exit successfully" error="signal: killed" module=libcontainerd
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x157e8f0]
goroutine 42 [running]:
github.com/docker/docker/vendor/github.com/containerd/containerd.(*Client).Close(0x0, 0x0, 0x0)
/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/client.go:536 +0x30
github.com/docker/docker/libcontainerd/supervisor.(*remote).monitorDaemon(0xc4201ea1a0, 0x2766900, 0xc4201f1a00)
/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:321 +0x262
created by github.com/docker/docker/libcontainerd/supervisor.Start
/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:90 +0x3fa
Client: Docker Engine - Community
Version: 18.09.1
API version: 1.39
Go version: go1.10.6
Git commit: 4c52b90
Built: Wed Jan 9 19:33:12 2019
OS/Arch: darwin/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.1
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 4c52b90
Built: Wed Jan 9 19:41:49 2019
OS/Arch: linux/amd64
Experimental: false
Description
From time to time, in Docker Desktop, when we start our VM (which include Docker 18.09.1), dockerd fails to start with a containerd timeout error. Is it possible to change the timeout ? Any ideas how to prevent this ?
On our CI, we have this error twice a day.
Steps to reproduce the issue:
Diagnostics tarball: aftersuite.diagkit.zip
Describe the results you received:
Describe the results you expected:
Additional information you deem important (e.g. issue happens only occasionally):
Output of
docker version: