-
Notifications
You must be signed in to change notification settings - Fork 18.9k
Closed as not planned
Labels
area/runtimeRuntimeRuntimekind/enhancementEnhancements are not bugs or new features but can improve usability or performance.Enhancements are not bugs or new features but can improve usability or performance.
Description
Description
From time to time, in Docker Desktop, when we start our VM (which include Docker 18.09.1), dockerd fails to start with a containerd timeout error. Is it possible to change the timeout ? Any ideas how to prevent this ?
On our CI, we have this error twice a day.
Steps to reproduce the issue:
- Start Docker Desktop
Diagnostics tarball: aftersuite.diagkit.zip
Describe the results you received:
time="2019-01-25T07:04:44.977151687Z" level=debug msg="Listener created for HTTP on unix (/var/run/docker.sock)"
time="2019-01-25T07:04:45.162571112Z" level=info msg="libcontainerd: started new containerd process" pid=1674
time="2019-01-25T07:04:45.162840398Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2019-01-25T07:04:45.162856961Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2019-01-25T07:04:45.410986427Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/containerd.sock 0 <nil>}]" module=grpc
time="2019-01-25T07:04:45.411073113Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2019-01-25T07:04:45.411704138Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420206230, CONNECTING" module=grpc
time="2019-01-25T07:04:54.569090623Z" level=info msg="starting containerd" revision=9754871865f7fe2f4e74d43e2fc7ccd237edcbce version=v1.2.2
time="2019-01-25T07:04:54.569234870Z" level=debug msg="changing OOM score to -500"
time="2019-01-25T07:04:54.782698053Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
time="2019-01-25T07:04:54.784545823Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
time="2019-01-25T07:04:54.786567883Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
time="2019-01-25T07:04:54.786612170Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
time="2019-01-25T07:04:54.885911502Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "modprobe: module aufs not found in modules.dep\n": exit status 1"
time="2019-01-25T07:04:54.885964794Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
time="2019-01-25T07:04:54.889901357Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
time="2019-01-25T07:04:54.891825699Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
time="2019-01-25T07:04:55.021518729Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
time="2019-01-25T07:04:55.021582732Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
time="2019-01-25T07:04:55.025851643Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
time="2019-01-25T07:04:55.025888221Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
time="2019-01-25T07:04:55.025895800Z" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "modprobe: module aufs not found in modules.dep\n": exit status 1"
time="2019-01-25T07:04:55.166231705Z" level=error msg="failed connecting to containerd" error="failed to dial \"/var/run/docker/containerd/containerd.sock\": context deadline exceeded" module=libcontainerd
time="2019-01-25T07:04:55.266561864Z" level=info msg="killing and restarting containerd" module=libcontainerd pid=1674
time="2019-01-25T07:04:55.267250447Z" level=debug msg="received signal" signal=user defined signal 1
time="2019-01-25T07:04:55.268086716Z" level=info msg="=== BEGIN goroutine stack dump ===
goroutine 14 [running]:
github.com/containerd/containerd/cmd/containerd/command.dumpStacks()
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/cmd/containerd/command/main_unix.go:78 +0x8c
github.com/containerd/containerd/cmd/containerd/command.handleSignals.func1(0xc4200e9920, 0xc4200e98c0, 0x190aba0, 0xc4200b0010, 0xc420094660)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/cmd/containerd/command/main_unix.go:53 +0x274
created by github.com/containerd/containerd/cmd/containerd/command.handleSignals
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/cmd/containerd/command/main_unix.go:43 +0x8b
goroutine 1 [syscall]:
syscall.Syscall(0x4b, 0x3, 0x0, 0x0, 0x67d3e4, 0xc4200b4370, 0xc420364000)
/usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.Fdatasync(0x3, 0x3, 0x0)
/usr/local/go/src/syscall/zsyscall_linux_amd64.go:446 +0x42
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.fdatasync(0xc4203f6d20, 0x4000, 0x4000)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bolt_linux.go:9 +0x3f
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*DB).init(0xc4203f6d20, 0x0, 0xc4200845b0)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:438 +0x1b4
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.Open(0xc4201035e0, 0x48, 0x1a4, 0x20cf680, 0xc420422ce8, 0x1, 0x0)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:218 +0x240
github.com/containerd/containerd/services/server.LoadPlugins.func2(0xc420357ab0, 0xc42023e720, 0x21, 0xc42032e620, 0x1e)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/services/server/server.go:264 +0x49d
github.com/containerd/containerd/plugin.(*Registration).Init(0xc4200b7900, 0xc420357ab0, 0xc4200b7900)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/plugin/plugin.go:100 +0x3a
github.com/containerd/containerd/services/server.New(0x190aba0, 0xc4200b0010, 0xc4204307e0, 0x1, 0xc4204dbc80, 0x0)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/services/server/server.go:120 +0x557
github.com/containerd/containerd/cmd/containerd/command.App.func1(0xc4201351e0, 0xc4201351e0, 0xc4204dbd07)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/cmd/containerd/command/main.go:141 +0x67e
github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0x16f0900, 0x18e7090, 0xc4201351e0, 0xc4200e9860, 0x0)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:502 +0xca
github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0xc4203cee00, 0xc4200b60f0, 0x5, 0x5, 0x0, 0x0)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:268 +0x60e
main.main()
github.com/containerd/containerd/cmd/containerd/main.go:33 +0x51
goroutine 19 [syscall]:
os/signal.signal_recv(0x18fa520)
/usr/local/go/src/runtime/sigqueue.go:139 +0xa8
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x24
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:28 +0x43
goroutine 20 [chan receive]:
github.com/containerd/containerd/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x20aa9a0)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/golang/glog/glog.go:879 +0x8d
created by github.com/containerd/containerd/vendor/github.com/golang/glog.init.0
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/golang/glog/glog.go:410 +0x205
goroutine 15 [select, locked to thread]:
runtime.gopark(0x18e9ed0, 0x0, 0x115f5ff, 0x6, 0x18, 0x1)
/usr/local/go/src/runtime/proc.go:291 +0x120
runtime.selectgo(0xc42005a750, 0xc420094720)
/usr/local/go/src/runtime/select.go:392 +0xe56
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal_unix.go:549 +0x1f6
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1
goroutine 16 [select]:
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Broadcaster).run(0xc4200b7950)
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:117 +0x3c4
created by github.com/containerd/containerd/vendor/github.com/docker/go-events.NewBroadcaster
/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:39 +0x1b1
=== END goroutine stack dump ==="
time="2019-01-25T07:04:55.567850463Z" level=error msg="containerd did not exit successfully" error="signal: killed" module=libcontainerd
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x157e8f0]
goroutine 42 [running]:
github.com/docker/docker/vendor/github.com/containerd/containerd.(*Client).Close(0x0, 0x0, 0x0)
/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/client.go:536 +0x30
github.com/docker/docker/libcontainerd/supervisor.(*remote).monitorDaemon(0xc4201ea1a0, 0x2766900, 0xc4201f1a00)
/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:321 +0x262
created by github.com/docker/docker/libcontainerd/supervisor.Start
/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:90 +0x3fa
Describe the results you expected:
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
Client: Docker Engine - Community
Version: 18.09.1
API version: 1.39
Go version: go1.10.6
Git commit: 4c52b90
Built: Wed Jan 9 19:33:12 2019
OS/Arch: darwin/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.1
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 4c52b90
Built: Wed Jan 9 19:41:49 2019
OS/Arch: linux/amd64
Experimental: false
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area/runtimeRuntimeRuntimekind/enhancementEnhancements are not bugs or new features but can improve usability or performance.Enhancements are not bugs or new features but can improve usability or performance.