Skip to content

Docker startup sometimes fails inside Docker Desktop #38636

@guillaumerose

Description

@guillaumerose

Description

From time to time, in Docker Desktop, when we start our VM (which include Docker 18.09.1), dockerd fails to start with a containerd timeout error. Is it possible to change the timeout ? Any ideas how to prevent this ?

On our CI, we have this error twice a day.

Steps to reproduce the issue:

  1. Start Docker Desktop

Diagnostics tarball: aftersuite.diagkit.zip

Describe the results you received:

 time="2019-01-25T07:04:44.977151687Z" level=debug msg="Listener created for HTTP on unix (/var/run/docker.sock)"
 time="2019-01-25T07:04:45.162571112Z" level=info msg="libcontainerd: started new containerd process" pid=1674
 time="2019-01-25T07:04:45.162840398Z" level=info msg="parsed scheme: \"unix\"" module=grpc
 time="2019-01-25T07:04:45.162856961Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
 time="2019-01-25T07:04:45.410986427Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}]" module=grpc
 time="2019-01-25T07:04:45.411073113Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
 time="2019-01-25T07:04:45.411704138Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420206230, CONNECTING" module=grpc
 time="2019-01-25T07:04:54.569090623Z" level=info msg="starting containerd" revision=9754871865f7fe2f4e74d43e2fc7ccd237edcbce version=v1.2.2
 time="2019-01-25T07:04:54.569234870Z" level=debug msg="changing OOM score to -500"
 time="2019-01-25T07:04:54.782698053Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
 time="2019-01-25T07:04:54.784545823Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
 time="2019-01-25T07:04:54.786567883Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
 time="2019-01-25T07:04:54.786612170Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
 time="2019-01-25T07:04:54.885911502Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "modprobe: module aufs not found in modules.dep\n": exit status 1"
 time="2019-01-25T07:04:54.885964794Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
 time="2019-01-25T07:04:54.889901357Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
 time="2019-01-25T07:04:54.891825699Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
 time="2019-01-25T07:04:55.021518729Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
 time="2019-01-25T07:04:55.021582732Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
 time="2019-01-25T07:04:55.025851643Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
 time="2019-01-25T07:04:55.025888221Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
 time="2019-01-25T07:04:55.025895800Z" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "modprobe: module aufs not found in modules.dep\n": exit status 1"
 time="2019-01-25T07:04:55.166231705Z" level=error msg="failed connecting to containerd" error="failed to dial \"/var/run/docker/containerd/containerd.sock\": context deadline exceeded" module=libcontainerd
 time="2019-01-25T07:04:55.266561864Z" level=info msg="killing and restarting containerd" module=libcontainerd pid=1674
 time="2019-01-25T07:04:55.267250447Z" level=debug msg="received signal" signal=user defined signal 1
 time="2019-01-25T07:04:55.268086716Z" level=info msg="=== BEGIN goroutine stack dump ===
 goroutine 14 [running]:
 github.com/containerd/containerd/cmd/containerd/command.dumpStacks()
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/cmd/containerd/command/main_unix.go:78 +0x8c
 github.com/containerd/containerd/cmd/containerd/command.handleSignals.func1(0xc4200e9920, 0xc4200e98c0, 0x190aba0, 0xc4200b0010, 0xc420094660)
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/cmd/containerd/command/main_unix.go:53 +0x274
 created by github.com/containerd/containerd/cmd/containerd/command.handleSignals
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/cmd/containerd/command/main_unix.go:43 +0x8b

 goroutine 1 [syscall]:
 syscall.Syscall(0x4b, 0x3, 0x0, 0x0, 0x67d3e4, 0xc4200b4370, 0xc420364000)
 	/usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
 syscall.Fdatasync(0x3, 0x3, 0x0)
 	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:446 +0x42
 github.com/containerd/containerd/vendor/go.etcd.io/bbolt.fdatasync(0xc4203f6d20, 0x4000, 0x4000)
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bolt_linux.go:9 +0x3f
 github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*DB).init(0xc4203f6d20, 0x0, 0xc4200845b0)
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:438 +0x1b4
 github.com/containerd/containerd/vendor/go.etcd.io/bbolt.Open(0xc4201035e0, 0x48, 0x1a4, 0x20cf680, 0xc420422ce8, 0x1, 0x0)
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:218 +0x240
 github.com/containerd/containerd/services/server.LoadPlugins.func2(0xc420357ab0, 0xc42023e720, 0x21, 0xc42032e620, 0x1e)
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/services/server/server.go:264 +0x49d
 github.com/containerd/containerd/plugin.(*Registration).Init(0xc4200b7900, 0xc420357ab0, 0xc4200b7900)
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/plugin/plugin.go:100 +0x3a
 github.com/containerd/containerd/services/server.New(0x190aba0, 0xc4200b0010, 0xc4204307e0, 0x1, 0xc4204dbc80, 0x0)
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/services/server/server.go:120 +0x557
 github.com/containerd/containerd/cmd/containerd/command.App.func1(0xc4201351e0, 0xc4201351e0, 0xc4204dbd07)
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/cmd/containerd/command/main.go:141 +0x67e
 github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0x16f0900, 0x18e7090, 0xc4201351e0, 0xc4200e9860, 0x0)
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:502 +0xca
 github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0xc4203cee00, 0xc4200b60f0, 0x5, 0x5, 0x0, 0x0)
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:268 +0x60e
 main.main()
 	github.com/containerd/containerd/cmd/containerd/main.go:33 +0x51

 goroutine 19 [syscall]:
 os/signal.signal_recv(0x18fa520)
 	/usr/local/go/src/runtime/sigqueue.go:139 +0xa8
 os/signal.loop()
 	/usr/local/go/src/os/signal/signal_unix.go:22 +0x24
 created by os/signal.init.0
 	/usr/local/go/src/os/signal/signal_unix.go:28 +0x43

 goroutine 20 [chan receive]:
 github.com/containerd/containerd/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x20aa9a0)
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/golang/glog/glog.go:879 +0x8d
 created by github.com/containerd/containerd/vendor/github.com/golang/glog.init.0
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/golang/glog/glog.go:410 +0x205

 goroutine 15 [select, locked to thread]:
 runtime.gopark(0x18e9ed0, 0x0, 0x115f5ff, 0x6, 0x18, 0x1)
 	/usr/local/go/src/runtime/proc.go:291 +0x120
 runtime.selectgo(0xc42005a750, 0xc420094720)
 	/usr/local/go/src/runtime/select.go:392 +0xe56
 runtime.ensureSigM.func1()
 	/usr/local/go/src/runtime/signal_unix.go:549 +0x1f6
 runtime.goexit()
 	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

 goroutine 16 [select]:
 github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Broadcaster).run(0xc4200b7950)
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:117 +0x3c4
 created by github.com/containerd/containerd/vendor/github.com/docker/go-events.NewBroadcaster
 	/tmp/tmp.uRi922615F/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:39 +0x1b1

 === END goroutine stack dump ==="
 time="2019-01-25T07:04:55.567850463Z" level=error msg="containerd did not exit successfully" error="signal: killed" module=libcontainerd
 panic: runtime error: invalid memory address or nil pointer dereference
 [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x157e8f0]

 goroutine 42 [running]:
 github.com/docker/docker/vendor/github.com/containerd/containerd.(*Client).Close(0x0, 0x0, 0x0)
 	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/client.go:536 +0x30
 github.com/docker/docker/libcontainerd/supervisor.(*remote).monitorDaemon(0xc4201ea1a0, 0x2766900, 0xc4201f1a00)
 	/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:321 +0x262
 created by github.com/docker/docker/libcontainerd/supervisor.Start
 	/go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:90 +0x3fa

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client: Docker Engine - Community
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:33:12 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:41:49 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/runtimeRuntimekind/enhancementEnhancements are not bugs or new features but can improve usability or performance.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions