Skip to content

Error initializing docker.server while starting daemon by systemd #37506

@godmbort

Description

@godmbort

Output of docker version:

Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:11:02 2018
 OS/Arch:           linux/amd64
 Experimental:      false
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Output of docker info:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Steps to reproduce the issue:

Try start or restart docker: systemctl start docker
Drop with panic: runtime error: invalid memory address or nil pointer dereference error.

Describe the results you received:

Doesn't start docker.server.
Docker.client start without errors.

Describe the results you expected:

Start docker.server

Additional information you deem important (e.g. issue happens only occasionally):

dockerd[8957]:  level=info msg="libcontainerd: started new docker-containerd process" pid=8969
dockerd[8957]:  level=info msg="parsed scheme: \"unix\"" module=grpc
dockerd[8957]:  level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
dockerd[8957]:  level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0  <nil
dockerd[8957]:  level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
dockerd[8957]:  level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42007f850, CONNECTING" module=grpc
dockerd[8957]:  level=info msg="starting containerd" revision=d64c661f1d51c48782c9cec8fda7604785f93587 version=v1.1.1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
dockerd[8957]:  level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.sna
dockerd[8957]:  level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
dockerd[8957]:  level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snaps
dockerd[8957]:  level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
dockerd[8957]:  level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshott
dockerd[8957]:  level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter
dockerd[8957]:  level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
dockerd[8957]:  level=info msg="loading plugin "io.containerd.grpc.v1.cri"..." type=io.containerd.grpc.v1
dockerd[8957]:  level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntime:{Type:io.containerd.runti
dockerd[8957]:  level=info msg="Connect containerd service"
dockerd[8957]:  level=info msg="Get image filesystem path "/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs""
dockerd[8957]:  level=error msg="Failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load f
dockerd[8957]:  level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
dockerd[8957]:  level=info msg="Start subscribing containerd event"
dockerd[8957]:  level=info msg="Start recovering state"
dockerd[8957]:  level=info msg=serving... address="/var/run/docker/containerd/docker-containerd-debug.sock"
dockerd[8957]:  level=info msg=serving... address="/var/run/docker/containerd/docker-containerd.sock"
dockerd[8957]:  level=info msg="containerd successfully booted in 0.004553s"
dockerd[8957]:  level=info msg="Start event monitor"
dockerd[8957]:  level=info msg="Start snapshots syncer"
dockerd[8957]:  level=info msg="Start streaming server"
dockerd[8957]:  level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42007f850, READY" module=grpc
dockerd[8957]:  level=info msg="parsed scheme: \"unix\"" module=grpc
dockerd[8957]:  level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
dockerd[8957]:  level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0  <nil
dockerd[8957]:  level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
dockerd[8957]:  level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42022e6b0, CONNECTING" module=grpc
dockerd[8957]:  level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42022e6b0, READY" module=grpc
dockerd[8957]:  level=info msg="[graphdriver] using prior storage driver: aufs"
dockerd[8957]:  level=info msg="Graph migration to content-addressability took 0.00 seconds"
dockerd[8957]:  level=warning msg="Your kernel does not support swap memory limit"
dockerd[8957]:  level=warning msg="Your kernel does not support cgroup rt period"
dockerd[8957]:  level=warning msg="Your kernel does not support cgroup rt runtime"
dockerd[8957]:  level=info msg="parsed scheme: \"unix\"" module=grpc
dockerd[8957]:  level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
dockerd[8957]:  level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0  <nil
dockerd[8957]:  level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
dockerd[8957]:  level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42007fb10, CONNECTING" module=grpc
dockerd[8957]:  level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42007fb10, READY" module=grpc
dockerd[8957]:  level=info msg="Loading containers: start."
dockerd[8957]:  level=error msg="Error looking up volumes-from container" error="No such container: nginx_tmp" from-container=nginx_tmp
dockerd[8957]:  level=info msg="Removing stale sandbox 2d1a1a359b706b431d238879c8058896674d024900c904fd3bd042be88a59aea (9e8e0a015e130064c8d6dafef
dockerd[8957]:  level=info msg="Removing stale sandbox 2facfc4a7575cdd1068191b89f9bb5be51886b79cede08d9f086e97a8a33713f (c55ea4de191bed702751d1d2f
dockerd[8957]:  level=info msg="Removing stale sandbox 8b26968ea28f5b8d02bec9110f7bdc31bdb86bbdd58cb84c4bb8a86a72959168 (15451828b6862127483479f4d
dockerd[8957]:  level=error msg="getNetworkFromStore for nid 24c07b727fcd9a26b55d2fabeea15ed4180635993e01d785f96e98586f7d4826 failed while trying 
dockerd[8957]:  level=error msg="getEndpointFromStore for eid ec4ce27ba19489124e9e94d0350cb6b4ca436373f365ffb5a1cfeddfaf8693b4 failed while trying
dockerd[8957]:  level=info msg="Removing stale sandbox cdfce5c36c9319e04d78f8d729a26e93514b8ce09dc112cc95e1a9b51eb91147 (ae73cb3f0eed4c1fe5dcd4dae
dockerd[8957]:  level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint aa663d95d46313623107d8a798d
dockerd[8957]:  level=warning msg="Failed getting network for ep 6908fe3dbfa7e246943d23a0d997cdaa0767c30b33366626733f2c765ed06fb1 during sandbox c
dockerd[8957]:  level=warning msg="Failed deleting endpoint ec4ce27ba19489124e9e94d0350cb6b4ca436373f365ffb5a1cfeddfaf8693b4: failed to get endpoi
dockerd[8957]:  level=error msg="Failed to delete sandbox cdfce5c36c9319e04d78f8d729a26e93514b8ce09dc112cc95e1a9b51eb91147 while trying to cleanup
dockerd[8957]:  level=info msg="Removing stale endpoint labgate-classifier-workers (7b841784d865d6f6e2a647a7311b2198d9246e6c40e3568ac701af15cb1996
dockerd[8957]:  level=info msg="Fixing inconsistent endpoint_cnt for network labgate. Expected=0, Actual=2"
dockerd[8957]:  level=info msg="Removing stale endpoint labgate_meteor (00dd85a246fa4bf1279f19afb4a8e0424c607bda84fd81a40b9c3e37c41ebb39)"
dockerd[8957]:  level=info msg="Fixing inconsistent endpoint_cnt for network bridge. Expected=0, Actual=1"
dockerd[8957]:  level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a pr
dockerd[8957]:  level=warning msg="Failed getting network for ep 6908fe3dbfa7e246943d23a0d997cdaa0767c30b33366626733f2c765ed06fb1 during sandbox c
dockerd[8957]:  level=warning msg="Failed detaching sandbox cdfce5c36c9319e04d78f8d729a26e93514b8ce09dc112cc95e1a9b51eb91147 from endpoint ec4ce27
dockerd[8957]:  level=warning msg="Failed deleting endpoint ec4ce27ba19489124e9e94d0350cb6b4ca436373f365ffb5a1cfeddfaf8693b4: failed to get endpoi
dockerd[8957]:  level=error msg="failed to cleanup up stale network sandbox for container ae73cb3f0eed4c1fe5dcd4dae9cc9106fbfaa66101938c5663e89fcf
dockerd[8957]:  level=warning msg="Failed to allocate and map port 9003-9003: Bind for 127.0.0.1:9003 failed: port is already allocated"
dockerd[8957]:  level=error msg="9e8e0a015e130064c8d6dafefce869033feccacc299c3d55a05c93366fe2794a cleanup: failed to delete container from contain
dockerd[8957]:  level=error msg="Failed to start container 9e8e0a015e130064c8d6dafefce869033feccacc299c3d55a05c93366fe2794a: driver failed program
dockerd[8957]: panic: runtime error: invalid memory address or nil pointer dereference
dockerd[8957]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x56278d414055]
goroutine 307 [running]:
dockerd[8957]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*endpoint).addServiceInfoToCluster(0xc42025ef20, 0xc4201cb000, 0x0, 0x0)
dockerd[8957]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:586 +0xa5
dockerd[8957]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*sandbox).EnableService(0xc4201cb000, 0x0, 0x0)
dockerd[8957]:         /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox.go:701 +0x16f
dockerd[8957]: github.com/docker/docker/daemon.(*Daemon).ActivateContainerServiceBinding(0xc4203aac40, 0xc4201f1970, 0x9, 0x0, 0x0)
dockerd[8957]:         /go/src/github.com/docker/docker/daemon/container_operations.go:1118 +0x89
dockerd[8957]: github.com/docker/docker/daemon.(*Daemon).connectToNetwork(0xc4203aac40, 0xc4201c6b40, 0x56278da5ab44, 0x6, 0xc420562300, 0x0, 0x0, 0x0)
dockerd[8957]:         /go/src/github.com/docker/docker/daemon/container_operations.go:796 +0x9db
dockerd[8957]: github.com/docker/docker/daemon.(*Daemon).allocateNetwork(0xc4203aac40, 0xc4201c6b40, 0x0, 0x0)
dockerd[8957]:         /go/src/github.com/docker/docker/daemon/container_operations.go:540 +0xa31
dockerd[8957]: github.com/docker/docker/daemon.(*Daemon).initializeNetworking(0xc4203aac40, 0xc4201c6b40, 0x0, 0x0)
dockerd[8957]:         /go/src/github.com/docker/docker/daemon/container_operations.go:953 +0xa2
dockerd[8957]: github.com/docker/docker/daemon.(*Daemon).containerStart(0xc4203aac40, 0xc4201c6b40, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0)
dockerd[8957]:         /go/src/github.com/docker/docker/daemon/start.go:150 +0x2c2
dockerd[8957]: github.com/docker/docker/daemon.(*Daemon).restore.func2(0xc420c3be10, 0xc4203aac40, 0xc4206cec00, 0xc4201c6b40, 0xc4202c6cc0)
dockerd[8957]:         /go/src/github.com/docker/docker/daemon/daemon.go:388 +0x2d4
dockerd[8957]: created by github.com/docker/docker/daemon.(*Daemon).restore
dockerd[8957]:         /go/src/github.com/docker/docker/daemon/daemon.go:368 +0x1025
systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
systemd[1]: Failed to start Docker Application Container Engine.
systemd[1]: docker.service: Unit entered failed state.
systemd[1]: docker.service: Failed with result 'exit-code'.
systemd[1]: docker.service: Service hold-off time over, scheduling restart.
systemd[1]: Stopped Docker Application Container Engine.
systemd[1]: docker.service: Start request repeated too quickly.
systemd[1]: Failed to start Docker Application Container Engine.

If /var/lib/docker or /var/lib/docker/aufs rename of delete start normally, but without containers.
The error was first on the old version, but nothing has changed after the update

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/networkingNetworkingkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.version/18.06

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions