Skip to content

Container crashed, cannot stop, cannot rm, lead to docker daemon crash #20997

@S-YOU

Description

@S-YOU

Output of docker version:

Client:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   c3959b1
 Built:        Mon Feb 22 22:37:33 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 116
Server Version: 1.10.2
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 184
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.1.18-boot2docker
Operating System: Boot2Docker 1.10.2 (TCL 6.4.1); master : 611be10 - Mon Feb 22 22:47:06 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 995.7 MiB
Name: d
ID: INFO:Q6YE:RHPH:XWDV:JHHE:ZPUL:5BAS:3HO2:JG2U:3CVD:5OQU:ZLMG

Provide additional environment details (AWS, VirtualBox, physical, etc.):
VirtualBox

List the steps to reproduce the issue:

  1. docker stop CONTAINER_ID or docker rm -f CONTAINER_ID

Describe the results you received:
Daemon Crashed, container still there, tried several times, including reboot of server and system.

Describe the results you expected:
Container should stop it

Provide additional info you think is important:
Some logs

docker@d:~$ docker -v
Docker version 1.10.2, build c3959b1
docker@d:~$ sudo docker daemon &
docker@d:~$ INFO[0000] [graphdriver] using prior storage driver "aufs" 
INFO[0000] Graph migration to content-addressability took 0.00 seconds 
WARN[0000] driver error deleting endpoint name_app_1 : endpoint not found: db3f038b147570597f1a8409a8b7cc4d8719db74409d20692b115648fe91dc3f 
INFO[0000] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
WARN[0000] Your kernel does not support cgroup blkio weight 
WARN[0000] Your kernel does not support cgroup blkio weight_device 
INFO[0000] Loading containers: start.                   
.ERRO[0000] Error unmounting container 1a55efa22c8c58614784f8aae1533a703cf55232615211096480ac9dabc9eb7a: not mounted 

INFO[0001] Loading containers: done.                    
INFO[0001] Daemon has completed initialization          
INFO[0001] Docker daemon                                 commit=c3959b1 execdriver=native-0.2 graphdriver=aufs version=1.10.2
INFO[0001] API listen on /var/run/docker.sock           

docker@d:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                  PORTS                          NAMES
1a55efa22c8c        noku                "/bin/sh -c 'IP=$(tai"   20 hours ago        Up Less than a second   80/tcp, 0.0.0.0:443->443/tcp   name_app_1
docker@d:~$ docker stop 1a55efa22c8c
panic: runtime error: index out of range

goroutine 106 [running]:
github.com/docker/libnetwork/etchosts.Delete(0xc820242850, 0x6a, 0xc820a4cdc0, 0x2, 0x2, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/etchosts/etchosts.go:154 +0x543
github.com/docker/libnetwork.(*sandbox).deleteHostsEntries(0xc82060e380, 0xc820a4cdc0, 0x2, 0x2)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox.go:815 +0x6c
github.com/docker/libnetwork.(*endpoint).sbLeave(0xc8208e6500, 0x7f1190a33ee8, 0xc82060e380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/endpoint.go:599 +0xb35
github.com/docker/libnetwork.(*endpoint).Leave(0xc82035b300, 0x7f1190a33ee8, 0xc82060e380, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/endpoint.go:536 +0x1bf
github.com/docker/libnetwork.(*sandbox).delete(0xc82060e380, 0xc8207aea00, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox.go:199 +0xad4
github.com/docker/libnetwork.(*sandbox).Delete(0xc82060e380, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox.go:166 +0x32
github.com/docker/docker/daemon.(*Daemon).releaseNetwork(0xc820390600, 0xc820b421c0)
    /go/src/github.com/docker/docker/daemon/container_operations_unix.go:1013 +0x4f1
github.com/docker/docker/daemon.(*Daemon).Cleanup(0xc820390600, 0xc820b421c0)
    /go/src/github.com/docker/docker/daemon/start.go:159 +0x3d
github.com/docker/docker/container.(*containerMonitor).Close(0xc820355b80, 0x0, 0x0)
    /go/src/github.com/docker/docker/container/monitor.go:127 +0x68
github.com/docker/docker/container.(*containerMonitor).start.func1(0xc82085bd5f, 0xc820355b80, 0xc82085bd60)
    /go/src/github.com/docker/docker/container/monitor.go:158 +0xaa
github.com/docker/docker/container.(*containerMonitor).start(0xc820355b80, 0x0, 0x0)
    /go/src/github.com/docker/docker/container/monitor.go:232 +0xb63
github.com/docker/docker/container.(*containerMonitor).(github.com/docker/docker/container.start)-fm(0x0, 0x0)
    /go/src/github.com/docker/docker/container/monitor.go:101 +0x2e
github.com/docker/docker/pkg/promise.Go.func1(0xc8206fc960, 0xc820ae2210)
    /go/src/github.com/docker/docker/pkg/promise/promise.go:8 +0x1d
created by github.com/docker/docker/pkg/promise.Go
    /go/src/github.com/docker/docker/pkg/promise/promise.go:9 +0x5d

goroutine 1 [chan receive]:
main.(*DaemonCli).CmdDaemon(0xc8202ea4a0, 0xc82000a070, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/docker/docker/daemon.go:305 +0x20a7
reflect.callMethod(0xc82060ae40, 0xc82085fc68)
    /usr/local/go/src/reflect/value.go:628 +0x1fc
reflect.methodValueCall(0xc82000a070, 0x0, 0x0, 0x1, 0xc82060ae40, 0x0, 0x0, 0xc82060ae40, 0xc82060ca20, 0x4745b4, ...)
    /usr/local/go/src/reflect/asm_amd64.s:29 +0x36
github.com/docker/docker/cli.(*Cli).Run(0xc82060adb0, 0xc82000a070, 0x1, 0x1, 0x0, 0x0)
    /go/src/github.com/docker/docker/cli/cli.go:89 +0x383
main.main()
    /go/src/github.com/docker/docker/docker/docker.go:63 +0x43c

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 18 [syscall]:
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 38 [select, locked to thread]:
runtime.gopark(0x1a01bd0, 0xc820027728, 0x16ffa58, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc820027728, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc820027728)
    /usr/local/go/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:227 +0x353
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 39 [chan receive]:
github.com/docker/docker/daemon.setupDumpStackTrap.func1(0xc82014cae0)
    /go/src/github.com/docker/docker/daemon/debugtrap_unix.go:17 +0x4d
created by github.com/docker/docker/daemon.setupDumpStackTrap
    /go/src/github.com/docker/docker/daemon/debugtrap_unix.go:20 +0x12a

goroutine 170 [IO wait]:
net.runtime_pollWait(0x7f1191ac6a18, 0x72, 0xc82000e1a0)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820bce3e0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820bce3e0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc820bce380, 0x0, 0x7f1191ac5828, 0xc820a1fd00)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820028390, 0xc820028458, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
github.com/miekg/dns.(*Server).serveTCP(0xc82006a680, 0xc820028390, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/miekg/dns/server.go:435 +0x1c2
github.com/miekg/dns.(*Server).ActivateAndServe(0xc82006a680, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/miekg/dns/server.go:363 +0x331
github.com/docker/libnetwork.(*resolver).Start.func2(0xc82006a680)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/resolver.go:121 +0x21
created by github.com/docker/libnetwork.(*resolver).Start
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/resolver.go:122 +0x26a

goroutine 6 [select]:
github.com/docker/libnetwork.(*controller).watchLoop(0xc820378370)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/store.go:409 +0x140
created by github.com/docker/libnetwork.(*controller).startWatch
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/store.go:423 +0x100

goroutine 31 [IO wait]:
net.runtime_pollWait(0x7f1191ac6b98, 0x72, 0xc82000e1a0)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8203d5410, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8203d5410, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8203d53b0, 0x0, 0x7f1191ac5828, 0xc820b3e000)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*UnixListener).AcceptUnix(0xc82000aa20, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/unixsock_posix.go:304 +0x53
net.(*UnixListener).Accept(0xc82000aa20, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/unixsock_posix.go:314 +0x41
github.com/docker/libnetwork.(*controller).acceptClientConnections(0xc820378370, 0xc8203d5340, 0x63, 0x7f1191ac5d20, 0xc82000aa20)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox_externalkey_unix.go:130 +0x45
created by github.com/docker/libnetwork.(*controller).startExternalKeyListener
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox_externalkey_unix.go:124 +0x28a

goroutine 30 [select]:
github.com/docker/libnetwork/osl.removeUnusedPaths()
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/osl/namespace_linux.go:74 +0x490
created by github.com/docker/libnetwork/osl.createBasePath
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/osl/namespace_linux.go:59 +0xbe

goroutine 169 [IO wait]:
net.runtime_pollWait(0x7f1191ac6ad8, 0x72, 0xc82000e1a0)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820bc1800, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820bc1800, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).readMsg(0xc820bc17a0, 0xc820bafa00, 0x200, 0x200, 0xc820b87e00, 0x28, 0x28, 0xffffffffffffffff, 0x0, 0x0, ...)
    /usr/local/go/src/net/fd_unix.go:286 +0x33d
net.(*UDPConn).ReadMsgUDP(0xc820028370, 0xc820bafa00, 0x200, 0x200, 0xc820b87e00, 0x28, 0x28, 0x200, 0x1241e80, 0x1, ...)
    /usr/local/go/src/net/udpsock_posix.go:96 +0x171
github.com/miekg/dns.ReadFromSessionUDP(0xc820028370, 0xc820bafa00, 0x200, 0x200, 0x200, 0x200, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/miekg/dns/udp.go:47 +0xc1
github.com/miekg/dns.(*Server).readUDP(0xc82006a5b0, 0xc820028370, 0x77359400, 0x0, 0x0, 0x0, 0xc82006a668, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/miekg/dns/server.go:605 +0x190
github.com/miekg/dns.(*defaultReader).ReadUDP(0xc820028450, 0xc820028370, 0x77359400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/miekg/dns/server.go:230 +0x5f
github.com/miekg/dns.(*Server).serveUDP(0xc82006a5b0, 0xc820028370, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/miekg/dns/server.go:478 +0x1fe
github.com/miekg/dns.(*Server).ActivateAndServe(0xc82006a5b0, 0x0, 0x0)
    /go/src/github.com/docker/docker/vendor/src/github.com/miekg/dns/server.go:354 +0x23c
github.com/docker/libnetwork.(*resolver).Start.func1(0xc82006a5b0)
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/resolver.go:115 +0x21
created by github.com/docker/libnetwork.(*resolver).Start
    /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/resolver.go:116 +0x163

goroutine 175 [runnable]:
sync.runtime_Semacquire(0xc8202601c4)
    /usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*Mutex).Lock(0xc8202601c0)
    /usr/local/go/src/sync/mutex.go:82 +0x1c4
github.com/docker/docker/container.(*State).getExitCode(0xc8202601c0, 0x2540be400)
    /go/src/github.com/docker/docker/container/state.go:176 +0x28
github.com/docker/docker/container.(*State).WaitStop(0xc8202601c0, 0x2540be400, 0xf, 0x0, 0x0)
    /go/src/github.com/docker/docker/container/state.go:156 +0xf8
github.com/docker/docker/daemon.(*Daemon).containerStop(0xc820390600, 0xc820b421c0, 0xa, 0x0, 0x0)
    /go/src/github.com/docker/docker/daemon/stop.go:50 +0x199
github.com/docker/docker/daemon.(*Daemon).ContainerStop(0xc820390600, 0xc820a4c917, 0xc, 0xa, 0x0, 0x0)
    /go/src/github.com/docker/docker/daemon/stop.go:25 +0x136
github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersStop(0xc820ad8900, 0x7f1190a3a3b0, 0xc820b12600, 0x7f1190a3a280, 0xc82011abb0, 0xc8208082a0, 0xc820b12360, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/router/container/container_routes.go:196 +0x18b
github.com/docker/docker/api/server/router/container.(*containerRouter).(github.com/docker/docker/api/server/router/container.postContainersStop)-fm(0x7f1190a3a3b0, 0xc820b12600, 0x7f1190a3a280, 0xc82011abb0, 0xc8208082a0, 0xc820b12360, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/router/container/container.go:51 +0x74
github.com/docker/docker/api/server.versionMiddleware.func1(0x7f1190a3a3b0, 0xc820b12600, 0x7f1190a3a280, 0xc82011abb0, 0xc8208082a0, 0xc820b12360, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/middleware.go:155 +0x83a
github.com/docker/docker/api/server.(*Server).corsMiddleware.func1(0x7f1191ac5168, 0xc82000f690, 0x7f1190a3a280, 0xc82011abb0, 0xc8208082a0, 0xc820b12360, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/middleware.go:134 +0xfa
github.com/docker/docker/api/server.(*Server).userAgentMiddleware.func1(0x7f1191ac5168, 0xc82000f690, 0x7f1190a3a280, 0xc82011abb0, 0xc8208082a0, 0xc820b12360, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/middleware.go:117 +0x4be
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x7f1190a3a280, 0xc82011abb0, 0xc8208082a0)
    /go/src/github.com/docker/docker/api/server/server.go:169 +0x2ad
net/http.HandlerFunc.ServeHTTP(0xc82074b360, 0x7f1190a3a280, 0xc82011abb0, 0xc8208082a0)
    /usr/local/go/src/net/http/server.go:1422 +0x3a
github.com/gorilla/mux.(*Router).ServeHTTP(0xc8209f00a0, 0x7f1190a3a280, 0xc82011abb0, 0xc8208082a0)
    /go/src/github.com/docker/docker/vendor/src/github.com/gorilla/mux/mux.go:98 +0x29e
github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc820c0f670, 0x7f1190a3a280, 0xc82011abb0, 0xc8208082a0)
    /go/src/github.com/docker/docker/api/server/router_swapper.go:29 +0x72
net/http.serverHandler.ServeHTTP(0xc82014ca80, 0x7f1190a3a280, 0xc82011abb0, 0xc8208082a0)
    /usr/local/go/src/net/http/server.go:1862 +0x19e
net/http.(*conn).serve(0xc82011ab00)
    /usr/local/go/src/net/http/server.go:1361 +0xbee
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:1910 +0x3f6

goroutine 88 [chan receive]:
github.com/docker/docker/daemon.(*statsCollector).run(0xc8209ecf00)
    /go/src/github.com/docker/docker/daemon/stats_collector_unix.go:99 +0xcd
created by github.com/docker/docker/daemon.(*Daemon).newStatsCollector
    /go/src/github.com/docker/docker/daemon/stats_collector_unix.go:38 +0x1af

goroutine 89 [chan receive]:
github.com/docker/docker/daemon.(*Daemon).execCommandGC(0xc820390600)
    /go/src/github.com/docker/docker/daemon/exec.go:237 +0x8e
created by github.com/docker/docker/daemon.NewDaemon
    /go/src/github.com/docker/docker/daemon/daemon.go:807 +0x30f1

goroutine 137 [chan receive]:
main.setupConfigReloadTrap.func1(0xc820a569c0, 0x17dfbc0, 0x17, 0xc820148580, 0xc820898000)
    /go/src/github.com/docker/docker/docker/daemon_unix.go:62 +0x68
created by main.setupConfigReloadTrap
    /go/src/github.com/docker/docker/docker/daemon_unix.go:67 +0x162

goroutine 138 [chan receive]:
github.com/docker/docker/api/server.(*Server).serveAPI(0xc82014ca20, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/server.go:102 +0x1af
github.com/docker/docker/api/server.(*Server).Wait(0xc82014ca20, 0xc820a56a20)
    /go/src/github.com/docker/docker/api/server/server.go:217 +0x2b
created by main.(*DaemonCli).CmdDaemon
    /go/src/github.com/docker/docker/docker/daemon.go:287 +0x1f7a

goroutine 139 [chan receive]:
github.com/docker/docker/pkg/signal.Trap.func1(0xc820a56a80, 0xc820ad95f0)
    /go/src/github.com/docker/docker/pkg/signal/trap.go:29 +0x71
created by github.com/docker/docker/pkg/signal.Trap
    /go/src/github.com/docker/docker/pkg/signal/trap.go:55 +0x21a

goroutine 115 [IO wait]:
net.runtime_pollWait(0x7f1191ac6c58, 0x72, 0xc82000e1a0)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8203c94f0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8203c94f0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8203c9490, 0x0, 0x7f1191ac5828, 0xc8209101c0)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*UnixListener).AcceptUnix(0xc8202ea9e0, 0xc820034dd8, 0x0, 0x0)
    /usr/local/go/src/net/unixsock_posix.go:304 +0x53
net.(*UnixListener).Accept(0xc8202ea9e0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/unixsock_posix.go:314 +0x41
net/http.(*Server).Serve(0xc82014ca80, 0x7f1191ac5d20, 0xc8202ea9e0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1887 +0xb3
github.com/docker/docker/api/server.(*HTTPServer).Serve(0xc8202eaa80, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/server.go:121 +0x47
github.com/docker/docker/api/server.(*Server).serveAPI.func1(0xc820c02960, 0xc8202eaa80)
    /go/src/github.com/docker/docker/api/server/server.go:94 +0x13d
created by github.com/docker/docker/api/server.(*Server).serveAPI
    /go/src/github.com/docker/docker/api/server/server.go:98 +0x148
Failed to stop container (1a55efa22c8c): An error occurred trying to connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.22/containers/1a55efa22c8c/stop?t=10: EOF
[1]+  Done(2)                    sudo docker daemon

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/networkingNetworkingkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions