Dockerd fail to receive containerd's exit signal when container process have be oom kill. #33192
Description
Description
Dockerd fail to receive containerd's exit signal when container process have be oom kill. Container's process, runc and contaierd has been all exit. But docker ps show the container is UP. And docker stop, docker kill, docker exec no longer work as expect. such as below:
docker exec -it MySQL-for-Developer_MySQL-for-Developer_1 sh
rpc error: code = 2 desc = containerd: container not found
Steps to reproduce the issue:
- run an container with memory limit.
- container use memory more than memory limit
- container process oom kill by kernel
Describe the results you received:
docker ps show the container still "UP", and docker stop, docker kill, docker exec no longer work as expect.
docker exec -it MySQL-for-Developer_MySQL-for-Developer_1 sh
rpc error: code = 2 desc = containerd: container not found
Describe the results you expected:
When container's process has be killed, the docker container should be "Exit", Not "Up".
Additional information you deem important (e.g. issue happens only occasionally):
issue happens only occasionally.
Output of docker version:
Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: 0801b25
Built: Tue Mar 28 08:41:15 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: 0801b25
Built: Tue Mar 28 08:41:15 2017
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 11
Running: 11
Paused: 0
Stopped: 0
Images: 14
Server Version: 17.03.1-ce
Storage Driver: overlay
Backing Filesystem: extfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local ossfs acd nas
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.4.41-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.67 GiB
Name: app6
ID: xxx
Docker Root Dir: /work1/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Additional environment details (AWS, VirtualBox, physical, etc.):
physical
docker daemon's log:
May 15 14:36:28 app6 dockerd: time="2017-05-15T14:36:28.987367854+08:00" level=warning msg="container kill failed because of 'container not found' or 'no such process': Cannot kill container 3f08d72500fc972a4f194afa5b7038dfe7a2be1d4b7355d857b82bd8b0b5332d: rpc error: code = 2 desc = containerd: container not found"
May 15 14:36:29 app6 dockerd: time="2017-05-15T14:36:29.617753767+08:00" level=error msg="collecting stats for 3f08d72500fc972a4f194afa5b7038dfe7a2be1d4b7355d857b82bd8b0b5332d: rpc error: code = 2 desc = containerd: container not found"
May 15 14:36:30 app6 dockerd: time="2017-05-15T14:36:30.617792626+08:00" level=error msg="collecting stats for 3f08d72500fc972a4f194afa5b7038dfe7a2be1d4b7355d857b82bd8b0b5332d: rpc error: code = 2 desc = containerd: container not found"
May 15 14:36:31 app6 dockerd: time="2017-05-15T14:36:31.617938914+08:00" level=error msg="collecting stats for 3f08d72500fc972a4f194afa5b7038dfe7a2be1d4b7355d857b82bd8b0b5332d: rpc error: code = 2 desc = containerd: container not found"