Skip to content

EmptyDir can't be removed when pod become evicted #35406

@mdshuai

Description

@mdshuai

Description of problem:
Mount a emptyDir in pod /tmp then create large file in emptyDir to trigger nodefs.available eviction, when pod become Evicted, the large file can't be removed, node always keep in DiskPressure

Kubernetes version (use kubectl version):
v1.5.0-alpha.1.804+a7db9bccb56abf-dirty

What happened:
File in emptyDir can't be removed after pod become Evicted, and node always keep in DiskPressure=True

What you expected to happen:
File in emptyDir should be removed after pod become Evicted, and node become DiskPressure=false after a period.

How to reproduce it (as minimally and precisely as possible):

1. configure kubelet with eviction-soft
--eviction-soft="nodefs.available<3Gi"
--eviction-soft-grace-period="nodefs.available=20s"
--eviction-pressure-transition-period=30s

2. Create a pod with emptyDir
$ kubectl create -f https://raw.githubusercontent.com/mdshuai/testfile-openshift/master/k8s/hello-pod.yaml

3. Create a large file in pod /tmp dir
$ kubectl exec hello-pod -- dd if=/dev/zero of=/tmp/test1 bs=10M count=1024

4. Wait pod become Evicted then check node status and file in emptyDir
[root@ip-172-18-6-200 kubernetes]# kubectl get pod
NAME        READY     STATUS    RESTARTS   AGE
hello-pod   0/1       Evicted   0          23m
[root@ip-172-18-6-200 kubernetes]# kubectl describe node 127.0.0.1|grep DiskPressure
  DiskPressure      True    Mon, 24 Oct 2016 02:56:22 -0400     Mon, 24 Oct 2016 02:33:01 -0400     KubeletHasDiskPressure      kubelet has disk pressure
  29m       29m     2   {kubelet 127.0.0.1}         Normal      NodeHasNoDiskPressure   Node 127.0.0.1 status is now: NodeHasNoDiskPressure
  23m       23m     1   {kubelet 127.0.0.1}         Normal      NodeHasDiskPressure Node 127.0.0.1 status is now: NodeHasDiskPressure
[root@ip-172-18-6-200 kubernetes]# ls -lh /var/lib/kubelet/pods/8095536f-99b3-11e6-9bc0-0ebd788c9e1f/volumes/kubernetes.io~empty-dir/tmp/test1
-rw-r--r--. 1 root root 5.3G Oct 24 02:33 /var/lib/kubelet/pods/8095536f-99b3-11e6-9bc0-0ebd788c9e1f/volumes/kubernetes.io~empty-dir/tmp/test1

Anything else do we need to know:
When delete the pod the large file in emptyDir then removed.

Metadata

Metadata

Assignees

Labels

area/kubeletsig/nodeCategorizes an issue or PR as relevant to SIG Node.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions