Kubelet I/O usage abnormally high #23255
Description
After upgrading to Kubernetes 1.2.0, kubelet's du sub-process is using 95-99% of my read I/O constantly:
$ iotop
Total DISK READ : 321.10 K/s | Total DISK WRITE : 10.76 M/s
Actual DISK READ: 321.10 K/s | Actual DISK WRITE: 4.24 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
18979 be/7 root 321.10 K/s 0.00 B/s 0.00 % 99.12 % du -s -B 1 /
27430 be/4 9999 0.00 B/s 6.49 M/s 0.00 % 31.45 % Passenger RubyApp: /app/public (production) [utils.rb:110]
14566 be/4 9999 0.00 B/s 3.82 K/s 0.00 % 0.00 % nginx: worker process
14639 be/4 9999 0.00 B/s 3.82 K/s 0.00 % 0.00 % nginx: worker process
$ ps axjf | grep 18979 -B 2
1 7889 7889 7889 ? -1 Ssl 0 3:16 /opt/bin/kubelet --api-servers=https://k8s.example.com --config=/etc/kubernetes/manifests --cloud-provider=aws --cluster-dns=10.10.0.10 --cluster-domain=cluster.local --image-gc-high-threshold=70 --image-gc-low-threshold=50 --maximum-dead-containers=15 --allow-privileged=true --v=2
7889 8030 7889 7889 ? -1 S 0 0:01 \_ journalctl -k -f
7889 18979 7889 7889 ? -1 DN 0 0:08 \_ du -s -B 1 /
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.4G 0 7.4G 0% /dev
tmpfs 7.4G 0 7.4G 0% /dev/shm
tmpfs 7.4G 3.0M 7.4G 1% /run
tmpfs 7.4G 0 7.4G 0% /sys/fs/cgroup
/dev/xvda9 56G 33G 21G 62% /
/dev/xvda3 985M 551M 384M 59% /usr
tmpfs 7.4G 0 7.4G 0% /media
tmpfs 7.4G 0 7.4G 0% /tmp
/dev/xvda1 128M 66M 63M 52% /boot
/dev/xvda6 108M 60K 99M 1% /usr/share/oem