Creating task failed with "failed to find pid" #4009
Description
Description
Sometimes (about 1 in 100 pods) containerd fails to create task and logs error failed to mountat: failed to find pid=12345 process: no child processes: unknown. Usually the problem goes away when re-creating the pod.
Not able to reproduce but the error has been observed in one of our production clusters.
Output of containerd --version:
containerd github.com/containerd/containerd v1.2.5 bb71b10fd8f58240ca47fbb579b9d1028eea7c84
Extra info
OS: CoreOS 2247.5.0
Kernel: Linux 4.19.78-coreos
Error log:
Feb 10 19:43:42 ip-10-28-14-134.ec2.internal containerd[2138]: time="2020-02-10T19:43:42.282543411Z" level=error msg="Failed to pipe stderr of container "db585f7a4673817c358f9291e200076691d9fb9fe8cee5ae1efb430751adc3fc"" error="reading from a closed fifo"
Feb 10 19:43:42 ip-10-28-14-134.ec2.internal containerd[2138]: time="2020-02-10T19:43:42.284857193Z" level=error msg="Failed to pipe stdout of container "db585f7a4673817c358f9291e200076691d9fb9fe8cee5ae1efb430751adc3fc"" error="reading from a closed fifo"
Feb 10 19:43:42 ip-10-28-14-134.ec2.internal containerd[2138]: time="2020-02-10T19:43:42.291562435Z" level=error msg="StartContainer for "db585f7a4673817c358f9291e200076691d9fb9fe8cee5ae1efb430751adc3fc" failed" error="failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/829/work upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/829/fs lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/630/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/604/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/603/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/602/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/601/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/600/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/599/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/598/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/597/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/581/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/579/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/578/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/555/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/554/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/553/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/552/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/551/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/550/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/549/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/548/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/547/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/546/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/545/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/544/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/543/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/542/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/540/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/539/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/537/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/536/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/535/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/534/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/533/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/532/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/531/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/530/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/529/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/528/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/527/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/526/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/525/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/524/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/523/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/522/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/521/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/520/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/519/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/518/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/517/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/516/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/515/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/514/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/513/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/512/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/511/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/510/fs]}: failed to mountat: failed to find pid=29186 process: no child processes: unknown"