Pulling image fails - failed to unmount temp mount - device or resource busy: unknown #5538
Closed as not planned
Description
Description
Trying to install Mirantis Kubernetes Engine on RHEL 7.9. Any time I run ctr i pull <image>, it fails with
Steps to reproduce the issue:
- ctr i pull docker.io/mirantis/ucp-hyperkube:3.3.9
ctr i pull docker.io/mirantis/ucp-hyperkube:3.3.9
docker.io/mirantis/ucp-hyperkube:3.3.9: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:c33dd9a587560191b55d6c6831738f977f4452505199651756ba491a34b66dcb: exists |++++++++++++++++++++++++++++++++++++++|
config-sha256:77e43fa8731d2a8655e921fd049f5c113075df0e1f428167155bafdc17c961df: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:44c5ded36b57e206e3c8c6ee7891f61db2afc1129ce970a953b839e5a2422e44: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f621385815828b7b813f6bb3a9558a0fdf1a47053b9788454c901d9acce3fe92: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:b5ee2f548017637d9b88ff01690e9c6ee19bc2a8d3eca7d2e99de32d02f22520: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:c804e25559599c2c08ce025eca93de83fdbd117a92251e9ac9b51a591d2aaaa6: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:20be0bce70df6fe6559c9a1dc211895c1a7f67a5f5e921c62879734fd54323ca: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7847f72fd3610206d55e593bd31f246aefdf24cc23871727955c8a213f8f4e75: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:ba09177b274d4003591a0997e64a688a97c9fb1ccb75e99e5e835d69a2ac60a0: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:72187ecbd309203499a9ad877c9717298039ca8c2a3db48164b1de4c6c69d0c9: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4c8fa809f39139b264c2aeb552b663b7be0251f334ad0528ee66949969585377: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6f1e41900c4abd1b502f871096aade786cf82ba999181425f8eec1d0b3e9c1ae: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:57671312ef6fdbecf340e5fed0fb0863350cd806c92b1fdd7978adbd02afc5c3: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3c040335d2d6b2f692aa1e061fabb7a36f1907fc0ff2a2b0171d12c6195334fa: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5e9250ddb7d0fa6d13302c7c3e6a0aa40390e42424caed1e5289077ee4054709: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6d5336bd2edf97742c3b91459ed76ab5393b47109686658d49c4ccb12d7c42e9: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:be0d493fa65e37bba9bcdfa4e10a87e19deb1ab8cb64d1c414eca47d8f624da1: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:345e3491a907bb7c6f1bdddcf4a94284b8b6ddd77eb7d93f09432b17b20f2bbe: exists |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.6 s total: 0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:c33dd9a587560191b55d6c6831738f977f4452505199651756ba491a34b66dcb...
INFO[0006] apply failure, attempting cleanup error="failed to extract layer sha256:ccdbb80308cc5ef43b605ac28fac29c6a597f89f5a169bbedbb8dec29c987439: failed to unmount /group/app/containerd/tmpmounts/containerd-mount093668013: failed to unmount target /group/app/containerd/tmpmounts/containerd-mount093668013: device or resource busy: unknown" key="extract-906700158-sFgo sha256:ccdbb80308cc5ef43b605ac28fac29c6a597f89f5a169bbedbb8dec29c987439"
ctr: failed to extract layer sha256:ccdbb80308cc5ef43b605ac28fac29c6a597f89f5a169bbedbb8dec29c987439: failed to unmount /group/app/containerd/tmpmounts/containerd-mount093668013: failed to unmount target /group/app/containerd/tmpmounts/containerd-mount093668013: device or resource busy: unknown
After it fails, the directory still shows it is mounted:
/dev/mapper/rhel-group on /group/app/containerd/tmpmounts/containerd-mount093668013 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
Thing is, /dev/mapper/rhel-group is already mounted at a higher mount point:
/dev/mapper/rhel-group on /group type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
Describe the results you expected:
Pull completes successfully.
What version of containerd are you using:
containerd containerd.io 1.3.9 ea765aba0d05254012b0b9e595e995c09186427f
Any other relevant information (runC version, CRI configuration, OS/Kernel version, etc.):
runc --version
runc version 1.0.0-rc10 commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd spec: 1.0.1-dev
crictl info
$ crictl info
uname -a
Linux localhost 3.10.0-1160.24.1.el7.x86_64 #1 SMP Thu Mar 25 21:21:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux