overlay-mode containerization breaks apt: Invalid cross-device link #13123
Description
Is this a BUG REPORT or FEATURE REQUEST?
/kind bug
Description
Running apt install libreadline-dev in certain container results in apt bailing out with error:
unable to install new version of './usr/include/readline': Invalid cross-device link
This is not reproducible with fuse-overlayfs, it is only reproducible with overlay. Image/container required to reproduce it is docker.io/kkharlamov/bugreport-enomem image.
Steps to reproduce the issue:
- Make sure you're in
overlaymode, e.g. runpodman info --debug | grep fuse-overlayfsand check that there's no output.overlayrequires kernel 5.16.0 or higher - Run
podman run --rm -it docker.io/kkharlamov/bugreport-enomem /bin/zsh - In container, execute
apt install libreadline-dev
Describe the results you received:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
gyp javascript-common libasan6:i386 libatomic1:i386 libboost-dev libboost1.65-dev libc-ares2 libc6-dev:i386 libgomp1:i386 libhttp-parser2.7.1 libitm1:i386 libjs-async libjs-inherits libjs-jquery libjs-node-uuid libjs-underscore libquadmath0:i386 libstdc++6:i386
libubsan1:i386 libuv1-dev libyaml-cpp0.5v5 linux-libc-dev:i386 nodejs-doc python-chardet python-pkg-resources
Use 'apt autoremove' to remove them.
Suggested packages:
readline-doc
The following NEW packages will be installed:
libreadline-dev
0 upgraded, 1 newly installed, 0 to remove and 148 not upgraded.
Need to get 133 kB of archives.
After this operation, 728 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libreadline-dev amd64 7.0-3 [133 kB]
Fetched 133 kB in 1s (190 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libreadline-dev:amd64.
(Reading database ... 156394 files and directories currently installed.)
Preparing to unpack .../libreadline-dev_7.0-3_amd64.deb ...
Unpacking libreadline-dev:amd64 (7.0-3) ...
dpkg: error processing archive /var/cache/apt/archives/libreadline-dev_7.0-3_amd64.deb (--unpack):
unable to install new version of './usr/include/readline': Invalid cross-device link
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
/var/cache/apt/archives/libreadline-dev_7.0-3_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
Describe the results you expected:
No errors
Workarounds:
-
Create
~/.config/containers/storage.confwith content:[storage] [storage.options] mount_program = "/usr/bin/fuse-overlayfs" -
(WARNING: this step will remove all images) execute
podman system reset -f
Output of podman version:
Version: 3.4.4
API Version: 3.4.4
Go Version: go1.17.4
Git Commit: f6526ada1025c2e3f88745ba83b8b461ca659933
Built: Thu Dec 9 21:30:40 2021
OS/Arch: linux/amd64
Output of podman info --debug:
host:
arch: amd64
buildahVersion: 1.23.1
cgroupControllers:
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.0.32-2
path: /usr/bin/conmon
version: 'conmon version 2.0.32, commit: 436b460d1586c2e4ab4e845448449ddd9136767a'
cpus: 4
distribution:
distribution: arch
version: unknown
eventLogger: journald
hostname: constantine-N61Ja
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 165536
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 165536
size: 65536
kernel: 5.16.0-zen1-1-zen
linkmode: dynamic
logDriver: journald
memFree: 374030336
memTotal: 8228016128
ociRuntime:
name: crun
package: /usr/bin/crun is owned by crun 1.4-1
path: /usr/bin/crun
version: |-
crun version 1.4
commit: 3daded072ef008ef0840e8eccb0b52a7efbd165d
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/user/1000/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
version: |-
slirp4netns version 1.1.12
commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
libslirp: 4.6.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.3
swapFree: 6222995456
swapTotal: 8053059584
uptime: 91h 40m 56.28s (Approximately 3.79 days)
plugins:
log:
- k8s-file
- none
- journald
network:
- bridge
- macvlan
volume:
- local
registries:
search:
- docker.io
- registry.fedoraproject.org
- quay.io
- registry.access.redhat.com
- registry.centos.org
store:
configFile: /home/constantine/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/constantine/.local/share/containers/storage
graphStatus:
Backing Filesystem: btrfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 0
runRoot: /run/user/1000/containers
volumePath: /home/constantine/.local/share/containers/storage/volumes
version:
APIVersion: 3.4.4
Built: 1639074640
BuiltTime: Thu Dec 9 21:30:40 2021
GitCommit: f6526ada1025c2e3f88745ba83b8b461ca659933
GoVersion: go1.17.4
OsArch: linux/amd64
Version: 3.4.4
Package info (e.g. output of rpm -q podman or apt list podman):
Name : podman
Version : 3.4.4-1
Description : Tool and library for running OCI-based containers in pods
Architecture : x86_64
URL : https://github.com/containers/podman
Licenses : Apache
Groups : None
Provides : None
Depends On : cni-plugins conmon containers-common crun fuse-overlayfs iptables libdevmapper.so=1.02-64 libgpgme.so=11-64 libseccomp.so=2-64 slirp4netns
Optional Deps : apparmor: for AppArmor support
btrfs-progs: support btrfs backend devices [installed]
catatonit: --init flag support
podman-docker: for Docker-compatible CLI
Required By : None
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 72.79 MiB
Packager : David Runge <dvzrv@archlinux.org>
Build Date : Чт 09 дек 2021 21:30:40
Install Date : Пт 17 дек 2021 23:29:19
Install Reason : Explicitly installed
Install Script : No
Validated By : Signature
Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)
Yes