Podman invocations using different --roots cannot share an --imagestore #2257
Description
Issue Description
I suspect this was never an intended feature, but our use case is CI runners. We have hundreds of replicas, each of which has podman installed. Jobs always start with container pulls (which is the slowest part of the job). If a job pulls an image, we'd like for it to be written to a shared imagestore for subsequent jobs to reuse.
I created a imgs volume on the host, which I mounted into all of the runners. Then, all the runners were given a storage.conf with the contents:
[storage]
driver = "overlay"
imagestore = "/home/runner/imgs"
My expectation was that any podman pull inside of a runner would read/write this shared imagestore. However, the pulls step on each other, corrupting the imagestore/root.
Steps to reproduce the issue
# pull with root1, shared imgstore (works)
podman --root /tmp/root1 --imagestore /tmp/imgs pull mirror.gcr.io/library/bash
# pull with root2, shared imgstore (doesn't work)
podman --root /tmp/root2 --imagestore /tmp/imgs pull mirror.gcr.io/library/bash
ERRO[0000] Image mirror.gcr.io/library/bash exists in local storage but may be corrupted (remove the image to resolve the issue): layer not known
Describe the results you received
Corrupted imagestore
Describe the results you expected
Non-corrupted imagestore
podman info output
host:
arch: amd64
buildahVersion: 1.37.6
cgroupControllers:
- cpuset
- cpu
- io
- memory
- hugetlb
- pids
- rdma
- misc
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.12-1.el9.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.12, commit: c0564282e9befb7804c3642230f8e94f1b2ba9f8'
cpuUtilization:
idlePercent: 95.6
systemPercent: 1.94
userPercent: 2.47
cpus: 2
databaseBackend: sqlite
distribution:
distribution: rhel
version: "9.4"
eventLogger: journald
freeLocks: 2033
hostname: slaffy-github1.fyre.ibm.com
idMappings:
gidmap: null
uidmap: null
kernel: 5.14.0-427.42.1.el9_4.x86_64
linkmode: dynamic
logDriver: journald
memFree: 357359616
memTotal: 3837394944
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns-1.12.1-1.el9.x86_64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.12.1
package: netavark-1.12.2-1.el9.x86_64
path: /usr/libexec/podman/netavark
version: netavark 1.12.2
ociRuntime:
name: crun
package: crun-1.16.1-1.el9.x86_64
path: /usr/bin/crun
version: |-
crun version 1.16.1
commit: afa829ca0122bd5e1d67f1f38e6cc348027e3c32
rundir: /run/user/0/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
pasta:
executable: /usr/bin/pasta
package: passt-0^20240806.gee36266-2.el9.x86_64
version: |
pasta 0^20240806.gee36266-2.el9.x86_64
Copyright Red Hat
GNU General Public License, version 2 or later
<https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: false
path: /run/podman/podman.sock
rootlessNetworkCmd: pasta
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.3.1-1.el9.x86_64
version: |-
slirp4netns version 1.3.1
commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
libslirp: 4.4.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.2
swapFree: 17118425088
swapTotal: 17175670784
uptime: 453h 46m 52.00s (Approximately 18.88 days)
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- registry.access.redhat.com
- registry.redhat.io
- docker.io
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 7
paused: 0
running: 7
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mountopt: nodev,metacopy=on
graphRoot: /var/lib/containers/storage
graphRootAllocated: 250059685888
graphRootUsed: 124696444928
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "false"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "true"
imageCopyTmpDir: /var/tmp
imageStore:
number: 1970
runRoot: /run/containers/storage
transientStore: false
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 5.2.2
Built: 1737721907
BuiltTime: Fri Jan 24 04:31:47 2025
GitCommit: ""
GoVersion: go1.22.9 (Red Hat 1.22.9-2.el9_5)
Os: linux
OsArch: linux/amd64
Version: 5.2.2Podman in a container
Yes
Privileged Or Rootless
Rootless
Upstream Latest Release
No
Additional environment details
No response
Additional information
No response