e2e tests: split imagestore test failed when setting STORAGE_FS=overlay #19748
Description
Issue Description
split imagestore test failed with the error message such as below.
Error: creating container storage: creating read-write layer with ID "xxx": Stat /tmp/podman_testxxx/root/overlay/xxx/diff: no such file or directory
This error looks like a bug.
However, I have not reproduced the error manually.
Steps to reproduce the issue
Run e2e tests with STORAGE_FS=overlay.
# FOCUS="podman pull and run on split imagestore" STORAGE_FS=overlay make localintegrationDescribe the results you received
Error log
> Enter [It] podman pull and run on split imagestore - /root/podman/test/e2e/pull_test.go:65 @ 08/25/23 10:58:22.034
Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay build --pull=never --tag splitstoretest build/basicalpine
STEP 1/1: FROM quay.io/libpod/alpine:latest
COMMIT splitstoretest
--> 961769676411
Successfully tagged localhost/splitstoretest:latest
Successfully tagged quay.io/libpod/alpine:latest
961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay save -o /tmp/podman_test3554367878/image.tar --format oci-archive splitstoretest
Copying blob sha256:03901b4a2ea88eeaad62dbe59b072b28b6efa00491962b8741081c5df50c65e0
Copying config sha256:4819930274ea2bd6ab611972f9928e07b84edcbfe1d2462fb2941957308dfae2
Writing manifest to image destination
Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay rmi splitstoretest
Untagged: localhost/splitstoretest:latest
Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay load --imagestore /tmp/podman_test3554367878/splitstore -q -i /tmp/podman_test3554367878/image.tar
Loaded image: localhost/splitstoretest:latest
Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay tag quay.io/libpod/busybox:latest busybox-test
Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay images --imagestore /tmp/podman_test3554367878/splitstore
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/libpod/registry 2.8.2 0030ba3d620c 2 weeks ago 24.6 MB
registry.fedoraproject.org/fedora-toolbox 36 5be0bf1ecd38 4 months ago 874 MB
quay.io/libpod/systemd-image 20230106 4f6126057b5e 7 months ago 132 MB
quay.io/libpod/alpine_nginx latest ecea49d99daa 16 months ago 23.1 MB
registry.fedoraproject.org/fedora-minimal 34 b85bdf5fbb1c 18 months ago 119 MB
quay.io/libpod/alpine_labels latest 5e9e9275e4d6 18 months ago 5.87 MB
quay.io/libpod/redis alpine 8835e6aeca99 2 years ago 32.3 MB
quay.io/libpod/busybox latest f0b02e9d092d 2 years ago 1.45 MB
quay.io/libpod/busybox latest f0b02e9d092d 2 years ago 1.45 MB
localhost/busybox-test latest f0b02e9d092d 2 years ago 1.45 MB
registry.k8s.io/pause 3.2 80d28bedfe5d 3 years ago 686 kB
localhost/splitstoretest latest 4819930274ea 4 years ago 5.85 MB
quay.io/libpod/alpine latest 961769676411 4 years ago 5.85 MB
quay.io/libpod/alpine latest 961769676411 4 years ago 5.85 MB
quay.io/libpod/alpine_healthcheck latest c1103abb47d6 4 years ago 10.4 MB
Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay rmi --imagestore /tmp/podman_test3554367878/splitstore busybox-test
Untagged: localhost/busybox-test:latest
Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay images
REPOSITORY TAG IMAGE ID CREATED SIZE R/O
quay.io/libpod/registry 2.8.2 0030ba3d620c 2 weeks ago 24.6 MB true
registry.fedoraproject.org/fedora-toolbox 36 5be0bf1ecd38 4 months ago 874 MB true
quay.io/libpod/systemd-image 20230106 4f6126057b5e 7 months ago 132 MB true
quay.io/libpod/alpine_nginx latest ecea49d99daa 16 months ago 23.1 MB true
registry.fedoraproject.org/fedora-minimal 34 b85bdf5fbb1c 18 months ago 119 MB true
quay.io/libpod/alpine_labels latest 5e9e9275e4d6 18 months ago 5.87 MB true
quay.io/libpod/redis alpine 8835e6aeca99 2 years ago 32.3 MB true
quay.io/libpod/busybox latest f0b02e9d092d 2 years ago 1.45 MB false
localhost/busybox-test latest f0b02e9d092d 2 years ago 1.45 MB false
quay.io/libpod/busybox latest f0b02e9d092d 2 years ago 1.45 MB true
registry.k8s.io/pause 3.2 80d28bedfe5d 3 years ago 686 kB true
quay.io/libpod/alpine latest 961769676411 4 years ago 5.85 MB false
quay.io/libpod/alpine latest 961769676411 4 years ago 5.85 MB true
quay.io/libpod/alpine_healthcheck latest c1103abb47d6 4 years ago 10.4 MB true
Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay run --name test --rm splitstoretest echo helloworld
time="2023-08-25T10:58:22+09:00" level=warning msg="The storage 'driver' option should be set in /tmp/podman_test3554367878/.config/containers/storage.conf. A driver was picked automatically."
Error: creating container storage: creating read-write layer with ID "49199fe425879e87febccf0dcf1c03bbcd28e8aae4b367c9220966ac03b4a966": Stat /tmp/podman_test3554367878/root/overlay/03901b4a2ea88eeaad62dbe59b072b28b6efa00491962b8741081c5df50c65e0/diff: no such file or directory
[FAILED] Expected
<int>: 125
to match exit code:
<int>: 0
In [It] at: /root/podman/test/e2e/pull_test.go:128 @ 08/25/23 10:58:22.607
Full Stack Trace
github.com/containers/podman/v4/test/e2e.glob..func72.5()
/root/podman/test/e2e/pull_test.go:128 +0x102a
< Exit [It] podman pull and run on split imagestore - /root/podman/test/e2e/pull_test.go:65 @ 08/25/23 10:58:22.607 (573ms)
> Enter [AfterEach] TOP-LEVEL - /root/podman/test/e2e/common_test.go:119 @ 08/25/23 10:58:22.607
Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay stop --all -t 0
Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay pod rm -fa -t 0
Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay rm -fa -t 0
< Exit [AfterEach] TOP-LEVEL - /root/podman/test/e2e/common_test.go:119 @ 08/25/23 10:58:22.702 (95ms)
Describe the results you expected
This test should succeed.
podman info output
host:
arch: amd64
buildahVersion: 1.32.0-dev
cgroupControllers:
- cpuset
- cpu
- io
- memory
- hugetlb
- pids
- misc
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.7-2.fc38.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.7, commit: '
cpuUtilization:
idlePercent: 99.98
systemPercent: 0.01
userPercent: 0.01
cpus: 36
databaseBackend: boltdb
distribution:
distribution: fedora
variant: server
version: "38"
eventLogger: journald
freeLocks: 2047
hostname: fedora38
idMappings:
gidmap: null
uidmap: null
kernel: 6.2.9-300.fc38.x86_64
linkmode: dynamic
logDriver: journald
memFree: 4698464256
memTotal: 8301707264
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns-1.7.0-1.fc38.x86_64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.7.0
package: netavark-1.7.0-1.fc38.x86_64
path: /usr/libexec/podman/netavark
version: netavark 1.7.0
ociRuntime:
name: crun
package: Unknown
path: /usr/local/bin/crun
version: |-
crun version 1.8.1.0.0.0.16-7be9e-dirty
commit: d7ee5496991c6ad40be6dd94796643cb0def9e3c
rundir: /run/user/0/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
os: linux
pasta:
executable: /usr/bin/pasta
package: passt-0^20230625.g32660ce-1.fc38.x86_64
version: |
pasta 0^20230625.g32660ce-1.fc38.x86_64
Copyright Red Hat
GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: false
path: /run/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.2.0-12.fc38.x86_64
version: |-
slirp4netns version 1.2.0
commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.3
swapFree: 8257269760
swapTotal: 8301572096
uptime: 19h 33m 7.00s (Approximately 0.79 days)
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- registry.fedoraproject.org
- registry.access.redhat.com
- docker.io
- quay.io
store:
configFile: /usr/share/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mountopt: nodev,metacopy=on
graphRoot: /var/lib/containers/storage
graphRootAllocated: 106229137408
graphRootUsed: 17473433600
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "false"
Supports d_type: "true"
Supports shifting: "true"
Supports volatile: "true"
Using metacopy: "true"
imageCopyTmpDir: /var/tmp
imageStore:
number: 0
runRoot: /run/containers/storage
transientStore: false
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 4.7.0-dev
Built: 1692837461
BuiltTime: Thu Aug 24 09:37:41 2023
GitCommit: 7c0e926820881583b181985b31f7485e1a32d24c
GoVersion: go1.20.6
Os: linux
OsArch: linux/amd64
Version: 4.7.0-devPodman in a container
No
Privileged Or Rootless
None
Upstream Latest Release
Yes
Additional environment details
Additional environment details
Additional information
When setting STORAGE_FS=vfs, this test successed.
# FOCUS="podman pull and run on split imagestore" STORAGE_FS=vfs make localintegratione2e test log
> Enter [It] podman pull and run on split imagestore - /root/podman/test/e2e/pull_test.go:65 @ 08/25/23 10:56:13.419
Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs build --pull=never --tag splitstoretest build/basicalpine
STEP 1/1: FROM quay.io/libpod/alpine:latest
COMMIT splitstoretest
--> 961769676411
Successfully tagged localhost/splitstoretest:latest
Successfully tagged quay.io/libpod/alpine:latest
961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs save -o /tmp/podman_test735789469/image.tar --format oci-archive splitstoretest
Copying blob sha256:03901b4a2ea88eeaad62dbe59b072b28b6efa00491962b8741081c5df50c65e0
Copying config sha256:4819930274ea2bd6ab611972f9928e07b84edcbfe1d2462fb2941957308dfae2
Writing manifest to image destination
Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs rmi splitstoretest
Untagged: localhost/splitstoretest:latest
Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs load --imagestore /tmp/podman_test735789469/splitstore -q -i /tmp/podman_test735789469/image.tar
Loaded image: localhost/splitstoretest:latest
Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs tag quay.io/libpod/busybox:latest busybox-test
Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs images --imagestore /tmp/podman_test735789469/splitstore
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/libpod/registry 2.8.2 0030ba3d620c 2 weeks ago 24.6 MB
registry.fedoraproject.org/fedora-toolbox 36 5be0bf1ecd38 4 months ago 874 MB
quay.io/libpod/systemd-image 20230106 4f6126057b5e 7 months ago 132 MB
quay.io/libpod/alpine_nginx latest ecea49d99daa 16 months ago 23.1 MB
registry.fedoraproject.org/fedora-minimal 34 b85bdf5fbb1c 18 months ago 119 MB
quay.io/libpod/alpine_labels latest 5e9e9275e4d6 18 months ago 5.87 MB
quay.io/libpod/redis alpine 8835e6aeca99 2 years ago 32.3 MB
quay.io/libpod/busybox latest f0b02e9d092d 2 years ago 1.45 MB
quay.io/libpod/busybox latest f0b02e9d092d 2 years ago 1.45 MB
localhost/busybox-test latest f0b02e9d092d 2 years ago 1.45 MB
registry.k8s.io/pause 3.2 80d28bedfe5d 3 years ago 686 kB
localhost/splitstoretest latest 4819930274ea 4 years ago 5.85 MB
quay.io/libpod/alpine latest 961769676411 4 years ago 5.85 MB
quay.io/libpod/alpine latest 961769676411 4 years ago 5.85 MB
quay.io/libpod/alpine_healthcheck latest c1103abb47d6 4 years ago 10.4 MB
Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs rmi --imagestore /tmp/podman_test735789469/splitstore busybox-test
Untagged: localhost/busybox-test:latest
Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs images
REPOSITORY TAG IMAGE ID CREATED SIZE R/O
quay.io/libpod/registry 2.8.2 0030ba3d620c 2 weeks ago 24.6 MB true
registry.fedoraproject.org/fedora-toolbox 36 5be0bf1ecd38 4 months ago 874 MB true
quay.io/libpod/systemd-image 20230106 4f6126057b5e 7 months ago 132 MB true
quay.io/libpod/alpine_nginx latest ecea49d99daa 16 months ago 23.1 MB true
registry.fedoraproject.org/fedora-minimal 34 b85bdf5fbb1c 18 months ago 119 MB true
quay.io/libpod/alpine_labels latest 5e9e9275e4d6 18 months ago 5.87 MB true
quay.io/libpod/redis alpine 8835e6aeca99 2 years ago 32.3 MB true
quay.io/libpod/busybox latest f0b02e9d092d 2 years ago 1.45 MB false
localhost/busybox-test latest f0b02e9d092d 2 years ago 1.45 MB false
quay.io/libpod/busybox latest f0b02e9d092d 2 years ago 1.45 MB true
registry.k8s.io/pause 3.2 80d28bedfe5d 3 years ago 686 kB true
quay.io/libpod/alpine latest 961769676411 4 years ago 5.85 MB false
quay.io/libpod/alpine latest 961769676411 4 years ago 5.85 MB true
quay.io/libpod/alpine_healthcheck latest c1103abb47d6 4 years ago 10.4 MB true
Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs run --name test --rm splitstoretest echo helloworld
time="2023-08-25T10:56:14+09:00" level=warning msg="The storage 'driver' option should be set in /tmp/podman_test735789469/.config/containers/storage.conf. A driver was picked automatically."
helloworld
< Exit [It] podman pull and run on split imagestore - /root/podman/test/e2e/pull_test.go:65 @ 08/25/23 10:56:14.326 (907ms)
> Enter [AfterEach] TOP-LEVEL - /root/podman/test/e2e/common_test.go:119 @ 08/25/23 10:56:14.326
Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs stop --all -t 0
Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs pod rm -fa -t 0
Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs rm -fa -t 0
< Exit [AfterEach] TOP-LEVEL - /root/podman/test/e2e/common_test.go:119 @ 08/25/23 10:56:14.442 (116ms)