We ran into a quite vexing issue when building a relatively simple stage for image-builder [0]. The code below copies a container ghcr.io/ondrejbudai/bootc:fedora in our tests into a /tmp/$tmpdir image store and then uses that image store in /tmp to mount the image using podman --imagestore /tmp/$tmpdir image mount .... The issue is that in that setup the final layer of the image (which in this case contains a modified /usr/bin/bootc file) is missing and the "old" bootc file is in the mounted path but not the expected one.

After some debugging it turns out that the way the final mount call is generated is incorrect for the case of additional storages. It will generate something like:

lowerdir=/var/lib/containers/storage/overlay/8d72fb04445c9b041287bf1aa467de4bb8a95fb2795bb6fbe214aee32f8f1cd0/diff:/tmp/tmpy7x_x8wm/overlay/l/EWJCNVO7BXHKURAPHEKM3HOTFC:....

however the /var/lib/containers/storage/overlay/8d72fb04445c9b041287bf1aa467de4bb8a95fb2795bb6fbe214aee32f8f1cd0/diff dir is empty because the actual data is stored in /tmp/tmp7x... and the "normal" store does not contain any data.

Attached is a small patch that works for my test-case in image-builder (and the "normal" case seems to be still working fine when running this manully) - unfortunately I do not have a better reproducer then running it with the image builder commit below right now also sorry that e patch needs -p4 to be applied, I tested it directly in a build podman, I hope this bugreport and the diff are still useful.

extra-storage.diff.txt

Thanks!
Michael

[0] ondrejbudai/osbuild@6d7d411#diff-2b1fc5cd8b2ebf1d121181a2570dff7a923f476fd8a9e6c928c9f9f7085d8f8fR34