Typical shared cache mounts are just made from scratch, but llb allows you to create them from any state. If you create one from something besides scratch, that results in the actual mount being an overlay.

That causes problems when you mount it to multiple runs at the same time because even though they are shared, the implementation creates multiple overlay mounts using the same upper+work dirs, which is not allowed according to overlay docs, but doesn't always result in an error. Instead, you'll see warnings printed to dmesg and strange behavior when trying to run with the mounts, such as ENOENT on files you just created.

The underlying issue is that calling MutableRef.Mounts multiple times will return separate mountables that create independent overlay mounts. One potential fix is to use a trick where you can create a single overlay mount somewhere and then bind-mount the root of that overlay mount multiple places. This lets you share the overlay without creating it multiple times. We could update the cache or snapshots package to handle this by creating the overlay mount to a temp dir and then returning a bind-mount pointing to that temp dir.