Looking at the issue;

0.027 runc run failed: unable to start container process: 
    error during container init: 
    error mounting "/var/lib/docker/165536.165536/buildkit/executor/resolv.conf" to rootfs at "/etc/resolv.conf": 
    open /var/lib/docker/165536.165536/buildkit/executor/30g5og94pc1it3ymc8ymjdpd8/rootfs/etc/resolv.conf: permission denied

Who creates that resolv.conf is this one created by Docker or by BuildKit? If it's by docker; we know what the mapping is; can we chown the resolv.conf when we create it, based on the daemons userns-mapping?

For the remapped files; I think this is the part in the daemon that sets up the storage to be chowned;

which returns the uid/gid to use as "root" for container filesystems;

Who creates that resolv.conf is this one created by Docker or by BuildKit? If it's by docker; we know what the mapping is; can we chown the resolv.conf when we create it, based on the daemons userns-mapping?

The original resolv.conf file that gets bind-mounted (e.g. /var/lib/docker/100000.100000/buildkit/executor/resolv.conf) is written by buildkit's oci.GetResolvConf. This function is correctly called with idmap *user.IdentityMapping set, and the file is chowned to the remapped uid/gid. Then, buildkit adds a bind-mount to the OCI spec (in oci.generateMountOpts):

    {
      "destination": "/etc/resolv.conf",
      "type": "bind",
      "source": "/var/lib/docker/100000.100000/buildkit/executor/resolv.conf",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "rbind",
        "ro"
      ]
    },

The original issue:

0.027 runc run failed: unable to start container process: error during container init: error mounting "/var/lib/docker/165536.165536/buildkit/executor/resolv.conf" to rootfs at "/etc/resolv.conf": open /var/lib/docker/165536.165536/buildkit/executor/30g5og94pc1it3ymc8ymjdpd8/rootfs/etc/resolv.conf: permission denied

comes from this runc code path: https://github.com/opencontainers/runc/blob/00aec12c711a4d1562578e1d5e99d77610a169d9/libcontainer/rootfs_linux.go#L160.

The rootfs bind-mounted by buildkit's runcexecutor isn't id-mapped (https://github.com/moby/buildkit/blob/1cff4fb6f7938128c24eacae7c064b589423cb0e/executor/runcexecutor/executor.go#L239) — the rootMount passed to mount.All() looks like this:

rootMount := mount.Mount{
    Type: "bind",
    Source: "/var/lib/docker/100000.100000/buildkit/containerd-overlayfs/cachemounts/buildkit1070698249",
    Options: []string{"rw", "rbind"},
}

That 'Source' itself comes from buildkit's (*fromContainerd).Mounts(). The call to (*fromContainerd).Prepare() made for that snapshot key has no opts — i.e. no client. WithRemapperLabels(), nor client.WithUserNSRemapperLabels() is set.

These two client opts add labels containerd.io/snapshot/[u|g]idmapping but the mount returned by the snapshotter doesn't have idmap / gidmap options set. I'm a bit surprised by this as it seems WithRemapperLabels() was added a long time ago.

If I add this to buildkit's (*fromContainerd).Mounts() (granted s.idmap is plumbed, which is not the case right now — see here):

	if s.idmap != nil {
		for i := range mounts {
			if mounts[i].Type != "bind" && mounts[i].Type != "overlay" {
				continue
			}

			for _, m := range s.idmap.UIDMaps {
				mounts[i].Options = append(mounts[i].Options, fmt.Sprintf("uidmap=%d:%d:%d", m.ID, m.ParentID, m.Count))
			}
			for _, m := range s.idmap.GIDMaps {
				mounts[i].Options = append(mounts[i].Options, fmt.Sprintf("gidmap=%d:%d:%d", m.ID, m.ParentID, m.Count))
			}
		}
	}

Then the rootfs is correctly id-mapped, and buildkit's withROBind() needs the following to get /etc/resolv.conf and /etc/hosts to be correctly id-mapped (values hardcoded for test purpose):

			UIDMappings: []specs.LinuxIDMapping{{ContainerID: 0, HostID: 100000, Size: 65536}},
			GIDMappings: []specs.LinuxIDMapping{{ContainerID: 0, HostID: 100000, Size: 65536}},

After that, everything works correctly.

I'm not sure if that's the proper fix though (Tõnis seemed to suggest it wasn't), and it'd require changes in buildkit which won't receive a new minor release before current target date for v29.0.

If I add this to buildkit's (*fromContainerd).Mounts() (granted s.idmap is plumbed, which is not the case right now — see here):

Thought I left a comment, but probably forgot to press the button; if wiring up is the issue, given that we (currently) only provide a single mapping ... wondering if we could (but it's a bit of a hack) just have a global var that's set to the right mapping for this (until things are wired up) 🤔