Skip to content

containerd fails to add a disk mount on Windows #5405

@kevpar

Description

@kevpar

When the Azure disk CSI plugin is used with containerd in k8s, containers fail to start. This is due to containerd's mount path resolution not supporting the way the CSI plugin sets up the mount. It is likely this same bug will impact other CSI plugins that mount disks into containers.

The CSI plugin creates a symlink to the disk's volume GUID path (something like \\?\Volume{1d7d818b-f43f-4c7a-90b5-68c0a9db2063}\), and passes the symlink path to containerd as a container mount. containerd then attempts to resolve the link by calling pkg/os.ResolveSymbolicLink, which is currently implemented as calling filepath.EvalSymlinks. This ends up returning ERROR_PATH_NOT_FOUND, however, as filepath.EvalSymlinks attempts to locate a drive letter for the disk, which fails due to no drive letter being assigned.

Requiring the CSI plugin to assign a drive letter to each disk is not feasible, as that would limit to 26 total disks for each node. Instead, we should fix ResolveSymbolicLink on Windows to properly resolve the path in this case.

Tracked by kubernetes/kubernetes#99342

The best fix here is likely to switch to opening the mount source and calling GetFinalPathNameByHandle to get the "real" path to the mount source.

We could attempt to fix this in filepath.EvalSymlinks, but the behavior of this function has been unpredictable on Windows in the past, and recently there has been discussion of replacing and deprecating it (golang/go#37113, golang/go#42201). This exact issue has even been filed as a bug on Go (golang/go#39786), but we don't know when (or if) a fix will be available in Go. So we should try to fix this in containerd for the time being, to unblock disk mounts for Windows containers.

As a side note, the CSI plugin in question works with moby, because moby doesn't call filepath.EvalSymlinks for path resolution. So another option here would be just removing the call to ResolveSymbolicLink altogether from the Windows path.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions