Description

This bug report is an extract with my findings during the discussion with @TBBle in #6589 (comment) which I'm copying/describing below:

I build a Windows container image based on mcr.microsoft.com/windows/servercore:ltsc2022 using Dockerfile which contains the following directive:

# Document volume mount points typically mounted from Azure file shares.
VOLUME ["D:", "Z:"]

which causes trouble to containerd failing it with error like this:

Error: failed to create containerd container: rootpath on
  mountPath C:\Windows\TEMP\ctd-volume3950197485\377,
  volume Z:: CreateFile C:\Windows\TEMP\ctd-volume3950197485\377\Z::
The filename, directory name, or volume label syntax is incorrect.

As far as I am (and @TBBle too) reading the docs correctly, my usage there looks valid. It also does not cause any trouble to Docker Desktop.

Steps to reproduce the issue

1. Create Dockerfile for a custom image based on mcr.microsoft.com/windows/servercore:ltsc2022

2. To Dockerfile add VOLUME ["D:", "Z:"] or perhaps even VOLUME ["D:"] is enough, I haven't tested that one.

3. Build the image e.g. myimage and push to remote registry

4. Deploy pod based on myimage to Azure Kubernetes Service (AKS) with. For example:

   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: test-bug6589-pv-on-d
     name: test-bug6589-pv-on-d
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: test-bug6589-pv-on-d
     template:
       metadata:
         labels:
           app: test-bug6589-pv-on-d
         name: test-bug6589-pv-on-d
       spec:
         nodeSelector:
           kubernetes.io/os: windows
         containers:
           - name: ctr-bug6589-pv-on-d
             image: mycr.azurecr.io/myimage:latest
             command:
             - "powershell.exe"
             - "-Command"
             - "while (1) { Write-Host $(Get-Date -Format u); Add-Content -Encoding Ascii D:\\data.txt $(Get-Date -Format u); sleep 5 }"
             volumeMounts:
               - name: pv-on-d
                 mountPath: "D:"
         volumes:
           - name: pv-on-d
             persistentVolumeClaim:
               claimName: test-bug6589-pv-on-d-pvc
   ---
   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: test-bug6589-pv-on-d-pvc
   spec:
     accessModes:
       - ReadWriteMany
     resources:
       requests:
         storage: 2Gi
     storageClassName: azurefile-csi

Describe the results you received and expected

The results I received were errors in the detailed AKS events for (failed) deployment of my pod:

Error: failed to create containerd container: rootpath on mountPath C:\Windows\TEMP\ctd-volume3950197485\377, volume Z:: CreateFile C:\Windows\TEMP\ctd-volume3950197485\377\Z:: The filename, directory name, or volume label syntax is incorrect.
Error: failed to create containerd container: rootpath on mountPath C:\Windows\TEMP\ctd-volume427796215\376, volume D:: CreateFile C:\Windows\TEMP\ctd-volume427796215\376\D:: The filename, directory name, or volume label syntax is incorrect.
Error: failed to create containerd container: rootpath on mountPath C:\Windows\TEMP\ctd-volume2161075262\375, volume Z:: CreateFile C:\Windows\TEMP\ctd-volume2161075262\375\Z:: The filename, directory name, or volume label syntax is incorrect.
Error: failed to create containerd container: rootpath on mountPath C:\Windows\TEMP\ctd-volume554088254\374, volume Z:: CreateFile C:\Windows\TEMP\ctd-volume554088254\374\Z:: The filename, directory name, or volume label syntax is incorrect.

The expected result was to get the pod up and running.

For example, if I replace my custom image with vanilla like below, then my pod is deployed and volume successfully mounted as D: drive:

---
# File pod-mount-pv-on-d.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test-bug6589-pv-on-d
  name: test-bug6589-pv-on-d
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-bug6589-pv-on-d
  template:
    metadata:
      labels:
        app: test-bug6589-pv-on-d
      name: test-bug6589-pv-on-d
    spec:
      nodeSelector:
        kubernetes.io/os: windows
      containers:
        - name: ctr-bug6589-pv-on-d
          image: mcr.microsoft.com/windows/servercore:ltsc2022
          command:
          - "powershell.exe"
          - "-Command"
          - "while (1) { Write-Host $(Get-Date -Format u); Add-Content -Encoding Ascii D:\\data.txt $(Get-Date -Format u); sleep 5 }"
          volumeMounts:
            - name: pv-on-d
              mountPath: "D:"
      volumes:
        - name: pv-on-d
          persistentVolumeClaim:
            claimName: test-bug6589-pv-on-d-pvc
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-bug6589-pv-on-d-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
  storageClassName: azurefile-csi

What version of containerd are you using?

1.6.14

Any other relevant information

AKS with Kubernetes 1.25.5 with containerd and Windows Server 2022 node

There is very relevant comment posted by @TBBle here #6589 (comment) as follow-up to mine linked earlier.

Show configuration if it is related to CRI plugin.

No response