I0708 11:17:50.385120 1860388 utils.go:265] ID: 384 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 GRPC call: /csi.v1.Node/NodeStageVolume
I0708 11:17:50.385228 1860388 utils.go:266] ID: 384 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/be344c85429e4f51d90940dfbfcade41e99ca990fafeda5d25c5bda139c4ffb1/globalmount","volume_capability":{"access_mode":{"mode":"MULTI_NODE_MULTI_WRITER"},"mount":{}},"volume_context":{"clusterID":"23fcac3f-1dd2-11b2-8000-080027b246c3","encrypted":"true","encryptionKMSID":"vault-cephcsi-kms-tenant-sa","fsName":"fs.ssd","mounter":"kernel","storage.kubernetes.io/csiProvisionerIdentity":"1751969579180-8943-cephfs.csi.ceph.com","subvolumeName":"csi-vol-f0317fad-23fd-45ad-85e0-abf4b26f7db1","subvolumePath":"/volumes/csi/csi-vol-f0317fad-23fd-45ad-85e0-abf4b26f7db1/0316892d-485a-49b1-8e66-eb0616b7a6e7","volumeNamePrefix":"csi-vol-"},"volume_id":"0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1"}
I0708 11:17:50.397831 1860388 omap.go:89] ID: 384 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 got omap values: (pool="fs.ssd.meta", namespace="csi", name="csi.volume.f0317fad-23fd-45ad-85e0-abf4b26f7db1"): map[csi.imagename:csi-vol-f0317fad-23fd-45ad-85e0-abf4b26f7db1 csi.volname:pvc-b32fa8f1-d9d4-4f4e-aaab-d7018fd601aa csi.volume.encryptKMS:vault-cephcsi-kms-tenant-sa csi.volume.encryptionType:file csi.volume.owner:test-pvc]
time="2025-07-08T11:17:50Z" level=info msg="Authenticated to Vault with kubernetes\n"
E0708 11:17:50.488070 1860388 fscrypt.go:357] ID: 384 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 fscrypt: config file "/etc/fscrypt.conf" already exists. Skipping fscrypt node setup
I0708 11:17:50.488204 1860388 nodeserver.go:358] ID: 384 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 cephfs: mounting volume 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 with Ceph kernel client
I0708 11:17:50.527607 1860388 cephcmds.go:106] ID: 384 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 command succeeded: mount [-t ceph kubernetes_cephfs@23fcac3f-1dd2-11b2-8000-080027b246c3.fs.ssd=/volumes/csi/csi-vol-f0317fad-23fd-45ad-85e0-abf4b26f7db1/0316892d-485a-49b1-8e66-eb0616b7a6e7 /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/be344c85429e4f51d90940dfbfcade41e99ca990fafeda5d25c5bda139c4ffb1/globalmount -o mon_addr=<list-mon-nodes-port-6789>,secretfile=/tmp/csi/keys/keyfile-1789868272,_netdev]
I0708 11:17:50.527663 1860388 nodeserver.go:298] ID: 384 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 cephfs: successfully mounted volume 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 to /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/be344c85429e4f51d90940dfbfcade41e99ca990fafeda5d25c5bda139c4ffb1/globalmount
I0708 11:17:50.527688 1860388 nodeserver.go:144] ID: 384 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 Creating lock for the following volume ID 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1
E0708 11:17:50.531638 1860388 nodeserver.go:157] ID: 384 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 failed to create lock for volume ID 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1: failed to lock volume ID 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1: rados: ret=-36, File name too long
E0708 11:17:50.531676 1860388 utils.go:270] ID: 384 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 GRPC error: rpc error: code = Internal desc = failed to lock volume ID 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1: rados: ret=-36, File name too long
I0708 11:17:51.087622 1860388 utils.go:265] ID: 385 GRPC call: /csi.v1.Node/NodeGetCapabilities
I0708 11:17:51.087662 1860388 utils.go:266] ID: 385 GRPC request: {}
I0708 11:17:51.087756 1860388 utils.go:272] ID: 385 GRPC response: {"capabilities":[{"rpc":{"type":"STAGE_UNSTAGE_VOLUME"}},{"rpc":{"type":"GET_VOLUME_STATS"}},{"rpc":{"type":"VOLUME_CONDITION"}},{"rpc":{"type":"SINGLE_NODE_MULTI_WRITER"}}]}
I0708 11:17:51.092169 1860388 utils.go:265] ID: 386 GRPC call: /csi.v1.Node/NodeGetCapabilities
I0708 11:17:51.092195 1860388 utils.go:266] ID: 386 GRPC request: {}
I0708 11:17:51.092263 1860388 utils.go:272] ID: 386 GRPC response: {"capabilities":[{"rpc":{"type":"STAGE_UNSTAGE_VOLUME"}},{"rpc":{"type":"GET_VOLUME_STATS"}},{"rpc":{"type":"VOLUME_CONDITION"}},{"rpc":{"type":"SINGLE_NODE_MULTI_WRITER"}}]}
I0708 11:17:51.093051 1860388 utils.go:265] ID: 387 GRPC call: /csi.v1.Node/NodeGetCapabilities
I0708 11:17:51.093066 1860388 utils.go:266] ID: 387 GRPC request: {}
I0708 11:17:51.093122 1860388 utils.go:272] ID: 387 GRPC response: {"capabilities":[{"rpc":{"type":"STAGE_UNSTAGE_VOLUME"}},{"rpc":{"type":"GET_VOLUME_STATS"}},{"rpc":{"type":"VOLUME_CONDITION"}},{"rpc":{"type":"SINGLE_NODE_MULTI_WRITER"}}]}
I0708 11:17:51.093947 1860388 utils.go:265] ID: 388 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 GRPC call: /csi.v1.Node/NodeStageVolume
I0708 11:17:51.094047 1860388 utils.go:266] ID: 388 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/be344c85429e4f51d90940dfbfcade41e99ca990fafeda5d25c5bda139c4ffb1/globalmount","volume_capability":{"access_mode":{"mode":"MULTI_NODE_MULTI_WRITER"},"mount":{}},"volume_context":{"clusterID":"23fcac3f-1dd2-11b2-8000-080027b246c3","encrypted":"true","encryptionKMSID":"vault-cephcsi-kms-tenant-sa","fsName":"fs.ssd","mounter":"kernel","storage.kubernetes.io/csiProvisionerIdentity":"1751969579180-8943-cephfs.csi.ceph.com","subvolumeName":"csi-vol-f0317fad-23fd-45ad-85e0-abf4b26f7db1","subvolumePath":"/volumes/csi/csi-vol-f0317fad-23fd-45ad-85e0-abf4b26f7db1/0316892d-485a-49b1-8e66-eb0616b7a6e7","volumeNamePrefix":"csi-vol-"},"volume_id":"0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1"}
I0708 11:17:51.097905 1860388 omap.go:89] ID: 388 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 got omap values: (pool="fs.ssd.meta", namespace="csi", name="csi.volume.f0317fad-23fd-45ad-85e0-abf4b26f7db1"): map[csi.imagename:csi-vol-f0317fad-23fd-45ad-85e0-abf4b26f7db1 csi.volname:pvc-b32fa8f1-d9d4-4f4e-aaab-d7018fd601aa csi.volume.encryptKMS:vault-cephcsi-kms-tenant-sa csi.volume.encryptionType:file csi.volume.owner:test-pvc]
time="2025-07-08T11:17:51Z" level=info msg="Authenticated to Vault with kubernetes\n"
E0708 11:17:51.174260 1860388 fscrypt.go:357] ID: 388 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 fscrypt: config file "/etc/fscrypt.conf" already exists. Skipping fscrypt node setup
I0708 11:17:51.174354 1860388 nodeserver.go:274] ID: 388 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 cephfs: volume 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 is already mounted to /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/be344c85429e4f51d90940dfbfcade41e99ca990fafeda5d25c5bda139c4ffb1/globalmount, skipping
I0708 11:17:51.174373 1860388 nodeserver.go:144] ID: 388 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 Creating lock for the following volume ID 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1
E0708 11:17:51.177767 1860388 nodeserver.go:157] ID: 388 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 failed to create lock for volume ID 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1: failed to lock volume ID 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1: rados: ret=-36, File name too long
E0708 11:17:51.177794 1860388 utils.go:270] ID: 388 Req-ID: 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1 GRPC error: rpc error: code = Internal desc = failed to lock volume ID 0001-0024-23fcac3f-1dd2-11b2-8000-080027b246c3-0000000000000002-f0317fad-23fd-45ad-85e0-abf4b26f7db1: rados: ret=-36, File name too long
Describe the bug
When encryption is enabled on a StorageClass using CephFS-CSI, the volume can be provisioned (when creating a standalone PVC), but it cannot be attached or mounted to a pod (the pod remains stuck in the
ContainerCreatingstatus). The potential issue may be related to a lock file, specifically in the code at https://github.com/ceph/ceph-csi/blob/devel/internal/cephfs/nodeserver.go#L155.Environment details
v3.14.13.14.1Linux K8S-NODE-WORKER-N01-A1B2C3D4E5F6A1 6.11.0-29-generic #29~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jun 26 14:16:59 UTC 2 x86_64 x86_64 x86_64 GNU/Linuxfuseorkernel. for rbd itskrbdorrbd-nbd) :cephFS-kernelv1.32.319.2.1Steps to reproduce
Steps to reproduce the behavior:
Configure CephFS-CSI Helm Values File:
encryptionKMSConfigin theceph-csi-cephfsHelm values file:Create StorageClasses:
sc-vaultandsc-metadata, with encryption enabled:Configure Ceph Keyring:
kubernetes_cephfs:Create PersistentVolumeClaims (PVCs):
Create Deployment and Mount PVCs:
Observe the Error:
At step 4, the
ceph-csi-cephfs-provisionerpod logs indicate successful volume provisioning for both PVCs (data-test-cephfs-1anddata-test-cephfs-2).At step 5, when creating the Deployment and mounting the PVCs (with
replicas=1orreplicas=2, the behavior is identical), theceph-csi-cephfs-nodepluginpod logs show an error related to locking the volume: rados: ret=-36, File name too long.The Deployment’s pod remains stuck in the
ContainerCreatingstate, andkubectl describe podshows:Actual results
The Deployment pod(s) cannot mount the volume from the PVC.
The
ceph-csi-cephfs-nodepluginpod logs indicate an error related to a lock file.Expected behavior
The Deployment should mount the volume from the PVC and run normally.
For more info: When using the same
ceph-csi-cephfssetup but with a StorageClass that does not enable encryption, everything works smoothly.Logs
csi-provisioner and csi-rbdplugin/csi-cephfsplugin container logs from the
provisioner pod.
csi-rbdplugin/csi-cephfsplugin and driver-registrar container logs from
plugin pod from the node where the mount is failing.
dmesg logs.