Skip to content

nfs: implement ControllerModifyVolume to change NFS-server hostname#5829

Merged
mergify[bot] merged 13 commits into
ceph:develfrom
nixpanic:nfs/ControllerModifyVolume
Feb 6, 2026
Merged

nfs: implement ControllerModifyVolume to change NFS-server hostname#5829
mergify[bot] merged 13 commits into
ceph:develfrom
nixpanic:nfs/ControllerModifyVolume

Conversation

@nixpanic

@nixpanic nixpanic commented Dec 4, 2025

Copy link
Copy Markdown
Member

The new hostname of the NFS-server is stored in a journal entry for the
volume. The new NFS-server will only be used the next time the NFS
Volume is mounted on a worker node.

It is not possible to update the VolumeContext of a Volume, so the old
hostname of the NFS-server will be stored there forever.


Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

  • /retest ci/centos/<job-name>: retest the <job-name> after unrelated
    failure (please report the failure too!)

@mergify mergify Bot added the component/nfs Issues related to NFS label Dec 4, 2025
@nixpanic nixpanic force-pushed the nfs/ControllerModifyVolume branch from d006863 to 1a11979 Compare December 4, 2025 17:08
Rakshith-R
Rakshith-R previously approved these changes Dec 10, 2025

@Madhu-1 Madhu-1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nixpanic is this tested code? i dont see ControllerServiceCapability_RPC_MODIFY_VOLUME capability is exposed for nFS

@nixpanic

Copy link
Copy Markdown
Member Author

@nixpanic is this tested code? i dont see ControllerServiceCapability_RPC_MODIFY_VOLUME capability is exposed for nFS

Still trying to add some e2e tests, the csi-provisioner and csi-resizer sidecars have just gained the ability to pass credentials to the ControllerModifyVolume request.

@nixpanic nixpanic marked this pull request as draft December 10, 2025 14:00
@nixpanic nixpanic force-pushed the nfs/ControllerModifyVolume branch from 1a11979 to ca55fc1 Compare December 12, 2025 10:18
@mergify mergify Bot dismissed Rakshith-R’s stale review December 12, 2025 10:19

Pull request has been modified.

@nixpanic

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e-helm/k8s-1.34/nfs

@nixpanic

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e/k8s-1.34/nfs

@nixpanic

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e/k8s-1.34

@nixpanic

Copy link
Copy Markdown
Member Author

@nixpanic is this tested code? i dont see ControllerServiceCapability_RPC_MODIFY_VOLUME capability is exposed for nFS

Verified 😜 that it fails with this:

  I1212 12:26:38.595866 75590 pvc.go:85] PVC cephcsi-nfs-pvc Event: ProvisioningFailed - CSI driver does not support VolumeAttributesClass: controller MODIFY_VOLUME capability is not reported

@nixpanic nixpanic force-pushed the nfs/ControllerModifyVolume branch from ca55fc1 to fe5d071 Compare December 12, 2025 12:35
@nixpanic

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e/k8s-1.34

2 similar comments
@nixpanic

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e/k8s-1.34

@nixpanic

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e/k8s-1.34

@nixpanic

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e/k8s-1.34/cephfs

@nixpanic

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e/k8s-1.34/cephfs

@nixpanic nixpanic force-pushed the nfs/ControllerModifyVolume branch from fc84b94 to df7f5dd Compare December 17, 2025 09:08
@nixpanic

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e/k8s-1.34/cephfs

@nixpanic nixpanic force-pushed the nfs/ControllerModifyVolume branch from df7f5dd to 0f12bea Compare December 17, 2025 12:19
@nixpanic

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e/k8s-1.34/cephfs

@nixpanic

Copy link
Copy Markdown
Member Author

Still trying to use the old nfs-server:

  Warning  FailedMount  8s (x13 over 10m)  kubelet            MountVolume.SetUp failed for volume "pvc-c64d91d7-9097-46f3-9594-eb993e5851b7" : rpc error: code = Internal desc = nfs: failed to mount "relocated.example.net:/0001-0024-e044d94f-5dfb-4f7b-a7e6-44895f196074-0000000000000001-38cbb13b-1f15-4a9b-80ff-3017574691a1" to "/var/lib/kubelet/pods/4ebe7989-0889-4af1-b138-48c67a90debd/volumes/kubernetes.io~csi/pvc-c64d91d7-9097-46f3-9594-eb993e5851b7/mount" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs relocated.example.net:/0001-0024-e044d94f-5dfb-4f7b-a7e6-44895f196074-0000000000000001-38cbb13b-1f15-4a9b-80ff-3017574691a1 /var/lib/kubelet/pods/4ebe7989-0889-4af1-b138-48c67a90debd/volumes/kubernetes.io~csi/pvc-c64d91d7-9097-46f3-9594-eb993e5851b7/mount
Output: mount.nfs: Failed to resolve server relocated.example.net: Name or service not known
 stderr: ""

@nixpanic

Copy link
Copy Markdown
Member Author

It seems I missed handling of mutable_parameters in CreateVolume CSI procedure.

@nixpanic nixpanic force-pushed the nfs/ControllerModifyVolume branch from 0f12bea to dc81558 Compare December 19, 2025 15:55
@nixpanic

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e/k8s-1.34/cephfs

1 similar comment
@nixpanic

nixpanic commented Jan 6, 2026

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e/k8s-1.34/cephfs

@nixpanic nixpanic force-pushed the nfs/ControllerModifyVolume branch from dc81558 to 8899dcc Compare January 6, 2026 13:24
@nixpanic

nixpanic commented Feb 4, 2026

Copy link
Copy Markdown
Member Author

@Madhu-1 , NFS (and CephFS) uses admin credentials everywhere. Just using user credentials for ControllerModifyVolume and NodePublishVolume isn't straight forward. Reverted to admin credentials again, so that things at least work again.

@mergify

mergify Bot commented Feb 4, 2026

Copy link
Copy Markdown
Contributor

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

@Madhu-1

Madhu-1 commented Feb 5, 2026

Copy link
Copy Markdown
Collaborator

@nixpanic lets address the merge conflict and have a tracker to address the admin creds

Signed-off-by: Niels de Vos <ndevos@ibm.com>
All functions use a fail-early-continue-on-success flow,
NodeGetVolumeStats was not following this.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
Signed-off-by: Niels de Vos <ndevos@ibm.com>
Signed-off-by: Niels de Vos <ndevos@ibm.com>
The new hostname of the NFS-server is stored in a journal entry for the
volume. The new NFS-server will only be used the next time the NFS
Volume is mounted on a worker node.

It is not possible to update the VolumeContext of a Volume, so the old
hostname of the NFS=server will be stored there forever.

Closes: ceph#5420
Signed-off-by: Niels de Vos <ndevos@ibm.com>
Signed-off-by: Niels de Vos <ndevos@ibm.com>
@nixpanic

nixpanic commented Feb 5, 2026

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e/k8s-1.35

Madhu-1
Madhu-1 previously approved these changes Feb 5, 2026
Rakshith-R
Rakshith-R previously approved these changes Feb 5, 2026
Signed-off-by: Niels de Vos <ndevos@ibm.com>
Only the canary version suppors the VolumeAttributesClass with secrets
in the ControllerModifyVolume gRPC.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
Signed-off-by: Niels de Vos <ndevos@ibm.com>
The NFS node-plugin does not support staging, it only uses publishing.
For VolumeAttributeClass / ControllerModifyVolume the mutable parameters
are stored in the Ceph backend. This means that during volume publishing
the node-plugin needs to get the updated parameters from the Ceph
cluster (hence the publish secret requirement).

/tmp/csi has been added to the node-plugin Pod so that the temporary
credentials file can be written.

Signed-off-by: Niels de Vos <ndevos@ibm.com>
Signed-off-by: Niels de Vos <ndevos@ibm.com>
Only Kubernetes 1.34 and newer support VolumeAttributeClass
functionality (GA). Ceph-CSI also depends on certain versions of the
Kubernetes CSI external-provisioner and external-resizer.

See-also: kubernetes-csi/docs#631
Signed-off-by: Niels de Vos <ndevos@ibm.com>
@nixpanic

nixpanic commented Feb 5, 2026

Copy link
Copy Markdown
Member Author

aaah! after renaming the VAC example, there was still a typo somewhere 😢

failed to create NFS voluemattributesclass: open ../examples/nfs//volumeattributeclass.yaml: no such file or directory

@nixpanic

nixpanic commented Feb 5, 2026

Copy link
Copy Markdown
Member Author

/test ci/centos/mini-e2e/k8s-1.35

@nixpanic

nixpanic commented Feb 5, 2026

Copy link
Copy Markdown
Member Author

@Madhu-1 and @Rakshith-R , this is now really ready for approval. The CI job is still running, but the NFS tests have passed.

@nixpanic

nixpanic commented Feb 5, 2026

Copy link
Copy Markdown
Member Author

@nixpanic lets address the merge conflict and have a tracker to address the admin creds

Tracked by #6026.

@Rakshith-R

Copy link
Copy Markdown
Contributor

@Mergifyio requeue

@mergify

mergify Bot commented Feb 6, 2026

Copy link
Copy Markdown
Contributor

requeue

✅ This pull request will be embarked automatically

Details

The head sha of this pull request, b4565bd, was never embarked in the merge queue.

But don't worry, Mergify will embark it automatically for you.

@mergify

mergify Bot commented Feb 6, 2026

Copy link
Copy Markdown
Contributor

queue

✅ The pull request has been merged automatically

Details

The pull request has been merged automatically at 94c5e36

@mergify

mergify Bot commented Feb 6, 2026

Copy link
Copy Markdown
Contributor

Merge Queue Status

✅ The pull request has been merged at b4565bd

This pull request spent 3 hours 5 minutes 16 seconds in the queue, including 3 hours 4 minutes 56 seconds running CI.
The checks were run on draft #6029.

Required conditions to merge
  • #approved-reviews-by >= 2 [🛡 GitHub branch protection]
  • #changes-requested-reviews-by = 0 [🛡 GitHub branch protection]
  • any of:
    • all of:
      • base=devel
      • status-success=codespell
      • status-success=go-test
      • status-success=golangci-lint
      • status-success=lint-extras
      • status-success=mod-check
      • status-success=multi-arch-build
      • status-success=uncommitted-code-check
      • any of:
        • all of:
          • status-success=ci/centos/k8s-e2e-external-storage/1.33
          • status-success=ci/centos/k8s-e2e-external-storage/1.34
          • status-success=ci/centos/k8s-e2e-external-storage/1.35
          • status-success=ci/centos/mini-e2e-helm/k8s-1.33
          • status-success=ci/centos/mini-e2e-helm/k8s-1.34
          • status-success=ci/centos/mini-e2e-helm/k8s-1.35
          • status-success=ci/centos/mini-e2e/k8s-1.33
          • status-success=ci/centos/mini-e2e/k8s-1.34
          • status-success=ci/centos/mini-e2e/k8s-1.35
          • status-success=ci/centos/upgrade-tests-cephfs
          • status-success=ci/centos/upgrade-tests-rbd
        • label=ci/skip/e2e
    • all of:
      • base~=^(release-.+)$
      • any of:
        • label=ci/skip/e2e
        • all of:
          • status-success=ci/centos/k8s-e2e-external-storage/1.32
          • status-success=ci/centos/mini-e2e-helm/k8s-1.32
          • status-success=ci/centos/mini-e2e/k8s-1.32
          • status-success=ci/centos/k8s-e2e-external-storage/1.33
          • status-success=ci/centos/k8s-e2e-external-storage/1.34
          • status-success=ci/centos/mini-e2e-helm/k8s-1.33
          • status-success=ci/centos/mini-e2e-helm/k8s-1.34
          • status-success=ci/centos/mini-e2e/k8s-1.33
          • status-success=ci/centos/mini-e2e/k8s-1.34
          • status-success=ci/centos/upgrade-tests-cephfs
          • status-success=ci/centos/upgrade-tests-rbd
      • status-success=codespell
      • status-success=go-test
      • status-success=golangci-lint
      • status-success=lint-extras
      • status-success=mod-check
      • status-success=multi-arch-build
      • status-success=uncommitted-code-check
    • all of:
      • base=release-v3.15
      • any of:
        • label=ci/skip/e2e
        • all of:
          • status-success=ci/centos/k8s-e2e-external-storage/1.31
          • status-success=ci/centos/k8s-e2e-external-storage/1.32
          • status-success=ci/centos/mini-e2e-helm/k8s-1.31
          • status-success=ci/centos/mini-e2e-helm/k8s-1.32
          • status-success=ci/centos/mini-e2e/k8s-1.31
          • status-success=ci/centos/mini-e2e/k8s-1.32
          • status-success=ci/centos/k8s-e2e-external-storage/1.33
          • status-success=ci/centos/mini-e2e-helm/k8s-1.33
          • status-success=ci/centos/mini-e2e/k8s-1.33
          • status-success=ci/centos/upgrade-tests-cephfs
          • status-success=ci/centos/upgrade-tests-rbd
      • status-success=codespell
      • status-success=go-test
      • status-success=golangci-lint
      • status-success=lint-extras
      • status-success=mod-check
      • status-success=multi-arch-build
      • status-success=uncommitted-code-check
    • all of:
      • base=ci/centos
      • status-success=ci/centos/jjb-validate
      • status-success=ci/centos/job-validation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/nfs Issues related to NFS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants