Skip to content

Workaround for unexpected NodeExpandVolume call for NFS#486

Merged
santhoshatdell merged 2 commits into
mainfrom
node-expand-nfs-no-op
May 9, 2025
Merged

Workaround for unexpected NodeExpandVolume call for NFS#486
santhoshatdell merged 2 commits into
mainfrom
node-expand-nfs-no-op

Conversation

@santhoshatdell

@santhoshatdell santhoshatdell commented May 9, 2025

Copy link
Copy Markdown
Contributor

Description

Temporary workaround to return No-Op in NodeExpandVolume for NFS until this is fixed in K8s (kubernetes/kubernetes#131419)

GitHub Issues

List the GitHub issues impacted by this PR:

GitHub Issue #
https://github.com/dell/csm/issues/1880

Checklist:

  • I have performed a self-review of my own code to ensure there are no formatting, vetting, linting, or security issues
  • I have verified that new and existing unit tests pass locally with my changes
  • I have not allowed coverage numbers to degenerate
  • I have maintained at least 90% code coverage
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • Backward compatibility is not broken

How Has This Been Tested?

With 'nightly' image, noticed the 'warning' event on the application pod due to 'volume not found' error.
With the workaround/fix, expand request on the node no longer fails thus no warning events.

Pod event:

  Normal   Scheduled                   9m26s                  default-scheduler        Successfully assigned nsps1/powerstoretest-0 to worker-2-gmlffggceyil6.domain
  Normal   SuccessfulAttachVolume      9m25s                  attachdetach-controller  AttachVolume.Attach succeeded for volume "csivol-sl-9e30860c80"
  Normal   Pulling                     9m15s                  kubelet                  Pulling image "quay.io/centos/centos:latest"
  Normal   Pulled                      9m15s                  kubelet                  Successfully pulled image "quay.io/centos/centos:latest" in 353ms (353ms including waiting). Image size: 111160413 bytes.
  Normal   Created                     9m15s                  kubelet                  Created container: test
  Normal   Started                     9m15s                  kubelet                  Started container test
  Normal   FileSystemResizeSuccessful  2m20s                  kubelet                  MountVolume.NodeExpandVolume succeeded for volume "csivol-sl-9e30860c80" worker-2-gmlffggceyil6.domain

PVC event:

  Normal  ProvisioningSucceeded       9m15s  csi-powerstore.dellemc.com_powerstore-controller-7d8754f9cf-tjckg_2629fa57-bf25-46c5-a9c2-ce875bdcbe83  Successfully provisioned volume csivol-sl-9e30860c80
  Normal  ExternalExpanding           3m35s  volume_expand                                                                                           waiting for an external controller to expand this PVC
  Normal  Resizing                    3m35s  external-resizer csi-powerstore.dellemc.com                                                             External resizer is resizing volume csivol-sl-9e30860c80
  Normal  VolumeResizeSuccessful      3m33s  external-resizer csi-powerstore.dellemc.com                                                             Resize volume succeeded
  Normal  FileSystemResizeSuccessful  2m9s   kubelet                                                                                                 MountVolume.NodeExpandVolume succeeded for volume "csivol-sl-9e30860c80" worker-2-gmlffggceyil6.domain

Controller log:

{"level":"info","msg":"/csi.v1.Controller/ControllerExpandVolume: REQ 0023: VolumeId=681e0d30-7e68-892f-f7d6-c2468531b4a9/<ARRAY-ID>/nfs, CapacityRange=required_bytes:19327352832 , VolumeCapability=mount:\u003cfs_type:\"nfs\" \u003e access_mode:\u003cmode:MULTI_NODE_MULTI_WRITER \u003e , XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2025-05-09T14:17:47.83584688Z"}
{"level":"info","msg":"/csi.v1.Controller/ControllerExpandVolume: REP 0023: CapacityBytes=19327352832, NodeExpansionRequired=false, XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2025-05-09T14:17:49.608318192Z"}

Node log:

{"level":"info","msg":"/csi.v1.Node/NodeExpandVolume: REQ 0018: VolumeId=681e0d30-7e68-892f-f7d6-c2468531b4a9/<ARRAY-ID>/nfs, VolumePath=/var/lib/kubelet/pods/091d11eb-15ce-4031-87b8-7e9d1dfd9f01/volumes/kubernetes.io~csi/csivol-sl-9e30860c80/mount, CapacityRange=required_bytes:19327352832 , StagingTargetPath=/var/lib/kubelet/plugins/kubernetes.io/csi/csi-powerstore.dellemc.com/374b5f1669fce1363b6e3f0ba3eb808630cb97374d71002c74eb8aa704334d0b/globalmount, VolumeCapability=mount:\u003cfs_type:\"nfs\" \u003e access_mode:\u003cmode:MULTI_NODE_MULTI_WRITER \u003e , XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2025-05-09T14:19:13.313050652Z"}
time="2025-05-09T14:19:13Z" level=info msg="NodeExpandVolume called req volume_id:\"681e0d30-7e68-892f-f7d6-c2468531b4a9/<ARRAY-ID>/nfs\" volume_path:\"/var/lib/kubelet/pods/091d11eb-15ce-4031-87b8-7e9d1dfd9f01/volumes/kubernetes.io~csi/csivol-sl-9e30860c80/mount\" capacity_range:<required_bytes:19327352832 > staging_target_path:\"/var/lib/kubelet/plugins/kubernetes.io/csi/csi-powerstore.dellemc.com/374b5f1669fce1363b6e3f0ba3eb808630cb97374d71002c74eb8aa704334d0b/globalmount\" volume_capability:<mount:<fs_type:\"nfs\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > "
{"level":"info","msg":"/csi.v1.Node/NodeExpandVolume: REP 0018: CapacityBytes=0, XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2025-05-09T14:19:13.313287206Z"}

cert-csi tests have passed.

[root@master-1-gMLffggcEYiL6 cert-csi]# ./cert-csi certify --cert-config pstore-config.yaml --vsc powerstore-snapshot
[2025-05-09 10:57:08]  INFO Starting cert-csi; ver. 1.6.0
.....
[2025-05-09 10:57:08]  INFO Suites to run with powerstore-nfs storage class:
[2025-05-09 10:57:08]  INFO 1. VolumeIoSuite {volumes: 2, volumeSize: 5Gi chains: 2-2}
[2025-05-09 10:57:08]  INFO 2. ScalingSuite {replicas: 2, volumes: 5, volumeSize: 5Gi}
[2025-05-09 10:57:08]  INFO 3. CloneVolumeSuite {pods: 2, volumes: 1, volumeSize: 5Gi}
[2025-05-09 10:57:08]  INFO 4. VolumeExpansionSuite {pods: 1, volumes: 1, size: 5Gi, expSize: 10Gi, block: false}
[2025-05-09 10:57:08]  INFO 5. SnapSuite {snapshots: 3, volumeSize; 5Gi}
[2025-05-09 10:57:08]  INFO 6. ReplicationSuite {pods: 2, volumes: 5, volumeSize: 5Gi}
[2025-05-09 10:57:08]  INFO 7. MultiAttachSuite {pods: 5, rawBlock: false, size: 5Gi, accMode: ReadWriteMany}
[2025-05-09 10:57:08]  INFO 8. EphemeralVolumeSuite {driver: csi-powerstore.dellemc.com, podNumber: 2, volAttributes: map[arrayID:<ID> nasName:nas1 protocol:NFS size:5Gi]}
Does it look OK? (Y)es/(n)o
...
[2025-05-09 11:01:58]  INFO Avg time of a run:   96.89s
[2025-05-09 11:01:58]  INFO Avg time of a del:   40.04s
[2025-05-09 11:01:58]  INFO Avg time of all:     140.26s
[2025-05-09 11:01:58]  INFO During this run 100.0% of suites succeeded

@github-actions

github-actions Bot commented May 9, 2025

Copy link
Copy Markdown
Contributor

Merging this branch will not change overall coverage

Impacted Packages Coverage Δ 🤖
github.com/dell/csi-powerstore/pkg/node 0.00% (ø)
Coverage by file

Changed files (no unit tests)

Changed File Coverage Δ Total Covered Missed 🤖
github.com/dell/csi-powerstore/pkg/node/node.go 0.00% (ø) 0 0 0

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

Changed unit test files

  • github.com/dell/csi-powerstore/pkg/node/node_test.go

@santhoshatdell santhoshatdell merged commit 9ea4eaf into main May 9, 2025
6 checks passed
@santhoshatdell santhoshatdell deleted the node-expand-nfs-no-op branch May 9, 2025 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alikdell alikdell alikdell approved these changes

@anandrajak1 anandrajak1 Awaiting requested review from anandrajak1

@adarsh-dell adarsh-dell Awaiting requested review from adarsh-dell

@AkshaySainiDell AkshaySainiDell Awaiting requested review from AkshaySainiDell

@donatwork donatwork Awaiting requested review from donatwork

+3 more reviewers

@atye atye atye approved these changes

@abhi16394 abhi16394 abhi16394 approved these changes

@lukeatdell lukeatdell lukeatdell approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@santhoshatdell @atye @abhi16394 @alikdell @lukeatdell