Mark volume as uncertain after Unmount* fails#100183
Conversation
|
@jsafrane: This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the The DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
3da7652 to
fe7d5bd
Compare
|
/test pull-kubernetes-unit |
|
/test pull-kubernetes-e2e-kind |
|
I tried to add e2e tests for the block mode, but the mock driver does not support block volumes. |
There was a problem hiding this comment.
For InvalidArgument, do we need to mark as uncertain?
There was a problem hiding this comment.
It's an example of a "final" response, where we can be reasonably sure that e.g. provisioning is not in progress. Not sure if we can apply the same logic to Unstage.
|
/approve |
|
PR lgtm. just want to make sure other reviewers have a chance to take a look. |
|
/lgtm |
When UnmountDevice fails, kubelet treat the volume mount as uncertain, because it does not know at which stage UnmountDevice failed. It may be already partially unmonted / destroyed. As result, MountDevice will be performer when a new Pod is started on the node after UnmountDevice faiure.
…ere unmounted podVolumesExist() should consider also uncertain volumes (where kubelet does not know if a volume was fully unmounted) when checking for pod's volumes. Added GetPossiblyMountedVolumesForPod for that. Adding uncertain mounts to GetMountedVolumesForPod would potentially break other callers (e.g. `verifyVolumesMountedFunc`).
To know when a volume has been fully unmounted (incl. uncertain mounts).
Change existing desiredStateOfWorldPopulator.findAndAddNewPods tests to use a common initialization function.
desiredStateOfWorldPopulator.findAndRemoveDeletedPods() should remove volumes from DSW when a pod is deleted on the API server and the volume is uncertain in ASW.
fca9dc1 to
d5da730
Compare
|
/retest |
|
@jsafrane: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jingxu97, jsafrane, mrunalp The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Any chance we can backport this change? |
…0183-upstream-release-1.21 Automated cherry pick of #100183: Mark volume as uncertain after Unmount* fails
|
Any updates, can we get to know if this is getting backported to 1.20 since we are facing a lot of such issues causing production downtimes? |
What type of PR is this?
/kind bug
What this PR does / why we need it:
Mark volume device / volume mount as uncertain after UnmountDevice / UnmountVolume operation fails, because kubelet can't be sure what's the actual state of the volume. Kubelet will then call MountDevice / SetUp to make sure the volume is fully mounted when a new pod wants to use the same volume.
Similarly, update Unmap* calls for block volumes.
Which issue(s) this PR fixes:
Fixes: #100182
Special notes for your reviewer:
Does this PR introduce a user-facing change?
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: