This repository was archived by the owner on Sep 9, 2020. It is now read-only.
This repository was archived by the owner on Sep 9, 2020. It is now read-only.
Persistent deployment update failures with "there is a meaningful conflict" #650
Description
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
ks apply fails (and never succeeds without intervention once it happens) with:
ERROR handle object: can't update deployments nginx: Operation cannot be fulfilled on deployments.apps "nginx": there is a meaningful conflict (firstResourceVersion: "26191208", currentResourceVersion: "26191211"):
diff1={"metadata":{"resourceVersion":"26191211"},"status":{"$setElementOrder/conditions":[{"type":"Available"},{"type":"Progressing"}],"conditions":[{"message":"ReplicaSet \"nginx-5d97bcd8db\" is progressing.","reason":"ReplicaSetUpdated","type":"Progressing"}],"replicas":3,"unavailableReplicas":4}}
, diff2={"metadata":{"annotations":{"deployment.kubernetes.io/revision":"1"},"resourceVersion":"26191195","selfLink":"/apis/apps/v1beta1/namespaces/default/deployments/nginx"},"spec":{"template":{"metadata":{"creationTimestamp":null}}},"status":{"$setElementOrder/conditions":[{"type":"Available"},{"type":"Progressing"}],"conditions":[{"lastUpdateTime":"2018-06-19T21:29:36Z","message":"ReplicaSet \"nginx-5f8bbdbf84\" is progressing.","reason":"ReplicaSetUpdated","type":"Progressing"}],"observedGeneration":1,"replicas":3,"updatedReplicas":3}}
What you expected to happen:
Successful ks apply with:
INFO Updating deployments nginx
How to reproduce it (as minimally and precisely as possible):
This is unfortunately fuzzy, as it seems to crop up at fairly random times. I've seen it happen after:
- Modifying deployment and/or pod template labels (not entirely unexpected, this is a breaking action in some cases)
- Trying to apply a deployment that has multiple old replicaSets seems to trigger it, but only sometimes. According to the diff above, one is in a "Progressing" state, while the other is... also "Progressing"? I'm unclear on what the conflict is in this case.
Anything else we need to know?:
As mentioned above there are instance where even after putting ks apply in a loop for 30 iterations, the issue doesn't go away, which seems to indicate that mimicking the kubectl behavior of "retry when there's a merge issue" (#619). Then again, it seems to be a different failure mode, so that might be expected.
Environment:
- ksonnet version (use
ks version):
ksonnet version: 0.11.0
jsonnet version: v0.10.0
client-go version:
- Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-26T16:14:03Z", GoVersion:"go1.9.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.1", GitCommit:"1883ce4eb0e057cfc2439ebeb9822da0a9d40405", GitTreeState:"clean", BuildDate:"2018-05-11T17:16:56Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}