apiserver: add --shutdown-delay-duration to keep serving until LBs stop sending traffic#74416
Conversation
|
/assign @deads2k |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: sttts The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/assign @stewart-yu |
There was a problem hiding this comment.
can can remove "" at the end of line 184?
There was a problem hiding this comment.
I am just following the style in this file.
staging/src/k8s.io/apiserver/pkg/server/options/server_run_options.go
Outdated
Show resolved
Hide resolved
1ec3faf to
6418b0c
Compare
|
This lgtm. It helps limit an unnecessary race. It's not foolproof because we cannot be aware of all of our consumers, but it makes it possible to avoid unnecessary dead endpoints. @kubernetes/sig-api-machinery-misc |
|
/hold not that I necessarily disagree, but I think this might be a big enough addition to the surface area that I want to think about it for a second. |
77bb860 to
bd1f77a
Compare
|
Rebased. @lavalamp @logicalhan please cancel the hold here. |
|
/retest |
bd1f77a to
e0d6b98
Compare
…op serving traffic
e0d6b98 to
408f36b
Compare
|
/lgtm |
|
/hold cancel |
|
/retest Review the full test history for this PR. Silence the bot with an |
1 similar comment
|
/retest Review the full test history for this PR. Silence the bot with an |
|
@sttts: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
We set the shutdown delay to `20s` which will lead to the fact that kube-apiserver will serve requests normally and `/healthz` returns success, but `/readyz` will immediately return `false`. Graceful termination starts after this delay has elapsed. We are using this to allow load balancers to stop sending traffic to this server (the SDN has time to update the iptables on all nodes and stop sending traffic). Previously, the kube-apiserver was stopping servering requests while it may still got sent traffic. See kubernetes/kubernetes#74416
We set the shutdown delay to `20s` which will lead to the fact that kube-apiserver will serve requests normally and `/healthz` returns success, but `/readyz` will immediately return `false`. Graceful termination starts after this delay has elapsed. We are using this to allow load balancers to stop sending traffic to this server (the SDN has time to update the iptables on all nodes and stop sending traffic). Previously, the kube-apiserver was stopping servering requests while it may still got sent traffic. See kubernetes/kubernetes#74416
This is meant to delay the apiserver shutdown for a defined time duration in order to give the SDN a chance to update changed endpoints.
The reconciler is part of the "master controller", also called "bootstrap controller". It has a pre shutdown hook triggered by the
stopCh. We delay theinternalStopChbeing closed which triggers to stop serving.