etcd: Fix session renewal controllers#12553
Merged
Conversation
Contributor
Author
|
test-me-please |
sayboras
reviewed
Jul 16, 2020
Due to creating a context with a timeout prior to blocking on the session channel to wait for it to end, the timeout of the context will likely already hvae expired by the time the kvstore operation is performed. It is thus cancelled immediately, requiring a subsequent controller call. This prolongs the time the session is renewed and causes unnecessary controller failures which can be mistaken for etcd issues. Also pass the etcd client context to the controller to bind the lifecycle of a controller run to it. This fixes errors like these: ``` kvstore-etcd-lock-session-renew 20s ago 10s ago 1 unable to renew etcd lock session: context deadline exceeded ``` Signed-off-by: Thomas Graf <thomas@cilium.io>
The controller functions could potentially block forever while waiting on channels to close. Bind waiting on the channel and check of the etcd version to the controller and etcd client context so these operations get cancelled. Signed-off-by: Thomas Graf <thomas@cilium.io>
35af682 to
d1c0c40
Compare
qmonnet
approved these changes
Jul 16, 2020
joestringer
approved these changes
Jul 16, 2020
| ec.controllers.UpdateController("kvstore-etcd-session-renew", | ||
| controller.ControllerParams{ | ||
| // Stop controller function when etcd client is terminating | ||
| Context: ec.client.Ctx(), |
Contributor
Author
|
test-me-please |
Member
|
Backport to v1.6 in #12748. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.