Fix CoordinatorTests.testUnresponsiveLeaderDetectedEventually#64462
Merged
fcofdez merged 2 commits intoelastic:masterfrom Nov 2, 2020
Merged
Conversation
Take into account messy scenarios where a 5 node clusters elections where multiple nodes can trigger an election concurrently, meaning that it takes longer to stabilise the cluster and elect a leader. Fixes elastic#63918
Collaborator
|
Pinging @elastic/es-distributed (:Distributed/Cluster Coordination) |
DaveCTurner
requested changes
Nov 2, 2020
Member
DaveCTurner
left a comment
There was a problem hiding this comment.
Hmm actually I think we need another defaultMillis(PUBLISH_TIMEOUT_SETTING) too -- the failing elections all try and publish to the unresponsive leader and must therefore wait for a timeout.
Contributor
Author
|
I'm not sure if we need the additional publish timeout as those publications go through as expected (since they have quorum) or fail due to term bumps. Am I missing something? |
Member
|
Yes, until the master is properly established each publication will go to the unresponsive node and therefore wait for the publish timeout before proceeding. That node is only removed from the cluster once the elections have settled down. In the failing test, we blackhole (second column is times relative to the blackhole time) Until version 11, all publications go to |
Contributor
Author
|
Thanks for the explanation @DaveCTurner ! I was missing that publications waits up until the timeout to succeed even if it got a Quorum. I've updated the PR. |
fcofdez
added a commit
to fcofdez/elasticsearch
that referenced
this pull request
Nov 2, 2020
Take into account messy scenarios of 5 node clusters elections where multiple nodes can trigger an election concurrently, meaning that it takes longer to stabilize the cluster and elect a leader. Fixes elastic#63918 Backport of elastic#64462
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this pull request
Oct 20, 2021
Today we require the cluster to stabilise in a time period that allows time for the first election to encounter conflicts. However on very rare occasions there might be an election conflict in the second election too. This commit extends the stabilisation timeout to allow for this. Similar to elastic#64462 Closes elastic#78370
elasticsearchmachine
pushed a commit
that referenced
this pull request
Oct 28, 2021
Today we require the cluster to stabilise in a time period that allows time for the first election to encounter conflicts. However on very rare occasions there might be an election conflict in the second election too. This commit extends the stabilisation timeout to allow for this. Similar to #64462 Closes #78370
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this pull request
Oct 28, 2021
Today we require the cluster to stabilise in a time period that allows time for the first election to encounter conflicts. However on very rare occasions there might be an election conflict in the second election too. This commit extends the stabilisation timeout to allow for this. Similar to elastic#64462 Closes elastic#78370
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this pull request
Oct 28, 2021
Today we require the cluster to stabilise in a time period that allows time for the first election to encounter conflicts. However on very rare occasions there might be an election conflict in the second election too. This commit extends the stabilisation timeout to allow for this. Similar to elastic#64462 Closes elastic#78370
elasticsearchmachine
pushed a commit
that referenced
this pull request
Nov 5, 2021
Today we require the cluster to stabilise in a time period that allows time for the first election to encounter conflicts. However on very rare occasions there might be an election conflict in the second election too. This commit extends the stabilisation timeout to allow for this. Similar to #64462 Closes #78370 Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
elasticsearchmachine
pushed a commit
that referenced
this pull request
Nov 5, 2021
Today we require the cluster to stabilise in a time period that allows time for the first election to encounter conflicts. However on very rare occasions there might be an election conflict in the second election too. This commit extends the stabilisation timeout to allow for this. Similar to #64462 Closes #78370 Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Take into account messy scenarios of 5 node clusters elections
where multiple nodes can trigger an election concurrently, meaning
that it takes longer to stabilize the cluster and elect a leader.
Fixes #63918