Skip to content

[WIP] Fix race condition in CuratorDruidLeaderSelector #16544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
kfaraz wants to merge 3 commits into from
Closed

[WIP] Fix race condition in CuratorDruidLeaderSelector #16544

kfaraz wants to merge 3 commits into from

Conversation

@kfaraz
Copy link
Contributor

@kfaraz kfaraz commented Jun 4, 2024 •
edited
Loading

Description

There is a race condition in CuratorDruidLeaderSelector which may occur as follows:

  • Current node is elected as leader
  • LeaderLatchListener.isLeader() is called which calls DruidLeaderSelector.Listener.becomeLeader()
  • listener.becomeLeader() fails
  • CuratorDruidLeaderSelector.isLeader() is called and returns false but CuratorDruidLeaderSelector.getCurrentLeader() returns self, because the leader latch has not been updated yet
  • LeaderLatch is recreated and updated atomically
  • Subsequent calls to getCurrentLeader() return null until some other node is elected leader

Fix

  • Update leader = true only after listener.becomeLeader() has succeeded
  • Return null in getCurrentLeader() if leader = false and LeaderLatch.getLeader() returns self

Other changes

  • Add some unit tests to CuratorDruidLeaderSelector
  • Remove unused parameter from K8sDruidLeaderSelector

@kfaraz kfaraz marked this pull request as draft June 4, 2024 14:46
@kfaraz kfaraz changed the title [WIP] Add unit tests for CuratorDruidLeaderSelector [WIP] Fix race condition in CuratorDruidLeaderSelector Jun 4, 2024
@kfaraz kfaraz mentioned this pull request Jun 7, 2024
Merged
10 tasks
@github-actions
Copy link

github-actions bot commented Aug 14, 2024

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

@github-actions github-actions bot added the stale label Aug 14, 2024
@kfaraz kfaraz removed the stale label Aug 16, 2024
@github-actions
Copy link

github-actions bot commented Oct 16, 2024

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

@github-actions github-actions bot added the stale label Oct 16, 2024
@github-actions
Copy link

github-actions bot commented Nov 14, 2024

This pull request/issue has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions bot closed this Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kfaraz