Skip to content

Make cluster meet reliable under link failures#461

Merged
madolson merged 11 commits into
valkey-io:unstablefrom
srgsanky:unstable
Jun 17, 2024
Merged

Make cluster meet reliable under link failures#461
madolson merged 11 commits into
valkey-io:unstablefrom
srgsanky:unstable

Conversation

@srgsanky

@srgsanky srgsanky commented May 8, 2024

Copy link
Copy Markdown
Contributor

When there is a link failure while an ongoing MEET request is sent the sending node stops sending anymore MEET and starts sending PINGs. Since every node responds to PINGs from unknown nodes with a PONG, the receiving node never adds the sending node. But the sending node adds the receiving node when it sees a PONG. This can lead to asymmetry in cluster membership. This changes makes the sender keep sending MEET until it sees a PONG, avoiding the asymmetry.

@srgsanky

srgsanky commented May 8, 2024

Copy link
Copy Markdown
Contributor Author

Posting this for initial comments. I can migrate the test based on the new framework once #442 is merged.

@codecov

codecov Bot commented May 8, 2024

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.20%. Comparing base (168da8b) to head (7ac84b6).
Report is 33 commits behind head on unstable.

Additional details and impacted files
@@             Coverage Diff              @@
##           unstable     #461      +/-   ##
============================================
- Coverage     70.22%   70.20%   -0.02%     
============================================
  Files           109      109              
  Lines         59956    59967      +11     
============================================
- Hits          42104    42102       -2     
- Misses        17852    17865      +13     
Files Coverage Δ
src/cluster_legacy.c 86.52% <100.00%> (+0.07%) ⬆️
src/debug.c 54.07% <100.00%> (+0.13%) ⬆️

... and 9 files with indirect coverage changes

@srgsanky

srgsanky commented May 8, 2024

Copy link
Copy Markdown
Contributor Author

@madolson @hpatro @PingXie can one of you help review this change?

Comment thread tests/cluster/tests/30-reliable-meet.tcl Outdated
Comment thread src/cluster_legacy.c Outdated
Comment thread src/cluster_legacy.c Outdated
Comment thread src/cluster_legacy.c Outdated
Comment thread src/server.h Outdated
@hpatro

hpatro commented May 13, 2024

Copy link
Copy Markdown
Contributor

I think it's worth investing on this redis/redis#11095 to avoid this issue altogether.

@srgsanky

Copy link
Copy Markdown
Contributor Author

I think it's worth investing on this redis/redis#11095 to avoid this issue altogether.

Thanks, I wasn't aware of this linked issue. IMO these two issues can be solved independently. The linked issue tries to make the admin experience better for MEET command where as this PR tries to address a specific gap in MEET implementation.

  • With SYNC MEET, we will have to make changes to admin client timeout. This timeout can possible trickle up the stack in a control plane implementation.
  • If we choose to attempt handshake for a longer period of time, we either have to filter out nodes in handshake in cluster nodes output for a non-admin client or make the clients filter out the nodes with this new flag. This can require a client side change to avoid connecting to a node in handshake, experiencing availability issues.

The problem addressed in this PR (asymmetric cluster membership) can happen with SYNC MEET as well due to link failures. So, it is worth solving it. The handshake nodes will still be removed after the handshake timeout (same as node_timeout of 15s). Wdyt?

@madolson

Copy link
Copy Markdown
Member

The problem addressed in this PR (asymmetric cluster membership) can happen with SYNC MEET as well due to link failures. So, it is worth solving it. The handshake nodes will still be removed after the handshake timeout (same as node_timeout of 15s). Wdyt?

Yeah, I still believe this a problem even with the #11095.

@zuiderkwast

Copy link
Copy Markdown
Contributor

Awesome material for our next release which will be full of cluster improvements. Is it worth mentioning in release notes?

Btw @srgsanky you need to commit with -s. See the instructions on the DCO CI job's details page.

@madolson madolson added the release-notes This issue should get a line item in the release notes label May 13, 2024
@madolson

Copy link
Copy Markdown
Member

Awesome material for our next release which will be full of cluster improvements. Is it worth mentioning in release notes?

I would also be inclined to backport it.

Comment thread tests/cluster/tests/30-reliable-meet.tcl Outdated
@srgsanky

Copy link
Copy Markdown
Contributor Author

Awesome material for our next release which will be full of cluster improvements. Is it worth mentioning in release notes?

Btw @srgsanky you need to commit with -s. See the instructions on the DCO CI job's details page.

When I tried to merge the new changes into my fork, I ended up with a merge commit

* 2ff9879fa (HEAD -> unstable, origin/unstable, origin/HEAD) Moved test under unit and addressed other comments
*   b826ef77a Merge branch 'valkey-io:unstable' into unstable
|\
| * d52c8f30e Include stddef in zmalloc.h (#516)
| * dcc9fd4fe Resolve numtests counter error (#514)
...
| * 315b7573c Update server function's name to valkey (#456)
* | 49a884c06 Make cluster meet reliable under link failures
|/
* 4e944cede Migrate kvstore.c unit tests to new test framework. (#446)

I want to signoff just 49a884c, but the rebase is adding a signoff to all commits 315b757..d52c8f3 which are not made by me.

Do you have any recommendation to fix this?

As an alternate option, I can start fresh and add a new commit from the tip of unstable. I am not sure if I will be able to reuse this PR.

@srgsanky srgsanky force-pushed the unstable branch 2 times, most recently from d8aa71c to 2ff9879 Compare May 19, 2024 21:22
@zuiderkwast

Copy link
Copy Markdown
Contributor

I believe it's possible to undo a merge by git reset --hard 49a884c06 (the commit before the merge commit), then rebase to add the --signoff, then do git merge unstable again. The commit you added after merge commit can be cherry-picked after all this. Just remember the commit id.

If nothing works, then it's always possible to start from scratch with a new branch and cherry-pick all your commits into it. Then you can rename the branches and force-push to this PR's branch.

@madolson

madolson commented May 20, 2024

Copy link
Copy Markdown
Member

@srgsanky The commit missing the DCO is just the top one. You should just be able to do git commit -s --amend with a no-op and force push over what you have. It's the base commit, nevermind. I believe git rebase -i HEAD~3 should allow you to manually add the signature, maybe there is a better way to do it.

@madolson madolson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some minor nitpicks around the tests, it overall LGTM.

Comment thread tests/unit/cluster/cluster-reliable-meet.tcl Outdated
Comment thread tests/unit/cluster/cluster-reliable-meet.tcl Outdated
Comment thread tests/unit/cluster/cluster-reliable-meet.tcl Outdated
Comment thread tests/cluster/tests/includes/init-tests.tcl Outdated
Comment thread tests/unit/cluster/cluster-reliable-meet.tcl Outdated
Comment thread src/cluster_legacy.c Outdated
srgsanky added 3 commits May 19, 2024 21:07
When there is a link failure while an ongoing MEET request is sent the
sending node stops sending anymore MEET and starts sending PINGs. Since
every node responds to PINGs from unknown nodes with a PONG, the
receiving node never adds the sending node. But the sending node adds
the receiving node when it sees a PONG. This can lead to asymmetry in
cluster membership. This changes makes the sender keep sending MEET
until it sees a PONG, avoiding the asymmetry.

Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
@srgsanky

Copy link
Copy Markdown
Contributor Author

B isn't processing these right? It's just immediately dropping the first three and not processing them, it only ever processes the 4th one once correct?

Correct. It starts processing when we drop the filter - which can be 4th or later.

@srgsanky

Copy link
Copy Markdown
Contributor Author

I believe it's possible to undo a merge by git reset --hard 49a884c (the commit before the merge commit), then rebase to add the --signoff, then do git merge unstable again. The commit you added after merge commit can be cherry-picked after all this. Just remember the commit id.

This worked. Thanks!

It's the base commit, nevermind. I believe git rebase -i HEAD~3 should allow you to manually add the signature, maybe there is a better way to do it.

I tried this and all the commits in the other branch of the merge was also annotated with my signoff. So, I decided to ask you folks for the best approach.

btw is there any reasoning behind the requirement for the signoff?

Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
@madolson

Copy link
Copy Markdown
Member

btw is there any reasoning behind the requirement for the signoff?

Technically we adopted it because it's an LF requirement, but it's also a good practice to force a trail of who committed what.

@PingXie PingXie left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this change LGTM overall.

Comment thread src/server.h Outdated
Comment thread src/cluster_legacy.c
Comment thread src/cluster_legacy.c Outdated
Comment thread src/cluster_legacy.c Outdated
Comment thread src/debug.c Outdated
Comment thread tests/unit/cluster/cluster-reliable-meet.tcl Outdated
srgsanky added 5 commits May 20, 2024 19:15
1. Reworked code comment
1. Added serverLogs
1. Renamed debug variable
1. Made close link filter to be directly coupled with drop filter

Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
Multiple MEETs will be handled like a normal PING message.

Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
@srgsanky

srgsanky commented May 29, 2024

Copy link
Copy Markdown
Contributor Author

The clang-format checker is currently failing due to changes introduced by another PR. Mentioned this in #118 (comment)

@LiiNen

LiiNen commented May 29, 2024

Copy link
Copy Markdown
Contributor

sorry. maybe i missed some. fixed #570

PingXie pushed a commit that referenced this pull request May 29, 2024
ref:
- #118 (my pervious change)
- #461 (issuing that clang
format checker fails due to my change)

There was an issue that clang-format cheker failed.
I don't know why I missed it and why it didn't catch.

just running `clang-format -i bitops.c` was all.

Signed-off-by: LiiNen <kjeonghoon065@gmail.com>
Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
Comment thread src/server.h Outdated
Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>

@PingXie PingXie left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Comment thread src/debug.c
Comment thread src/debug.c
Comment thread src/cluster_legacy.c
Comment thread src/cluster_legacy.c
Comment thread tests/unit/cluster/cluster-multiple-meets.tcl
Comment thread tests/unit/cluster/cluster-multiple-meets.tcl
Comment thread tests/unit/cluster/cluster-multiple-meets.tcl
@madolson madolson merged commit a81c320 into valkey-io:unstable Jun 17, 2024
@madolson

Copy link
Copy Markdown
Member

@PingXie Sorry, I saw your approval and had intended to merge this on Friday but ran out of time and missed that you had comments. They seemed like small comments, so @srgsanky feel free to address, I commented on one of them.

@PingXie

PingXie commented Jun 17, 2024

Copy link
Copy Markdown
Member

No worries. These can all be addressed incrementally.

PingXie pushed a commit to PingXie/valkey that referenced this pull request Jul 8, 2024
When there is a link failure while an ongoing MEET request is sent the
sending node stops sending anymore MEET and starts sending PINGs. Since
every node responds to PINGs from unknown nodes with a PONG, the
receiving node never adds the sending node. But the sending node adds
the receiving node when it sees a PONG. This can lead to asymmetry in
cluster membership. This changes makes the sender keep sending MEET
until it sees a PONG, avoiding the asymmetry.

---------

Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
PingXie pushed a commit to PingXie/valkey that referenced this pull request Jul 9, 2024
When there is a link failure while an ongoing MEET request is sent the
sending node stops sending anymore MEET and starts sending PINGs. Since
every node responds to PINGs from unknown nodes with a PONG, the
receiving node never adds the sending node. But the sending node adds
the receiving node when it sees a PONG. This can lead to asymmetry in
cluster membership. This changes makes the sender keep sending MEET
until it sees a PONG, avoiding the asymmetry.

---------

Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
PingXie pushed a commit to PingXie/valkey that referenced this pull request Jul 9, 2024
When there is a link failure while an ongoing MEET request is sent the
sending node stops sending anymore MEET and starts sending PINGs. Since
every node responds to PINGs from unknown nodes with a PONG, the
receiving node never adds the sending node. But the sending node adds
the receiving node when it sees a PONG. This can lead to asymmetry in
cluster membership. This changes makes the sender keep sending MEET
until it sees a PONG, avoiding the asymmetry.

---------

Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
Signed-off-by: Ping Xie <pingxie@google.com>
PingXie pushed a commit that referenced this pull request Jul 10, 2024
When there is a link failure while an ongoing MEET request is sent the
sending node stops sending anymore MEET and starts sending PINGs. Since
every node responds to PINGs from unknown nodes with a PONG, the
receiving node never adds the sending node. But the sending node adds
the receiving node when it sees a PONG. This can lead to asymmetry in
cluster membership. This changes makes the sender keep sending MEET
until it sees a PONG, avoiding the asymmetry.

---------

Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>
Signed-off-by: Ping Xie <pingxie@google.com>
enjoy-binbin added a commit to enjoy-binbin/redis that referenced this pull request May 25, 2026
When there is a link failure while an ongoing MEET request is sent the
sending node stops sending anymore MEET and starts sending PINGs. Since
every node responds to PINGs from unknown nodes with a PONG, the
receiving node never adds the sending node. But the sending node adds
the receiving node when it sees a PONG. This can lead to asymmetry in
cluster membership. This changes makes the sender keep sending MEET
until it sees a PONG, avoiding the asymmetry.

Signed-off-by: Sankar <1890648+srgsanky@users.noreply.github.com>

See valkey-io/valkey#461
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cluster release-notes This issue should get a line item in the release notes

Projects

Status: Done
Status: Done

Development

Successfully merging this pull request may close these issues.

7 participants