I very occasionally experience PartitionSelectorException when execute CLUSTER MEET or CLUSTER FAILOVER
an issue concerning this: How to reproduce PartitionSelectorException?
In Redis, while executing CLUSTER FAILOVER / CLUSTER MEET, if I repeatedly execute CLUSTER NODES commands and look at the results, they appear as follows:
For a brief moment, the cluster nodes command shows results where slots are not assigned.
CLUSTER MEET
3661a0608631299123dcb4ef435c5dd7355bc1ac xxx.xxx.xxx.237:6379@16379 handshake - 0 0 0 disconnected
42a9fe7c0c38dc67283429a9d9821cdd687b4bf0 xxx.xxx.xxx.235:6379@16379 handshake - 0 0 0 disconnected
42b0d7971450eff84e81da9bfde7708fce45d360 xxx.xxx.xxx.236:6379@16379 handshake - 0 0 0 disconnected
7364f8c4d9c098d65eae311d538c8dfcc4dd33a8 xxx.xxx.xxx.232:6379@16379 handshake - 0 0 0 disconnected
923812a5465d746e4e39636e52f8044c609dacce xxx.xxx.xxx.234:6379@16379 handshake - 0 0 0 disconnected
c69cfa86d79197dd67e3083272b4b632fe7ca4f1 xxx.xxx.xxx.238:6379@16379 handshake - 0 0 0 disconnected
e6a2f2ba406df7df321b7c58afbe44743177c33a xxx.xxx.xxx.233:6379@16379 handshake - 0 0 0 disconnected
c516943be39de0d92ef7171591905499968ffe6a :6379@16379 myself,master - 0 0 0 connected
b7ee0b55242040e44a7819b509a2f8b734eb6eef xxx.xxx.xxx.235:6379@16379 master - 0 1704265449102 0 connected
e7c05bbb2b7b0084b6c89af147b522b4740e2e1c xxx.xxx.xxx.234:6379@16379 master - 0 1704265449102 0 connected
c516943be39de0d92ef7171591905499968ffe6a xxx.xxx.xxx.239:6379@16379 myself,master - 0 0 0 connected
1efc246294fa201bafa4b9b4b0b8deef2e02d22b xxx.xxx.xxx.236:6379@16379 slave - 0 1704265449102 0 connected
2ed2beb3f24d8e2042b8df6dc17cdd03aa74fc23 xxx.xxx.xxx.237:6379@16379 slave - 0 1704265449102 0 connected
4635bfc1ee70b917b195df4f1c9183440c6e82e9 xxx.xxx.xxx.232:6379@16379 slave - 0 1704265449102 0 connected
70843e065d6a7adc21829fdc2bb33034397dec8d xxx.xxx.xxx.238:6379@16379 slave - 0 1704265449102 318 connected
bdb0b93a8f36ad0e80348a27e093eaca7912d803 xxx.xxx.xxx.233:6379@16379 slave - 0 1704265449102 0 connected
b7ee0b55242040e44a7819b509a2f8b734eb6eef xxx.xxx.xxx.235:6379@16379 master - 0 1704265449102 0 connected
e7c05bbb2b7b0084b6c89af147b522b4740e2e1c xxx.xxx.xxx.234:6379@16379 master - 0 1704265449102 0 connected
ebb05a5a59714d76c48c342ba831d1691a5a2d2a xxx.xxx.xxx.231:6379@16379 master - 0 1704265448000 0 disconnected
c516943be39de0d92ef7171591905499968ffe6a xxx.xxx.xxx.239:6379@16379 myself,master - 0 0 0 connected
4635bfc1ee70b917b195df4f1c9183440c6e82e9 xxx.xxx.xxx.232:6379@16379 slave - 0 1704265449102 0 connected
70843e065d6a7adc21829fdc2bb33034397dec8d xxx.xxx.xxx.238:6379@16379 slave - 0 1704265449102 318 connected
bdb0b93a8f36ad0e80348a27e093eaca7912d803 xxx.xxx.xxx.233:6379@16379 slave - 0 1704265449102 0 connected
2ed2beb3f24d8e2042b8df6dc17cdd03aa74fc23 xxx.xxx.xxx.237:6379@16379 slave b7ee0b55242040e44a7819b509a2f8b734eb6eef 0 1704265449102 0 connected
1efc246294fa201bafa4b9b4b0b8deef2e02d22b xxx.xxx.xxx.236:6379@16379 slave ebb05a5a59714d76c48c342ba831d1691a5a2d2a 0 1704265449102 0 connected
b7ee0b55242040e44a7819b509a2f8b734eb6eef xxx.xxx.xxx.235:6379@16379 master - 0 1704265449102 0 connected
e7c05bbb2b7b0084b6c89af147b522b4740e2e1c xxx.xxx.xxx.234:6379@16379 master - 0 1704265449102 0 connected
ebb05a5a59714d76c48c342ba831d1691a5a2d2a xxx.xxx.xxx.231:6379@16379 master - 0 1704265448000 0 disconnected
c516943be39de0d92ef7171591905499968ffe6a xxx.xxx.xxx.239:6379@16379 myself,master - 0 0 0 connected
70843e065d6a7adc21829fdc2bb33034397dec8d xxx.xxx.xxx.238:6379@16379 slave - 0 1704265449102 318 connected
bdb0b93a8f36ad0e80348a27e093eaca7912d803 xxx.xxx.xxx.233:6379@16379 slave - 0 1704265449102 0 connected
2ed2beb3f24d8e2042b8df6dc17cdd03aa74fc23 xxx.xxx.xxx.237:6379@16379 slave b7ee0b55242040e44a7819b509a2f8b734eb6eef 0 1704265449102 0 connected
4635bfc1ee70b917b195df4f1c9183440c6e82e9 xxx.xxx.xxx.232:6379@16379 slave e7c05bbb2b7b0084b6c89af147b522b4740e2e1c 0 1704265449102 0 connected
1efc246294fa201bafa4b9b4b0b8deef2e02d22b xxx.xxx.xxx.236:6379@16379 slave ebb05a5a59714d76c48c342ba831d1691a5a2d2a 0 1704265449102 0 connected
CLUSTER FAILOVER
024a8d902c0335776c83a05166c2c127fe6b3d62 xxx.xxx.xxx.239:6379@16379 master - 0 1704443648630 2 connected 5461-10922
3150168de85e73b690c6057ddbf428c2520a875e xxx.xxx.xxx.231:6379@16379 master - 0 1704443648000 941 connected 0-5460
439ef35d64cee63e421feef4e2d47ba834008a2b xxx.xxx.xxx.236:6379@16379 master - 0 1704443648528 3 connected 10923-16383
ee5e5c75024345b186eb3d23386fc5c32e21e0bd xxx.xxx.xxx.233:6379@16379 myself,slave 024a8d902c0335776c83a05166c2c127fe6b3d62 0 1704443646000 2 connected
de82dd1d1a9abae5a49d6102c463b2de83fee1d6 xxx.xxx.xxx.235:6379@16379 slave 024a8d902c0335776c83a05166c2c127fe6b3d62 0 1704443647422 2 connected
397e9d8a845afecacccd2e8e861f1d56cb31298c xxx.xxx.xxx.238:6379@16379 slave 3150168de85e73b690c6057ddbf428c2520a875e 0 1704443648000 941 connected
f850efbb082244da444d4dc5a67d1858904deeee xxx.xxx.xxx.232:6379@16379 slave 3150168de85e73b690c6057ddbf428c2520a875e 1704443648528 1704443646000 941 connected
35d051d7c9e78a80e8cf790a47e120f78ca81616 xxx.xxx.xxx.234:6379@16379 slave 439ef35d64cee63e421feef4e2d47ba834008a2b 0 1704443648428 3 connected
9e991599847f45dadda99aefc055d6d4d2c5dd5d xxx.xxx.xxx.237:6379@16379 slave 439ef35d64cee63e421feef4e2d47ba834008a2b 0 1704443646518 3 connected
024a8d902c0335776c83a05166c2c127fe6b3d62 xxx.xxx.xxx.239:6379@16379 master - 0 1704443648630 2 connected 5461-10922
439ef35d64cee63e421feef4e2d47ba834008a2b xxx.xxx.xxx.236:6379@16379 master - 0 1704443648528 3 connected 10923-16383
ee5e5c75024345b186eb3d23386fc5c32e21e0bd xxx.xxx.xxx.233:6379@16379 myself,slave 024a8d902c0335776c83a05166c2c127fe6b3d62 0 1704443646000 2 connected
de82dd1d1a9abae5a49d6102c463b2de83fee1d6 xxx.xxx.xxx.235:6379@16379 slave 024a8d902c0335776c83a05166c2c127fe6b3d62 0 1704443648000 2 connected
f850efbb082244da444d4dc5a67d1858904deeee xxx.xxx.xxx.232:6379@16379 slave 3150168de85e73b690c6057ddbf428c2520a875e 1704443648528 1704443646000 942 connected
35d051d7c9e78a80e8cf790a47e120f78ca81616 xxx.xxx.xxx.234:6379@16379 slave 439ef35d64cee63e421feef4e2d47ba834008a2b 0 1704443648428 3 connected
9e991599847f45dadda99aefc055d6d4d2c5dd5d xxx.xxx.xxx.237:6379@16379 slave 439ef35d64cee63e421feef4e2d47ba834008a2b 0 1704443648000 3 connected
3150168de85e73b690c6057ddbf428c2520a875e xxx.xxx.xxx.231:6379@16379 slave f850efbb082244da444d4dc5a67d1858904deeee 0 1704443648000 941 connected
397e9d8a845afecacccd2e8e861f1d56cb31298c xxx.xxx.xxx.238:6379@16379 slave f850efbb082244da444d4dc5a67d1858904deeee 0 1704443648000 941 connected
024a8d902c0335776c83a05166c2c127fe6b3d62 xxx.xxx.xxx.239:6379@16379 master - 0 1704443651000 2 connected 5461-10922
439ef35d64cee63e421feef4e2d47ba834008a2b xxx.xxx.xxx.236:6379@16379 master - 0 1704443651044 3 connected 10923-16383
f850efbb082244da444d4dc5a67d1858904deeee xxx.xxx.xxx.232:6379@16379 master - 0 1704443651000 942 connected 0-5460
ee5e5c75024345b186eb3d23386fc5c32e21e0bd xxx.xxx.xxx.233:6379@16379 myself,slave 024a8d902c0335776c83a05166c2c127fe6b3d62 0 1704443649000 2 connected
de82dd1d1a9abae5a49d6102c463b2de83fee1d6 xxx.xxx.xxx.235:6379@16379 slave 024a8d902c0335776c83a05166c2c127fe6b3d62 0 1704443650000 2 connected
35d051d7c9e78a80e8cf790a47e120f78ca81616 xxx.xxx.xxx.234:6379@16379 slave 439ef35d64cee63e421feef4e2d47ba834008a2b 0 1704443651000 3 connected
9e991599847f45dadda99aefc055d6d4d2c5dd5d xxx.xxx.xxx.237:6379@16379 slave 439ef35d64cee63e421feef4e2d47ba834008a2b 0 1704443651549 3 connected
3150168de85e73b690c6057ddbf428c2520a875e xxx.xxx.xxx.231:6379@16379 slave f850efbb082244da444d4dc5a67d1858904deeee 0 1704443651226 942 connected
397e9d8a845afecacccd2e8e861f1d56cb31298c xxx.xxx.xxx.238:6379@16379 slave f850efbb082244da444d4dc5a67d1858904deeee 0 1704443651448 942 connected
I have a suspicion that, during a cluster failover, the changes from the old master are being delayed for some reason.
It seems that PartitionSelectorException occurs when a PARTITIONS is selected based on a node that has not been properly updated.
Along with checking the knownNode, adding the total number of slots assigned in partitions might help prevent such cases.
Below are the changes I made to conduct the test.
kell0gg@ec84294
This cases is very very rare, and changing the conditions might reduce the efficiency in selecting partitions.
I am uncertain if adding other conditions beyond knownNode is the right approach. Therefore, I would greatly appreciate hearing your thoughts on this matter.
Thanks. :)
I very occasionally experience PartitionSelectorException when execute
CLUSTER MEETorCLUSTER FAILOVERan issue concerning this: How to reproduce PartitionSelectorException?
In Redis, while executing CLUSTER FAILOVER / CLUSTER MEET, if I repeatedly execute
CLUSTER NODEScommands and look at the results, they appear as follows:For a brief moment, the cluster nodes command shows results where slots are not assigned.
CLUSTER MEET
CLUSTER FAILOVER
I have a suspicion that, during a cluster failover, the changes from the old master are being delayed for some reason.
It seems that PartitionSelectorException occurs when a PARTITIONS is selected based on a node that has not been properly updated.
Along with checking the
knownNode, adding the total number of slots assigned in partitions might help prevent such cases.Below are the changes I made to conduct the test.
kell0gg@ec84294
This cases is very very rare, and changing the conditions might reduce the efficiency in selecting partitions.
I am uncertain if adding other conditions beyond
knownNodeis the right approach. Therefore, I would greatly appreciate hearing your thoughts on this matter.Thanks. :)