In order to support #56197 and #57184, we'd like to be able to relocate the replicas of a range such that a store that has a VOTER should have a NON_VOTER or vice-versa. Currently, this would require multiple AdminChangeReplicas requests. For instance, in order to promote a NON_VOTER to a VOTER on store X, we would have to first remove the NON_VOTER and then add a new VOTER to X. This entails unnecessary data movement and potentially leaves a range in violation of its constraints in the interim.
We already have the machinery in place to be able to perform such promotions, demotions, and even VOTER <-> NON_VOTER swaps atomically as we support these things for the LEARNER replica type. We should support the same functionality for NON_VOTERs.
#58627 adds support in AdminRelocateRange to be able to execute such a swap, much in the same way we implicitly swap LEARNERs with VOTER_INCOMINGs during rebalancing. [DONE]
The final piece of the puzzle will be to get the allocator to incorporate this ability into its rebalancing decisions. It will always be preferable to promote/swap an existing non-voter to a voting replica over adding a new voter, as the new voter has to get upreplicated via a snapshot whereas the existing non-voter just needs to catch up to the raft log index corresponding to the promotion.
In order to support #56197 and #57184, we'd like to be able to relocate the replicas of a range such that a store that has a
VOTERshould have aNON_VOTERor vice-versa. Currently, this would require multipleAdminChangeReplicasrequests. For instance, in order to promote aNON_VOTERto aVOTERon store X, we would have to first remove theNON_VOTERand then add a newVOTERto X. This entails unnecessary data movement and potentially leaves a range in violation of its constraints in the interim.We already have the machinery in place to be able to perform such promotions, demotions, and even
VOTER<->NON_VOTERswaps atomically as we support these things for theLEARNERreplica type. We should support the same functionality forNON_VOTERs.#58627 adds support in
AdminRelocateRangeto be able to execute such a swap, much in the same way we implicitly swapLEARNERs withVOTER_INCOMINGs during rebalancing. [DONE]The final piece of the puzzle will be to get the allocator to incorporate this ability into its rebalancing decisions. It will always be preferable to promote/swap an existing non-voter to a voting replica over adding a new voter, as the new voter has to get upreplicated via a snapshot whereas the existing non-voter just needs to catch up to the raft log index corresponding to the promotion.