I haven't seen an issue for this yet, so I'm opening this now for discussion.

Purpose: Achieve quorum in small clusters with 1-2 shards.

Desired properties:

• Allow both voting and non-voting replicas in the same cluster.
• Backward compatibility with nodes that don't recognize voting replicas.

How:

• New config allow-voting-replicas which controls two things: whether a node
  is a voting replica and whether it counts votes from other voting replicas.
• A flag in the cluster bus that marks that a node counts voting replicas (a bit
  in clusterMsg and clusterMsgDataGossip and in clusterNode).
• An aux field in nodes.conf.
• Include voting replicas in server.cluster->size, which affects the quorum.
• Count votes from voting replicas when receiving FAILOVER AUTH ACK.
• Count voting replicas in failure detection, i.e. when counting failure reports
  to mark a node as FAIL.

Two competing ways of counting votes

1. A node without this feature counts the quorum (i.e. server.cluster->size)
   as before, and counts votes from masters only.

2. A node with this feature (allow-voting-replicas yes) includes the voting
   replicas in the quorum and counts their votes.

Each replica decides (based on the config) how it counts the votes. If a node
considers itself a winner, it promotes itself to master. Therefore, there is a
risk that two nodes will consider themselves the winner with the same epoch and
both promote themselves to master.

Ideally, all nodes in a cluster count votes and quorum in the same way, but
in some setups, at least during rolling upgrade, there will be two ways of
counting.

Worst case scenario

Two replicas A and B, with the same replica rank, initiate an election at
roughly the same time. A majority of the masters vote for replica A while if the
voting replicas are counted, there's a majority for replica B. Due to different
configuration of A and B, both replicas consider themselves the winner. Both
promote themselves to master with the same epoch and broadcast PONG to the rest
of the cluster.

When A and B discover they have the same epoch, the conflict resolution
algorithm kicks in and favors one of them, say A, which bumps its epoch.

When other nodes discover that B claims the same slots as A and A has the
greater epoch, they send an UPDATE to B. B acknowledges that the exact same set
of slots is owned by A, so B assumes that it was failed over by A and turns
itself into a replica of A.

Any writes that were done to B while B considered itself a master are discarded,
so there is potential data loss here.

How to prevent and mitigate this scenario?

Factors that mitigate the risk:

• Often, the result of the election is the same in both ways of counting.

• Replicas typically don't initiate an election at the same time. Each replica
  has a randomized delay before initiating an election.

To further mitigate the risk that a voting replica and a non-voting replica
initiate an election at the same time, the voting replica can back off by giving
itself a higher replica rank than the non-voting replica, if both nodes share
the same replication offset.

When the two winners scenario has happened, we can try to sort out the situation
faster. The epoch conflict resolution algorithm can be adjusted so that if a
node find that it looses the conflict resolution and both nodes claim the exact
same set of slots, the losing node turns itself into a replica of the winning
node. There is no need to wait for UPDATE messages from other nodes to do this.