Ceph » rbd » nvme-of

Motivation - to send less bytes on the network , to spend less real time in NVMeofGw Monitor upon accept beacons from all Gateways,
Preparation for fast NVMeof Failover - we want to send Beacons each second.

Simple protocol was designed for handle exceptions like beacon sequence out of order.
1. Sequence number is sent in each beacon (uint64)
2. Monitor remembers last beacon's correct sequence
3. 2 fields added to Monitorclient map - expected-beacon-sequence, out-of-order indication(bool)

Logic:

- MonitorClient calculates a difference from previous beacon when it sends a beacon.
resolution is Subsystem - if some field of parent Subsystem was changed the whole subsystem considered changed and change-descriptor is added under Subsystem.
- Monitor applies the beacon considering it sends just differences from the previous beacon
- If beacon sequence is not equal to monitor's last correct sequence + 1
then monitor responds with ack (monclient map) expected-beacon-sequence =last correct sequence +1 , out-of-order indication = true
- When monitor client receives out of order indication it resets its beacon sequence to expected-beacon-sequence ad sends all subsystems in the next beacon (not sending the diff this time)
- When monitor receives the good sequence after out-of-order being detected it invalidates all its subsystems before applying the beacon
- When new leader is elected monitor forces out-of-order indication for all GWs