@tdcmeehan dumb question : It wasn't clear looking at the PR description why the order of heartbeats is important.
Say Node A sends heartbeats to Node B. Node A dies sometime and If Heartbeat 101 comes after Heartbeat 102, the worst case scenario is Node B thinks Node A is alive for a delta time it wasn't supposed to be alive because eventually Heartbeat 103 will not arrive. Is that the problem we are trying to solve.?

@tdcmeehan dumb question : It wasn't clear looking at the PR description why the order of heartbeats is important. Say Node A sends heartbeats to Node B. Node A dies sometime and If Heartbeat 101 comes after Heartbeat 102, the worst case scenario is Node B thinks Node A is alive for a delta time it wasn't supposed to be alive because eventually Heartbeat 103 will not arrive. Is that the problem we are trying to solve.?

The exact issue is there are certain scenarios where the state has changed and it's not desirable to to revert to a previous revision of the state. This also enables us to send heartbeats more aggressively, for example on state transitions, and to lower the interval for in between events such as queueing and running states.