Background

A region becomes leader while a new log backup task is initialized.

Reproduction Timeline

0. The log backup endpoint register ranges of the new log backup task to the observers.
   1/2. The region becomes pre-candidate, which trigger to send Task::ModifyObserve(Observe::Stop) to the Endpoint::scheduler.
   3/4. The region becomes candidate, which trigger to send Task::ModifyObserve(Observe::Stop) to the Endpoint::scheduler.
   5/6. The region becomes leader, which trigger to send Task::ModifyObserve(Observe::Start{ObserveId: 191299}) to the Endpoint::scheduler.

After the step 6, the Endpoint::scheduler has the followed messages that have not been consumed.

Task::ModifyObserve(Observe::Stop)
Task::ModifyObserve(Observe::Stop)
Task::ModifyObserve(Observe::Start{ObserveId: 191299})

7. The initialization of new log backup scan the region and send Observe::Start{ObserveId:191374} to the Observer::scheduler.

After the step 7, the Observer::scheduler has the followed messages that have not been consumed.

Observe::Start{ObserveId:191374}

8. The Endpoint starts to consumes the above messages in the scheduler, and sends them to Observer::scheduler.

After the step 8, the Observer::scheduler has the followed messages that have not been consumed.

Observe::Start{ObserveId:191374}
Observe::Stop
Observe::Stop
Observe::Start{ObserveId: 191299}

This order causes the region to not be observed with the error stale observe id ObserveId(191299), current id: ObserveId(191374).