What is changed and how it works?

Issue Number: Close #18243

What's Changed:

Keep the order of observation IDs consistent with the order in which they were received

Additional notes

There are always three observe operations (Stop[Pre-Candidate], Stop[Candidate] and Start[Leader]) generated when any peer becomes leader. But the observe operation Start[Leader] may lost due to no task registered yet. Besides, when a log backup task is being registered, the endpoint will send a observe operation Start[Scanned] for a leader.

Case 1: If the observe operation Start[Leader] is ignored because the task is not registered yet.
We can make sure the endpoint must get the region when a new task is being registered. We have the following execution order:

1. RaftStoreEvent::RoleChange -> region_info_accessor.scheduler
2. Start[Leader] -> backup_stream::Endpoint.scheduler [IGNORED]
3. register task ranges so that any observe operation won't be ignored
4. RegionInfoQuery::SeekRegion -> region_info_accessor.scheduler

In this case, the step 2 is already done, so we can make sure that the region update query is already in the queue of region_info_accessor.scheduler when the endpoint sends RegionInfoQuery::SeekRegion to the region_info_accessor. Therefore, the endpoint can get the region from seek_region.

Case2: If the endpoint can not get the region from seek_region.
We can make sure the observe operation Start[Leader] is not ignored. We have the following execution order:

1. register task ranges so that any observe operation won't be ignored
2. RegionInfoQuery::SeekRegion -> region_info_accessor.scheduler
3. RaftStoreEvent::RoleChange -> region_info_accessor.scheduler
4. Start[Leader] -> backup_stream::Endpoint.scheduler

In this case, the step 2 is already done, so we can make sure that the task range is registered. Therefore, the step 4 is not ignored and the observe operation Start[Leader] is scheduled.

In summary, the region_operator may meet the Start[Scanned] -> Stop[Pre-Candidate] -> Stop[Candidate] -> Start[Leader] and repeat scanning the region. But it won't lost the region if it is leader.

Related changes

• PR to update pingcap/docs/pingcap/docs-cn:
• Need to cherry-pick to the release branch

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Release note

Fix the issue that the log backup observer loses observation of a region.