Leader transfer should be postponed until on-going split is finished #12410
Description
Development Task
Leader transfer should be postponed util on-going split is finished, otherwise newly splited region can not campaign leader immediately in the following case.
| Time | tikv_10_19_85_6_20163 | tikv_10_19_85_4_20163 |
|---|---|---|
| 14:07:40.495 | region 33977192 starts transfer leader | / |
| 14:07:40.496 | / | region 33977192 splits region 34144801 |
| 14:07:40.496 | / | region 33977192 became leader |
| 14:07:40.502 | region 33977192 became follower | / |
| 14:07:40.502 | region 33977192 splits region 34144801 | / |
It causes TiDB write/read latency spike and TiCDC replication spike.
tikv_10_19_85_4_20163.log:6170:[2022/04/18 14:06:18.991 +08:00] [Info] [raft.rs:783] ["became follower at term 362"] [term=362] [raft_id=34144658] [region_id=33977192]
tikv_10_19_85_4_20163.log:6559:[2022/04/18 14:07:40.495 +08:00] [Info] [raft.rs:1739] ["[term 362] received MsgTimeoutNow from 34144442 and starts an election to get leadership."] [from=34144442] [term=362] [raft_id=34144658] [region_id=33977192]
tikv_10_19_85_4_20163.log:6561:[2022/04/18 14:07:40.495 +08:00] [Info] [raft.rs:807] ["became candidate at term 363"] [term=363] [raft_id=34144658] [region_id=33977192]
tikv_10_19_85_4_20163.log:6566:[2022/04/18 14:07:40.495 +08:00] [Info] [apply.rs:1234] ["execute admin command"] [command="cmd_type: BatchSplit splits { requests { split_key: 7480000000000018FF435F7280000004AEFF9C78BD0000000000FA new_region_id: 34144801 new_peer_ids: 34144802 new_peer_ids: 34144803 new_peer_ids: 34144804 new_peer_ids: 34144805 } right_derive: true }"] [index=4807851] [term=362] [peer_id=34144658] [region_id=33977192]
tikv_10_19_85_4_20163.log:6568:[2022/04/18 14:07:40.496 +08:00] [Info] [peer.rs:2106] ["insert new region"] [region="id: 34144801 start_key: 7480000000000018FF435F7280000004AEFF9A4E430000000000FA end_key: 7480000000000018FF435F7280000004AEFF9C78BD0000000000FA region_epoch { conf_ver: 40837 version: 22856 } peers { id: 34144802 store_id: 11 } peers { id: 34144803 store_id: 14 } peers { id: 34144804 store_id: 1 } peers { id: 34144805 store_id: 9 }"] [region_id=34144801]
tikv_10_19_85_4_20163.log:6575:[2022/04/18 14:07:40.496 +08:00] [Info] [raft.rs:874] ["became leader at term 363"] [term=363] [raft_id=34144658] [region_id=33977192]
tikv_10_19_85_4_20163.log:6675:[2022/04/18 14:07:52.541 +08:00] [Info] [raft.rs:1739] ["[term 6] received MsgTimeoutNow from 34144805 and starts an election to get leadership."] [from=34144805] [term=6] [raft_id=34144804] [region_id=34144801]
tikv_10_19_85_5_20160.log:4737:[2022/04/18 14:07:40.495 +08:00] [Info] [apply.rs:1234] ["execute admin command"] [command="cmd_type: BatchSplit splits { requests { split_key: 7480000000000018FF435F7280000004AEFF9C78BD0000000000FA new_region_id: 34144801 new_peer_ids: 34144802 new_peer_ids: 34144803 new_peer_ids: 34144804 new_peer_ids: 34144805 } right_derive: true }"] [index=4807851] [term=362] [peer_id=34144322] [region_id=33977192]
tikv_10_19_85_5_20160.log:4739:[2022/04/18 14:07:40.495 +08:00] [Info] [peer.rs:2106] ["insert new region"] [region="id: 34144801 start_key: 7480000000000018FF435F7280000004AEFF9A4E430000000000FA end_key: 7480000000000018FF435F7280000004AEFF9C78BD0000000000FA region_epoch { conf_ver: 40837 version: 22856 } peers { id: 34144802 store_id: 11 } peers { id: 34144803 store_id: 14 } peers { id: 34144804 store_id: 1 } peers { id: 34144805 store_id: 9 }"] [region_id=34144801]
tikv_10_19_85_5_20160.log:4745:[2022/04/18 14:07:40.496 +08:00] [Info] [raft.rs:783] ["became follower at term 363"] [term=363] [raft_id=34144322] [region_id=33977192]
tikv_10_19_85_6_20162.log:3775:[2022/04/18 14:07:35.113 +08:00] [Info] [raft.rs:783] ["became follower at term 362"] [term=362] [raft_id=34144780] [region_id=33977192]
tikv_10_19_85_6_20162.log:3789:[2022/04/18 14:07:40.496 +08:00] [Info] [raft.rs:783] ["became follower at term 363"] [term=363] [raft_id=34144780] [region_id=33977192]
tikv_10_19_85_6_20162.log:3791:[2022/04/18 14:07:40.496 +08:00] [Info] [apply.rs:1234] ["execute admin command"] [command="cmd_type: BatchSplit splits { requests { split_key: 7480000000000018FF435F7280000004AEFF9C78BD0000000000FA new_region_id: 34144801 new_peer_ids: 34144802 new_peer_ids: 34144803 new_peer_ids: 34144804 new_peer_ids: 34144805 } right_derive: true }"] [index=4807851] [term=362] [peer_id=34144780] [region_id=33977192]
tikv_10_19_85_6_20162.log:3793:[2022/04/18 14:07:40.497 +08:00] [Info] [peer.rs:2106] ["insert new region"] [region="id: 34144801 start_key: 7480000000000018FF435F7280000004AEFF9A4E430000000000FA end_key: 7480000000000018FF435F7280000004AEFF9C78BD0000000000FA region_epoch { conf_ver: 40837 version: 22856 } peers { id: 34144802 store_id: 11 } peers { id: 34144803 store_id: 14 } peers { id: 34144804 store_id: 1 } peers { id: 34144805 store_id: 9 }"] [region_id=34144801]
tikv_10_19_85_6_20162.log:3909:[2022/04/18 14:07:52.541 +08:00] [Info] [raft.rs:1389] ["sends MsgTimeoutNow to 34144804 immediately as 34144804 already has up-to-date log"] [lead_transferee=34144804] [raft_id=34144805] [region_id=34144801]
tikv_10_19_85_6_20163.log:4450:[2022/04/18 14:07:40.495 +08:00] [Info] [raft.rs:1289] ["sent MsgTimeoutNow to 34144658 after received MsgAppResp"] [from=34144658] [raft_id=34144442] [region_id=33977192]
tikv_10_19_85_6_20163.log:4451:[2022/04/18 14:07:40.501 +08:00] [Info] [apply.rs:1234] ["execute admin command"] [command="cmd_type: BatchSplit splits { requests { split_key: 7480000000000018FF435F7280000004AEFF9C78BD0000000000FA new_region_id: 34144801 new_peer_ids: 34144802 new_peer_ids: 34144803 new_peer_ids: 34144804 new_peer_ids: 34144805 } right_derive: true }"] [index=4807851] [term=362] [peer_id=34144442] [region_id=33977192]
tikv_10_19_85_6_20163.log:4456:[2022/04/18 14:07:40.502 +08:00] [Info] [raft.rs:783] ["became follower at term 363"] [term=363] [raft_id=34144442] [region_id=33977192]
tikv_10_19_85_6_20163.log:4460:[2022/04/18 14:07:40.502 +08:00] [Info] [peer.rs:2106] ["insert new region"] [region="id: 34144801 start_key: 7480000000000018FF435F7280000004AEFF9A4E430000000000FA end_key: 7480000000000018FF435F7280000004AEFF9C78BD0000000000FA region_epoch { conf_ver: 40837 version: 22856 } peers { id: 34144802 store_id: 11 } peers { id: 34144803 store_id: 14 } peers { id: 34144804 store_id: 1 } peers { id: 34144805 store_id: 9 }"] [region_id=34144801]