-
Notifications
You must be signed in to change notification settings - Fork 755
Description
Enhancement Task
After a PD instance is restarted, it may take some time to load region information. If a PD leader doesn't have all up-to-date region information, it may cause some problems.
When updating/upgrading PD, the TiDB Operator restarts PD instances one by one, e.g. if the current PD instance is ready, the next one will be restarted. But, this ready doesn't include region information sync.
A case with problems:
- All PD instances run for a while and with all region information synced
- PD-2 is the leader, and an updating operation is triggered
- TiDB Operator calls PD API to transfer leader from PD-2 to PD-1
- TiDB Operator restarts PD-2 and waits for PD-2 to be ready (but without the additional wait for region information sync)
- TiDB Operator calls PD API to transfer leader from PD-1 to PD-2
- As the region information in PD-2 is not synced, problems happen
At this point, the problem arises that PD-2 is elected as the leader, but it can't provide services related to region query until regions are loaded, it can only provide tso services.
The current workaround is to wait a while after the pd rolls reboot until the load region is complete and then let it be the leader
Typically, for a 10 million cluster, five to ten minutes is enough.
But in the longer term, it is more flexible for pd to provide an interface to query if the load region is complete.