-
Notifications
You must be signed in to change notification settings - Fork 6.1k
Description
Enhancement
We want to use tidb-lightning to import data to a cluster that is running online services. To reduce the impact on online services, we need to reduce the impact of local-backend to the table-level.
We have to do several enhancements:
First, tidb-lightning shouldn't disable the whole PD scheduler during importing. Instead, lightning should only disable the scheduler on the region that is being imported.tikv/pd#4695 supported TTL for region label. We can use the region label to disable the scheduler on a specific key range. With the help of TTL, we don't need to worry that the scheduler won't be recovered when tidb-lightning is crashed. For region label, please see tikv/pd#3839 for more details.
Second, tidb-lightning need to provide a mechanism to limit the write speed to each TiKV store. We can add a new config tikv-importer.store-write-bwlimit and limit the write speed to TiKV nodes.