Skip to content

lightning: support table-level local-backend #35148

@sleepymole

Description

@sleepymole

Enhancement

We want to use tidb-lightning to import data to a cluster that is running online services. To reduce the impact on online services, we need to reduce the impact of local-backend to the table-level.

We have to do several enhancements:

First, tidb-lightning shouldn't disable the whole PD scheduler during importing. Instead, lightning should only disable the scheduler on the region that is being imported.tikv/pd#4695 supported TTL for region label. We can use the region label to disable the scheduler on a specific key range. With the help of TTL, we don't need to worry that the scheduler won't be recovered when tidb-lightning is crashed. For region label, please see tikv/pd#3839 for more details.

Second, tidb-lightning need to provide a mechanism to limit the write speed to each TiKV store. We can add a new config tikv-importer.store-write-bwlimit and limit the write speed to TiKV nodes.

Metadata

Metadata

Assignees

Labels

component/lightningThis issue is related to Lightning of TiDB.type/enhancementThe issue or PR belongs to an enhancement.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions