Skip to content

Adding TiFlash node to a cluster with 20k tables is slow #1423

@JaySon-Huang

Description

@JaySon-Huang

If we add a TiFlash node to a TiDB cluster that already has 20k tables, TiFlash may be slow to start.
We will DO Full Schema sync when starting the new TiFlash node. And no matter those tables have TiFlash replica or not, we will create empty tables on the TiFlash node.

During this period, the TiFlash node

  • does not register itself as a store on PD
  • does not start reporting metrics
  • does not listen on the coprocessor port

The period could be quite long, especially when the user enables the feature "Encryption at Rest" (in DBaaS, for example), it takes more time for TiFlash to "be ready" on PD.

Actually, we met a situation that adding a TiFlash node (v4.0.9) to a cluster with more than 20k tables. It took about 15~20 minutes to become "normal" on the DBaaS dashboard. The user is confused that why TiFlash is "Unavailable" for 15 minutes after he/she create the node. (DBaaS show "unavailable" if a TiFlash is UP but not found in PD store list)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions