Skip to content

Refactor DDL Module to support large number tables with frequently ddl #7630

@hongyunyan

Description

@hongyunyan

Enhancement

Our DDL module is too complex and inefficient, it can't work well under a large number tables with multiple ddl. In order to upport for a larger number of tables, we need to refactor ddl module.

For the new DDL infrastructure, we will do the following settings:

  • Make each table deal with its own inner schema changes( such as add columns, drop columns, modify columns, ..,etc), to avoid that multiple tables affect each other's performance due to schema changes
  • Only the schema changes between tables( such as create table, truncate table .. etc) will still sync in the background thread.
  • For the table without tiflash replica or table with no data, we will not create a physical table in tiflash, to reduce unnecessary space occupation.
  • For each keyspace, we use an independent schemaSyncer to update the schema synchronization to support the scalability under a large number of keyspaces.
  • Optimize DDLGuard(Optimize DDLGuard #7636)
  • Enhance TiFlashMetrics containing keyspace info(Enhance TiFlashMetrics containing keyspace info  #7640)
  • Don't store table schema locally by using schema.sql(Don't store table schema locally by using schema.sql #7311)
  • Parallel sync schema to speed up(Parallel sync schema to speed up #7641)

Metadata

Metadata

Assignees

Labels

report/customerCustomers have encountered this bug.type/enhancementThe issue or PR belongs to an enhancement.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions