Skip to content

Make ddl operations for placement rule atomic #28159

@lcwangchao

Description

@lcwangchao

Enhancement

For now, the ddl operations for placement rule is not atomic. For the current implement, if it fails to send request to PD, meta operations will rollback:

tidb/ddl/partition.go

Lines 1131 to 1135 in 680de92

err = infosync.PutRuleBundles(context.TODO(), bundles)
if err != nil {
job.State = model.JobStateCancelled
return ver, errors.Wrapf(err, "failed to notify PD the placement rules")
}

It works well in most times. However, it is possible that PD have received and processed the request successfully but the response packet is lost in the network. In this case, the meta operations will rollback and left an inconsistent state with PD.

Some ways to resolve the problem:

  • Rollback PD states when the ddl job fails even if we got a failed response. In this way, rollback for PD should be idempotent and should retry infinitely if it still fails.
  • Left it to users. In this way, we should allow user to retry his operation to make the state consistent manually. It's better to provide some commands to check the consistency too.

Metadata

Metadata

Assignees

Labels

type/enhancementThe issue or PR belongs to an enhancement.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions