-
Notifications
You must be signed in to change notification settings - Fork 6.1k
Closed
Labels
type/enhancementThe issue or PR belongs to an enhancement.The issue or PR belongs to an enhancement.
Description
Enhancement
For now, the ddl operations for placement rule is not atomic. For the current implement, if it fails to send request to PD, meta operations will rollback:
Lines 1131 to 1135 in 680de92
| err = infosync.PutRuleBundles(context.TODO(), bundles) | |
| if err != nil { | |
| job.State = model.JobStateCancelled | |
| return ver, errors.Wrapf(err, "failed to notify PD the placement rules") | |
| } |
It works well in most times. However, it is possible that PD have received and processed the request successfully but the response packet is lost in the network. In this case, the meta operations will rollback and left an inconsistent state with PD.
Some ways to resolve the problem:
- Rollback PD states when the ddl job fails even if we got a failed response. In this way, rollback for PD should be idempotent and should retry infinitely if it still fails.
- Left it to users. In this way, we should allow user to retry his operation to make the state consistent manually. It's better to provide some commands to check the consistency too.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
type/enhancementThe issue or PR belongs to an enhancement.The issue or PR belongs to an enhancement.