This is an automated cherry-pick of #44024

What problem does this PR solve?

Issue Number: close #43997

Problem Summary:

The basic idea of checkpoint is to recover the progress:

| add index for partition 1
|  [ local checkpoint ]
| add index for partition 2
|  [ local checkpoint ]
|  [ global checkpoint ]
| ...
| add index for partition k
|  [ local checkpoint ]
| add index for partition k+1
v  (TiDB 1) crash
  | (TiDB 2) get DDL owner
  | add index for partition 2
  | ...

Note that we can only begin with partition 2 because the local checkpoint is lost when TiDB 1 crashes.

In order to represent which partition we should begin with, reorg meta is used. The reorg meta contains a tuple: (partition ID or physical table ID, start key, end key). Every time TiDB restarts in the middle state of adding an index, it tries to reset the reorg meta to the state exactly before the last global checkpoint.

Previously, we store the reorg meta in the checkpoint manager. However, we did not distinguish the "local" reorg meta and the "global" reorg meta. When a partition is complete, the reorg meta is updated immediately, leading to a new TiDB reset to the wrong partition. Finally, the index data from some of the partitions is lost.

What is changed and how it works?

• Distinguish the "local" reorg meta from the "global" one.
• When the mysql.ddl_reorg_meta is initialized, we also initialize the checkpoint.
• Move the creation of checkpoint manager to a proper place(which needs the info from mysql.ddl_reorg_meta).
• After the checkpoint manager is created, we try recover the global checkpoint and overwrite the reorg info.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

  1. Create two TiDB.
  2. Prepare a table with a lot of partitions.
  3. Add index.
  4. Kill the TiDB owner. 
  5. Check if the other TiDB can reset the reorg meta to a correct partition ID. 
  

• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None