Development Task

The current pitr is pretty slow due to there are too many small files, also putting files via raft protocol make us suffer from write amplification.

If we can merge files as SST, it will be possible to get rid of the problems above.

Tracking of tasks

Can be merge without kvproto change:

• Add interfaces needed by compact-log-backup. #17284

This modification includes all modifications applies to the external storage. Including:

• Add the iter_prefix method to external storage.
• Add the lock method to external storage. This is used to control concurrent accessing the external storage.
• Allow finished in memory SstWriters can be reopened.

Waiting kvproto changes:

• Add CRC64 for logs generated by log backup. #17406
• compact_log_backup: implement the basic library for compacting #17631

This will provide a library that can be used to compact logs.
The library exposes a structure named Execution for caller. The Execution contains the core things of a compaction:

• Collect files from backup data.
• Generate compaction works.
• Execute the compaction works.

The Execution can be extended by adding Hooks to it.
The hooks includes:

• Saving the metadata of compaction.
• Locking the Storage.
• Print the logs.
• etc..

• Add compacting log backup to tikv-ctl. #17844

This will provide an entrance for the function.