BR support common cloud storage

Overview

Integrate BR with common cloud object storage (S3, GCS, Azure Blob storage etc).

Problem statement

Currently, BR supports local storage where backup files are stored on local directory. But the backup files need to be collected together and copied to every TiKV node. This is difficult to use in practice, so it's better to mount an NFS like filesystem to every TiKV node and BR node. However, mounting NFS to every node is difficult to set up and error-prone.

Alternatively, object storage is better for this scenario, especially that it's quite common to backup to S3/GCS on public cloud.

TODO list

S3 Support (2400 points)

• Extend current backup protobuf to support S3 compatible (S3, Ceph object storage, Minio, Aliyun OSS) storage (900 points / medium) @tennix backup: extend backup storage for S3 kvproto#507
• Store BR metadata to S3 compatible object storage (600 points / easy) @DanielZhangQD backup: store backupmeta to s3 #103
• Store TiKV SST files to S3 compatible object storage (900 points / medium) @yiwu-arbug external_storage: add S3 support tikv/tikv#6209

GCS Support (1800 points)

• Extend backup protobuf to support GCS (300 points / easy) @SunRunAway backup: add GCS proto kvproto#514
• Store BR metadata to GCS (600 points / easy) @SunRunAway storage: support GCS backup and restore #102
• Store TiKV SST files to GCS (900 points / medium) @yiwu-arbug external_storage: add GCS support tikv/tikv#6283

TiDB Operator integration (900 points)

• Integrate BR with tidb-operator to allow easy deployment (900 points / medium) @DanielZhangQD support backup to S3 with br tidb-operator#1280

Test (2100 points)

• Unit test coverage >= 60% (900 points / easy)
• e2e test (1200 points / medium) [WIP] GCS: add e2e test #108

Mentors

• @tennix
• @overvenus
• @kennytm

Recommended skills

• Go language
• Rust language
• Familiar with S3/GCS/Azure blob storage