br: batch download and merge download sst before ingest by RidRisR · Pull Request #19062 · tikv/tikv

RidRisR · 2025-10-22T03:44:41Z

What is changed and how it works?

Issue Number: Close #19086

What's Changed:

Add a new PRC method called batch-download to download batch SST.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Release note

Add a new PRC method called batch-download to download batch SST.

ti-chi-bot · 2025-10-22T03:44:51Z

Hi @RidRisR. Thanks for your PR.

I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Leavrth · 2025-11-04T11:40:22Z

/ok-to-test

RidRisR · 2025-11-05T02:53:03Z

/retest

Leavrth · 2025-11-05T02:56:46Z

/retest

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>

ti-chi-bot · 2025-11-10T09:32:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Leavrth, YuJuncen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [YuJuncen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-11-10T09:32:09Z

[LGTM Timeline notifier]

Timeline:

2025-11-05 09:07:15.751691845 +0000 UTC m=+261085.194721724: ☑️ agreed by Leavrth.
2025-11-10 09:32:08.363562649 +0000 UTC m=+694577.806592518: ☑️ agreed by YuJuncen.

close tikv#19086 Add a new PRC method called batch-download to download batch SST. Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>

close tikv#19086 Add a new PRC method called batch-download to download batch SST. Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* br: batch download and merge download sst before ingest (#19062) close #19086 Add a new PRC method called batch-download to download batch SST. Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * make format Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* br: batch download and merge download sst before ingest (#19062) close #19086 Add a new PRC method called batch-download to download batch SST. Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * make format Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * implement the base library for compacting logs (#17632) close #17631 Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * added `compact-log-bakcup` to `tikv-ctl` (#17845) close #17844 Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (#18085) close #18084 `min_input_ts` and `max_input_ts` will present in a log files compaction. Signed-off-by: hillium <yu745514916@live.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: fix typo (#18090) ref #15990 Fixed a typo: `Migartion` -> `Migration`. Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: filter out meta files by migration (#18123) close #18122 Now, `StreamMetaStorage` is able to filter out files by meta edits. Signed-off-by: hillium <yu745514916@live.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: added minimal compactions size (#18235) close #18234 Added `--minimal-compact-size` to `compact-log-backup`. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * log backup: fix several issues during compact log backup. (#18298) close #18308 log backup compact: fix several issues during compact a log backup Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: correct version assignment in subcompaction metadata (#18389) close #18390 Fixed a bug that caused the time range of compaction generated SSTs are too huge. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (#18837) close #18843 compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: use max ts among all storage checkpoint ts (#18848) close #18847 Now, `consistency` hook checks the storage checkpoint by the max value among them. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: fix compact meta edit filter (#18842) close #18843 Merge the same meta edit from different migrations instead of replacing. Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: offload reading meta to diff cpus (#18885) close #18884 This PR spawns read s3 file tasks to remote threads. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: read meta from checkpoint (#19068) close #19069 This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint. Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> --------- Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>

* br: batch download and merge download sst before ingest (tikv#19062) close tikv#19086 Add a new PRC method called batch-download to download batch SST. Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * make format Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * implement the base library for compacting logs (tikv#17632) close tikv#17631 Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * added `compact-log-bakcup` to `tikv-ctl` (tikv#17845) close tikv#17844 Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (tikv#18085) close tikv#18084 `min_input_ts` and `max_input_ts` will present in a log files compaction. Signed-off-by: hillium <yu745514916@live.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: fix typo (tikv#18090) ref tikv#15990 Fixed a typo: `Migartion` -> `Migration`. Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: filter out meta files by migration (tikv#18123) close tikv#18122 Now, `StreamMetaStorage` is able to filter out files by meta edits. Signed-off-by: hillium <yu745514916@live.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: added minimal compactions size (tikv#18235) close tikv#18234 Added `--minimal-compact-size` to `compact-log-backup`. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * log backup: fix several issues during compact log backup. (tikv#18298) close tikv#18308 log backup compact: fix several issues during compact a log backup Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: correct version assignment in subcompaction metadata (tikv#18389) close tikv#18390 Fixed a bug that caused the time range of compaction generated SSTs are too huge. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (tikv#18837) close tikv#18843 compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: use max ts among all storage checkpoint ts (tikv#18848) close tikv#18847 Now, `consistency` hook checks the storage checkpoint by the max value among them. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: fix compact meta edit filter (tikv#18842) close tikv#18843 Merge the same meta edit from different migrations instead of replacing. Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: offload reading meta to diff cpus (tikv#18885) close tikv#18884 This PR spawns read s3 file tasks to remote threads. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: read meta from checkpoint (tikv#19068) close tikv#19069 This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint. Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> --------- Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>

* br: batch download and merge download sst before ingest (tikv#19062) close tikv#19086 Add a new PRC method called batch-download to download batch SST. Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * make format Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. dco-signoff: no Indicates the PR's author has not signed dco. labels Oct 22, 2025

RidRisR force-pushed the mergeSST branch from b5c4066 to 6440c4d Compare October 22, 2025 03:45

ti-chi-bot bot added dco-signoff: yes Indicates the PR's author has signed the dco. and removed dco-signoff: no Indicates the PR's author has not signed dco. labels Oct 22, 2025

RidRisR force-pushed the mergeSST branch from 6440c4d to 256b575 Compare October 22, 2025 09:44

ti-chi-bot bot added dco-signoff: no Indicates the PR's author has not signed dco. and removed dco-signoff: yes Indicates the PR's author has signed the dco. labels Nov 4, 2025

RidRisR force-pushed the mergeSST branch from edd5b3a to 72ec7e1 Compare November 4, 2025 11:18

RidRisR changed the title ~~[DNM] merge download sst~~ [DNM] batch download and merge download sst before ingest Nov 4, 2025

ti-chi-bot bot added ok-to-test Indicates a PR is ready to be tested. and removed needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Nov 4, 2025

RidRisR force-pushed the mergeSST branch from 72ec7e1 to 7aba320 Compare November 5, 2025 03:07

ti-chi-bot bot added dco-signoff: no Indicates the PR's author has not signed dco. and removed dco-signoff: yes Indicates the PR's author has signed the dco. labels Nov 5, 2025

RidRisR force-pushed the mergeSST branch from d3878a0 to 5397ab1 Compare November 5, 2025 03:24

ti-chi-bot bot removed the dco-signoff: no Indicates the PR's author has not signed dco. label Nov 5, 2025

RidRisR added 8 commits November 5, 2025 13:43

add download lock

96579dc

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>

add test helper

7ec0045

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>

add tests

91fd7dd

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>

make format

0cdd18b

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>

update cargo

63e91c0

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>

make linter happy

0685b88

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>

make lint happy

9039d8b

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>

make clippy happy

ebe3504

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>

RidRisR force-pushed the mergeSST branch from c6c06df to ebe3504 Compare November 5, 2025 05:43

ti-chi-bot bot added dco-signoff: yes Indicates the PR's author has signed the dco. and removed dco-signoff: no Indicates the PR's author has not signed dco. labels Nov 5, 2025

Leavrth approved these changes Nov 5, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Nov 5, 2025

RidRisR changed the title ~~[DNM] batch download and merge download sst before ingest~~ br: batch download and merge download sst before ingest Nov 6, 2025

YuJuncen approved these changes Nov 10, 2025

View reviewed changes

ti-chi-bot bot added the lgtm label Nov 10, 2025

ti-chi-bot bot added approved and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 10, 2025

ti-chi-bot bot merged commit 811b5ab into tikv:master Nov 10, 2025
9 checks passed

ti-chi-bot bot added this to the Pool milestone Nov 10, 2025

RidRisR deleted the mergeSST branch November 11, 2025 07:12

YuJuncen mentioned this pull request Dec 5, 2025

tikv-ctl: pick compact-log-backup related stuff to release-8.5 #19169

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

br: batch download and merge download sst before ingest#19062

br: batch download and merge download sst before ingest#19062
ti-chi-bot[bot] merged 14 commits intotikv:masterfrom
RidRisR:mergeSST

RidRisR commented Oct 22, 2025 •

edited

Loading

Uh oh!

ti-chi-bot bot commented Oct 22, 2025

Uh oh!

Leavrth commented Nov 4, 2025

Uh oh!

RidRisR commented Nov 5, 2025

Uh oh!

Leavrth commented Nov 5, 2025

Uh oh!

ti-chi-bot bot commented Nov 10, 2025

Uh oh!

ti-chi-bot bot commented Nov 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

RidRisR commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is changed and how it works?

Related changes

Check List

Release note

Uh oh!

ti-chi-bot bot commented Oct 22, 2025

Uh oh!

Leavrth commented Nov 4, 2025

Uh oh!

RidRisR commented Nov 5, 2025

Uh oh!

Leavrth commented Nov 5, 2025

Uh oh!

ti-chi-bot bot commented Nov 10, 2025

Uh oh!

ti-chi-bot bot commented Nov 10, 2025

[LGTM Timeline notifier]

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

RidRisR commented Oct 22, 2025 •

edited

Loading