Skip to content

compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering#18837

Merged
ti-chi-bot[bot] merged 4 commits intotikv:masterfrom
3pointer:fix_metafile_filter
Aug 18, 2025
Merged

compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering#18837
ti-chi-bot[bot] merged 4 commits intotikv:masterfrom
3pointer:fix_metafile_filter

Conversation

@3pointer
Copy link
Contributor

@3pointer 3pointer commented Aug 15, 2025 •
edited by YuJuncen
Loading

What is changed and how it works?

Issue Number: Close #18843

What's Changed:

This PR introduces a new separate field to record whether all data KV files have been compacted previously, allowing future compactions to skip already-handled metaEdit entries.

compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Release note

compact-log-backup: improve metaEdit filtering to reduce meta iteration cost

@3pointer
compact_log_backup: fine-grained skip migrations data
d517efb 
Signed-off-by: 3pointer <luancheng@pingcap.com>
@3pointer
add test case
0d092ef 
Signed-off-by: 3pointer <luancheng@pingcap.com>
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 15, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has signed the dco. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 15, 2025
@3pointer
format
90b23c9 
Signed-off-by: 3pointer <luancheng@pingcap.com>
@ti-chi-bot ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Aug 15, 2025
@3pointer 3pointer changed the title compact_log_backup: Fix metafile filter with new field compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering Aug 15, 2025
@3pointer
Update meta.rs
c061518 
Signed-off-by: 3pointer <luancheng@pingcap.com>
@3pointer 3pointer force-pushed the fix_metafile_filter branch from 6b8710f to c061518 Compare August 15, 2025 09:23
@3pointer 3pointer marked this pull request as ready for review August 15, 2025 09:23
@ti-chi-bot ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 15, 2025
@ti-chi-bot ti-chi-bot bot added the lgtm label Aug 18, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 18, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Leavrth, YuJuncen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added approved and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Aug 18, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 18, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-08-18 10:01:48.382507187 +0000 UTC m=+261316.325682693: ☑️ agreed by Leavrth.
  • 2025-08-18 10:32:11.342262687 +0000 UTC m=+263139.285438193: ☑️ agreed by YuJuncen.

@Leavrth
Copy link
Contributor

Leavrth commented Aug 18, 2025

/check-issue-triage-complete

@ti-chi-bot ti-chi-bot bot merged commit 7c70ed7 into tikv:master Aug 18, 2025
8 checks passed
@ti-chi-bot ti-chi-bot bot added this to the Pool milestone Aug 18, 2025
3AceShowHand pushed a commit to 3AceShowHand/tikv that referenced this pull request Oct 13, 2025
@3pointer @3AceShowHand
compact_log_backup: add new field to track fully compacted data KV fi…
8de2208 
…les and fix metafile filtering (tikv#18837)

close tikv#18843

compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: 3AceShowHand <jinl1037@hotmail.com>
YuJuncen pushed a commit to YuJuncen/tikv that referenced this pull request Dec 4, 2025
@3pointer @YuJuncen
compact_log_backup: add new field to track fully compacted data KV fi…
f4bdeae 
…les and fix metafile filtering (tikv#18837)

close tikv#18843

compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
YuJuncen added a commit that referenced this pull request Dec 4, 2025
@YuJuncen @RidRisR
@ti-chi-bot @3pointer @Leavrth
tikv-ctl: pick compact-log-bakcup (#19164)
c8d972b 
* br: batch download and merge download sst before ingest (#19062)

close #19086

Add a new PRC method called batch-download to download batch SST.

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* make format

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* implement the base library for compacting logs (#17632)

close #17631

Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* added `compact-log-bakcup` to `tikv-ctl` (#17845)

close #17844

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (#18085)

close #18084

`min_input_ts` and `max_input_ts` will present in a log files compaction.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix typo (#18090)

ref #15990

Fixed a typo: `Migartion` -> `Migration`.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: filter out meta files by migration (#18123)

close #18122

Now, `StreamMetaStorage` is able to filter out files by meta edits.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: added minimal compactions size (#18235)

close #18234

Added `--minimal-compact-size` to `compact-log-backup`.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* log backup: fix several issues during compact log backup.  (#18298)

close #18308

log backup compact: fix several issues during compact a log backup

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: correct version assignment in subcompaction metadata (#18389)

close #18390

Fixed a bug that caused the time range of compaction generated SSTs are too huge.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (#18837)

close #18843

compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: use max ts among all storage checkpoint ts (#18848)

close #18847

Now, `consistency` hook checks the storage checkpoint by the max value among them.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix compact meta edit filter (#18842)

close #18843

Merge the same meta edit from different migrations instead of replacing.

Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: offload reading meta to diff cpus (#18885)

close #18884

This PR spawns read s3 file tasks to remote threads.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: read meta from checkpoint (#19068)

close #19069

This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

---------

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>
YuJuncen added a commit to YuJuncen/tikv that referenced this pull request Dec 5, 2025
@YuJuncen @RidRisR
@ti-chi-bot @3pointer @Leavrth
tikv-ctl: pick compact-log-bakcup (tikv#19164)
53fca27 
* br: batch download and merge download sst before ingest (tikv#19062)

close tikv#19086

Add a new PRC method called batch-download to download batch SST.

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* make format

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* implement the base library for compacting logs (tikv#17632)

close tikv#17631

Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* added `compact-log-bakcup` to `tikv-ctl` (tikv#17845)

close tikv#17844

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (tikv#18085)

close tikv#18084

`min_input_ts` and `max_input_ts` will present in a log files compaction.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix typo (tikv#18090)

ref tikv#15990

Fixed a typo: `Migartion` -> `Migration`.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: filter out meta files by migration (tikv#18123)

close tikv#18122

Now, `StreamMetaStorage` is able to filter out files by meta edits.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: added minimal compactions size (tikv#18235)

close tikv#18234

Added `--minimal-compact-size` to `compact-log-backup`.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* log backup: fix several issues during compact log backup.  (tikv#18298)

close tikv#18308

log backup compact: fix several issues during compact a log backup

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: correct version assignment in subcompaction metadata (tikv#18389)

close tikv#18390

Fixed a bug that caused the time range of compaction generated SSTs are too huge.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (tikv#18837)

close tikv#18843

compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: use max ts among all storage checkpoint ts (tikv#18848)

close tikv#18847

Now, `consistency` hook checks the storage checkpoint by the max value among them.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix compact meta edit filter (tikv#18842)

close tikv#18843

Merge the same meta edit from different migrations instead of replacing.

Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: offload reading meta to diff cpus (tikv#18885)

close tikv#18884

This PR spawns read s3 file tasks to remote threads.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: read meta from checkpoint (tikv#19068)

close tikv#19069

This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

---------

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>
@YuJuncen YuJuncen mentioned this pull request Dec 5, 2025
Merged
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@YuJuncen YuJuncen YuJuncen approved these changes

+1 more reviewer

@Leavrth Leavrth Leavrth approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Milestone

Pool

Development

Successfully merging this pull request may close these issues.

backup-compact: compact duplicated metadata

3 participants

@3pointer @Leavrth @YuJuncen