Skip to content

compact_log_backup: filter out meta files by migration#18123

Merged
ti-chi-bot[bot] merged 4 commits intotikv:masterfrom
YuJuncen:filter-out-by-migration
Jan 27, 2025
Merged

compact_log_backup: filter out meta files by migration#18123
ti-chi-bot[bot] merged 4 commits intotikv:masterfrom
YuJuncen:filter-out-by-migration

Conversation

@YuJuncen
Copy link
Contributor

@YuJuncen YuJuncen commented Jan 14, 2025 •
edited
Loading

What is changed and how it works?

Issue Number: Close #18122

What's Changed:

Now, `StreamMetaStorage` is able to filter out files by meta edits.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Release note

None

@ti-chi-bot ti-chi-bot bot added dco-signoff: yes Indicates the PR's author has signed the dco. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jan 14, 2025
@YuJuncen
filter out meta files by migration
59143e0 
Signed-off-by: hillium <yu745514916@live.com>
@YuJuncen
added more logs
af447e7 
Signed-off-by: hillium <yu745514916@live.com>
@YuJuncen YuJuncen force-pushed the filter-out-by-migration branch from b09c013 to af447e7 Compare January 15, 2025 03:26
@YuJuncen
make clippy happy
765075f 
Signed-off-by: hillium <yu745514916@live.com>
@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jan 20, 2025
@YuJuncen
Copy link
Contributor Author

YuJuncen commented Jan 22, 2025

/test pull-unit-test

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 26, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 26, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-01-20 07:09:10.651987634 +0000 UTC m=+78277.982907037: ☑️ agreed by Leavrth.
  • 2025-01-26 07:06:43.777051306 +0000 UTC m=+596531.107970709: ☑️ agreed by 3pointer.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 26, 2025

@YuJuncen: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@YuJuncen
Copy link
Contributor Author

YuJuncen commented Jan 26, 2025

/test pull-unit-test

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 27, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3pointer, iosmanthus, Leavrth

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the approved label Jan 27, 2025
@ti-chi-bot ti-chi-bot bot merged commit a54bb5a into tikv:master Jan 27, 2025
8 checks passed
@ti-chi-bot ti-chi-bot bot added this to the Pool milestone Jan 27, 2025
YuJuncen added a commit to YuJuncen/tikv that referenced this pull request Dec 4, 2025
@YuJuncen @ti-chi-bot
compact_log_backup: filter out meta files by migration (tikv#18123)
6d95c2c 
close tikv#18122

Now, `StreamMetaStorage` is able to filter out files by meta edits.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
YuJuncen added a commit that referenced this pull request Dec 4, 2025
@YuJuncen @RidRisR
@ti-chi-bot @3pointer @Leavrth
tikv-ctl: pick compact-log-bakcup (#19164)
c8d972b 
* br: batch download and merge download sst before ingest (#19062)

close #19086

Add a new PRC method called batch-download to download batch SST.

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* make format

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* implement the base library for compacting logs (#17632)

close #17631

Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* added `compact-log-bakcup` to `tikv-ctl` (#17845)

close #17844

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (#18085)

close #18084

`min_input_ts` and `max_input_ts` will present in a log files compaction.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix typo (#18090)

ref #15990

Fixed a typo: `Migartion` -> `Migration`.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: filter out meta files by migration (#18123)

close #18122

Now, `StreamMetaStorage` is able to filter out files by meta edits.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: added minimal compactions size (#18235)

close #18234

Added `--minimal-compact-size` to `compact-log-backup`.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* log backup: fix several issues during compact log backup.  (#18298)

close #18308

log backup compact: fix several issues during compact a log backup

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: correct version assignment in subcompaction metadata (#18389)

close #18390

Fixed a bug that caused the time range of compaction generated SSTs are too huge.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (#18837)

close #18843

compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: use max ts among all storage checkpoint ts (#18848)

close #18847

Now, `consistency` hook checks the storage checkpoint by the max value among them.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix compact meta edit filter (#18842)

close #18843

Merge the same meta edit from different migrations instead of replacing.

Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: offload reading meta to diff cpus (#18885)

close #18884

This PR spawns read s3 file tasks to remote threads.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: read meta from checkpoint (#19068)

close #19069

This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

---------

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>
YuJuncen added a commit to YuJuncen/tikv that referenced this pull request Dec 5, 2025
@YuJuncen @RidRisR
@ti-chi-bot @3pointer @Leavrth
tikv-ctl: pick compact-log-bakcup (tikv#19164)
53fca27 
* br: batch download and merge download sst before ingest (tikv#19062)

close tikv#19086

Add a new PRC method called batch-download to download batch SST.

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* make format

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* implement the base library for compacting logs (tikv#17632)

close tikv#17631

Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* added `compact-log-bakcup` to `tikv-ctl` (tikv#17845)

close tikv#17844

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (tikv#18085)

close tikv#18084

`min_input_ts` and `max_input_ts` will present in a log files compaction.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix typo (tikv#18090)

ref tikv#15990

Fixed a typo: `Migartion` -> `Migration`.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: filter out meta files by migration (tikv#18123)

close tikv#18122

Now, `StreamMetaStorage` is able to filter out files by meta edits.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: added minimal compactions size (tikv#18235)

close tikv#18234

Added `--minimal-compact-size` to `compact-log-backup`.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* log backup: fix several issues during compact log backup.  (tikv#18298)

close tikv#18308

log backup compact: fix several issues during compact a log backup

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: correct version assignment in subcompaction metadata (tikv#18389)

close tikv#18390

Fixed a bug that caused the time range of compaction generated SSTs are too huge.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (tikv#18837)

close tikv#18843

compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: use max ts among all storage checkpoint ts (tikv#18848)

close tikv#18847

Now, `consistency` hook checks the storage checkpoint by the max value among them.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix compact meta edit filter (tikv#18842)

close tikv#18843

Merge the same meta edit from different migrations instead of replacing.

Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: offload reading meta to diff cpus (tikv#18885)

close tikv#18884

This PR spawns read s3 file tasks to remote threads.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: read meta from checkpoint (tikv#19068)

close tikv#19069

This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

---------

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>
@YuJuncen YuJuncen mentioned this pull request Dec 5, 2025
Merged
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@iosmanthus iosmanthus iosmanthus approved these changes

+2 more reviewers

@3pointer 3pointer 3pointer approved these changes

@Leavrth Leavrth Leavrth approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. lgtm release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

Pool

Development

Successfully merging this pull request may close these issues.

compact_log_backup: when reading metadata, should filter out deleted metadata by migration

4 participants

@YuJuncen @3pointer @iosmanthus @Leavrth