Skip to content

compact_log_backup: correct version assignment in subcompaction metadata#18389

Merged
ti-chi-bot[bot] merged 1 commit intotikv:masterfrom
YuJuncen:fix-start-end-version
Apr 16, 2025
Merged

compact_log_backup: correct version assignment in subcompaction metadata#18389
ti-chi-bot[bot] merged 1 commit intotikv:masterfrom
YuJuncen:fix-start-end-version

Conversation

@YuJuncen
Copy link
Contributor

@YuJuncen YuJuncen commented Apr 14, 2025 •
edited
Loading

… SST verification tests

What is changed and how it works?

Issue Number: Close #18390

What's Changed:

Fixed a bug that caused the time range of compaction generated SSTs are too huge.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Release note

Fixed a bug that caused the time range of compaction generated SSTs are too huge.

@YuJuncen
fix: correct version assignment in subcompaction metadata and enhance…
d8feca9 
… SST verification tests

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has signed the dco. labels Apr 14, 2025
@YuJuncen YuJuncen changed the title fix: correct version assignment in subcompaction metadata compact_log_backup: correct version assignment in subcompaction metadata Apr 14, 2025
@ti-chi-bot ti-chi-bot bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed do-not-merge/needs-linked-issue labels Apr 14, 2025
@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Apr 14, 2025
@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Apr 15, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Apr 15, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-04-14 08:43:08.182217117 +0000 UTC m=+2677881.866453206: ☑️ agreed by 3pointer.
  • 2025-04-15 07:44:19.535590528 +0000 UTC m=+2760753.219826624: ☑️ agreed by Leavrth.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Apr 16, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3pointer, iosmanthus, Leavrth

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the approved label Apr 16, 2025
@ti-chi-bot ti-chi-bot bot merged commit 6462828 into tikv:master Apr 16, 2025
8 checks passed
@ti-chi-bot ti-chi-bot bot added this to the Pool milestone Apr 16, 2025
YuJuncen added a commit to YuJuncen/tikv that referenced this pull request Dec 4, 2025
@YuJuncen
compact_log_backup: correct version assignment in subcompaction metad…
576a2af 
…ata (tikv#18389)

close tikv#18390

Fixed a bug that caused the time range of compaction generated SSTs are too huge.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
YuJuncen added a commit that referenced this pull request Dec 4, 2025
@YuJuncen @RidRisR
@ti-chi-bot @3pointer @Leavrth
tikv-ctl: pick compact-log-bakcup (#19164)
c8d972b 
* br: batch download and merge download sst before ingest (#19062)

close #19086

Add a new PRC method called batch-download to download batch SST.

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* make format

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* implement the base library for compacting logs (#17632)

close #17631

Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* added `compact-log-bakcup` to `tikv-ctl` (#17845)

close #17844

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (#18085)

close #18084

`min_input_ts` and `max_input_ts` will present in a log files compaction.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix typo (#18090)

ref #15990

Fixed a typo: `Migartion` -> `Migration`.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: filter out meta files by migration (#18123)

close #18122

Now, `StreamMetaStorage` is able to filter out files by meta edits.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: added minimal compactions size (#18235)

close #18234

Added `--minimal-compact-size` to `compact-log-backup`.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* log backup: fix several issues during compact log backup.  (#18298)

close #18308

log backup compact: fix several issues during compact a log backup

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: correct version assignment in subcompaction metadata (#18389)

close #18390

Fixed a bug that caused the time range of compaction generated SSTs are too huge.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (#18837)

close #18843

compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: use max ts among all storage checkpoint ts (#18848)

close #18847

Now, `consistency` hook checks the storage checkpoint by the max value among them.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix compact meta edit filter (#18842)

close #18843

Merge the same meta edit from different migrations instead of replacing.

Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: offload reading meta to diff cpus (#18885)

close #18884

This PR spawns read s3 file tasks to remote threads.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: read meta from checkpoint (#19068)

close #19069

This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

---------

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>
YuJuncen added a commit to YuJuncen/tikv that referenced this pull request Dec 5, 2025
@YuJuncen @RidRisR
@ti-chi-bot @3pointer @Leavrth
tikv-ctl: pick compact-log-bakcup (tikv#19164)
53fca27 
* br: batch download and merge download sst before ingest (tikv#19062)

close tikv#19086

Add a new PRC method called batch-download to download batch SST.

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* make format

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* implement the base library for compacting logs (tikv#17632)

close tikv#17631

Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* added `compact-log-bakcup` to `tikv-ctl` (tikv#17845)

close tikv#17844

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (tikv#18085)

close tikv#18084

`min_input_ts` and `max_input_ts` will present in a log files compaction.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix typo (tikv#18090)

ref tikv#15990

Fixed a typo: `Migartion` -> `Migration`.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: filter out meta files by migration (tikv#18123)

close tikv#18122

Now, `StreamMetaStorage` is able to filter out files by meta edits.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: added minimal compactions size (tikv#18235)

close tikv#18234

Added `--minimal-compact-size` to `compact-log-backup`.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* log backup: fix several issues during compact log backup.  (tikv#18298)

close tikv#18308

log backup compact: fix several issues during compact a log backup

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: correct version assignment in subcompaction metadata (tikv#18389)

close tikv#18390

Fixed a bug that caused the time range of compaction generated SSTs are too huge.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (tikv#18837)

close tikv#18843

compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: use max ts among all storage checkpoint ts (tikv#18848)

close tikv#18847

Now, `consistency` hook checks the storage checkpoint by the max value among them.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix compact meta edit filter (tikv#18842)

close tikv#18843

Merge the same meta edit from different migrations instead of replacing.

Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: offload reading meta to diff cpus (tikv#18885)

close tikv#18884

This PR spawns read s3 file tasks to remote threads.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: read meta from checkpoint (tikv#19068)

close tikv#19069

This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

---------

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>
@YuJuncen YuJuncen mentioned this pull request Dec 5, 2025
Merged
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@iosmanthus iosmanthus iosmanthus approved these changes

+2 more reviewers

@3pointer 3pointer 3pointer approved these changes

@Leavrth Leavrth Leavrth approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Milestone

Pool

Development

Successfully merging this pull request may close these issues.

Incorrect end_version field in compact log backup

4 participants

@YuJuncen @3pointer @iosmanthus @Leavrth