Skip to content

compact_log_backup: record min_input_ts and max_input_ts in Compaction#18085

Merged
ti-chi-bot[bot] merged 4 commits intotikv:masterfrom
YuJuncen:added-involved-ts-for-compaction
Jan 6, 2025
Merged

compact_log_backup: record min_input_ts and max_input_ts in Compaction#18085
ti-chi-bot[bot] merged 4 commits intotikv:masterfrom
YuJuncen:added-involved-ts-for-compaction

Conversation

@YuJuncen
Copy link
Contributor

@YuJuncen YuJuncen commented Jan 3, 2025

What is changed and how it works?

Issue Number: Close #18084

What's Changed:

`min_input_ts` and `max_input_ts` will present in a log files compaction.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Release note

None

@YuJuncen
record min_input_ts and max_input_ts in Compaction
ebfc4fa 
Signed-off-by: hillium <yu745514916@live.com>
@ti-chi-bot ti-chi-bot bot added dco-signoff: yes Indicates the PR's author has signed the dco. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 3, 2025
@YuJuncen
make clippy happy
10c3953 
Signed-off-by: hillium <yu745514916@live.com>
@YuJuncen YuJuncen force-pushed the added-involved-ts-for-compaction branch from 2a88f55 to 10c3953 Compare January 3, 2025 05:31
Tristan1900
Tristan1900 approved these changes Jan 3, 2025
View reviewed changes
Copy link
Contributor

@Tristan1900 Tristan1900 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't have permission to approve but lgtm

cmd/tikv-ctl/src/cmd.rs
"until when we need to compact. \
Also note that records out of the [--from, --until) range may also be compacted \
if their neighbour in the same file needs to be compacted."
"until when we need to include files into the compaction.\
Copy link
Contributor

@Tristan1900 Tristan1900 Jan 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a nit comment, can ignore.
feels like --from and --to is more common than --until.

Copy link
Contributor Author

@YuJuncen YuJuncen Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You know changing a released flag name isn't trivial... So I prefer keeping their old name...

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 3, 2025

@Tristan1900: adding LGTM is restricted to approvers and reviewers in OWNERS files.

Details

In response to this:

doesn't have permission to approve but lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jan 6, 2025
@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 6, 2025
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 6, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-01-06 02:57:28.654608141 +0000 UTC m=+149591.943439845: ☑️ agreed by 3pointer.
  • 2025-01-06 03:48:02.734371671 +0000 UTC m=+152626.023203376: ☑️ agreed by Leavrth.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 6, 2025

@YuJuncen: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@YuJuncen
address comments
3f5d1de 
Signed-off-by: hillium <yu745514916@live.com>
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 6, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3pointer, iosmanthus, Leavrth, Tristan1900

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the approved label Jan 6, 2025
@ti-chi-bot ti-chi-bot bot merged commit 6bb07ba into tikv:master Jan 6, 2025
1 check passed
@ti-chi-bot ti-chi-bot bot added this to the Pool milestone Jan 6, 2025
YuJuncen added a commit to YuJuncen/tikv that referenced this pull request Dec 4, 2025
@YuJuncen @ti-chi-bot
compact_log_backup: record min_input_ts and max_input_ts in Compa…
1ba133f 
…ction (tikv#18085)

close tikv#18084

`min_input_ts` and `max_input_ts` will present in a log files compaction.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
YuJuncen added a commit that referenced this pull request Dec 4, 2025
@YuJuncen @RidRisR
@ti-chi-bot @3pointer @Leavrth
tikv-ctl: pick compact-log-bakcup (#19164)
c8d972b 
* br: batch download and merge download sst before ingest (#19062)

close #19086

Add a new PRC method called batch-download to download batch SST.

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* make format

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* implement the base library for compacting logs (#17632)

close #17631

Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* added `compact-log-bakcup` to `tikv-ctl` (#17845)

close #17844

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (#18085)

close #18084

`min_input_ts` and `max_input_ts` will present in a log files compaction.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix typo (#18090)

ref #15990

Fixed a typo: `Migartion` -> `Migration`.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: filter out meta files by migration (#18123)

close #18122

Now, `StreamMetaStorage` is able to filter out files by meta edits.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: added minimal compactions size (#18235)

close #18234

Added `--minimal-compact-size` to `compact-log-backup`.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* log backup: fix several issues during compact log backup.  (#18298)

close #18308

log backup compact: fix several issues during compact a log backup

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: correct version assignment in subcompaction metadata (#18389)

close #18390

Fixed a bug that caused the time range of compaction generated SSTs are too huge.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (#18837)

close #18843

compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: use max ts among all storage checkpoint ts (#18848)

close #18847

Now, `consistency` hook checks the storage checkpoint by the max value among them.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix compact meta edit filter (#18842)

close #18843

Merge the same meta edit from different migrations instead of replacing.

Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: offload reading meta to diff cpus (#18885)

close #18884

This PR spawns read s3 file tasks to remote threads.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: read meta from checkpoint (#19068)

close #19069

This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

---------

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>
YuJuncen added a commit to YuJuncen/tikv that referenced this pull request Dec 5, 2025
@YuJuncen @RidRisR
@ti-chi-bot @3pointer @Leavrth
tikv-ctl: pick compact-log-bakcup (tikv#19164)
53fca27 
* br: batch download and merge download sst before ingest (tikv#19062)

close tikv#19086

Add a new PRC method called batch-download to download batch SST.

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* make format

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* implement the base library for compacting logs (tikv#17632)

close tikv#17631

Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* added `compact-log-bakcup` to `tikv-ctl` (tikv#17845)

close tikv#17844

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (tikv#18085)

close tikv#18084

`min_input_ts` and `max_input_ts` will present in a log files compaction.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix typo (tikv#18090)

ref tikv#15990

Fixed a typo: `Migartion` -> `Migration`.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: filter out meta files by migration (tikv#18123)

close tikv#18122

Now, `StreamMetaStorage` is able to filter out files by meta edits.

Signed-off-by: hillium <yu745514916@live.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: added minimal compactions size (tikv#18235)

close tikv#18234

Added `--minimal-compact-size` to `compact-log-backup`.

Signed-off-by: hillium <yujuncen@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* log backup: fix several issues during compact log backup.  (tikv#18298)

close tikv#18308

log backup compact: fix several issues during compact a log backup

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: correct version assignment in subcompaction metadata (tikv#18389)

close tikv#18390

Fixed a bug that caused the time range of compaction generated SSTs are too huge.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (tikv#18837)

close tikv#18843

compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering

Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: use max ts among all storage checkpoint ts (tikv#18848)

close tikv#18847

Now, `consistency` hook checks the storage checkpoint by the max value among them.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: fix compact meta edit filter (tikv#18842)

close tikv#18843

Merge the same meta edit from different migrations instead of replacing.

Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: offload reading meta to diff cpus (tikv#18885)

close tikv#18884

This PR spawns read s3 file tasks to remote threads.

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* compact_log_backup: read meta from checkpoint (tikv#19068)

close tikv#19069

This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint.

Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* fix build

Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

---------

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
Signed-off-by: hillium <yu745514916@live.com>
Signed-off-by: 3pointer <luancheng@pingcap.com>
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>
@YuJuncen YuJuncen mentioned this pull request Dec 5, 2025
Merged
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@iosmanthus iosmanthus iosmanthus approved these changes

+3 more reviewers

@3pointer 3pointer 3pointer approved these changes

@Tristan1900 Tristan1900 Tristan1900 approved these changes

@Leavrth Leavrth Leavrth approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. lgtm release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Milestone

Pool

Development

Successfully merging this pull request may close these issues.

compact_log_backup: input_max_ts and input_min_ts should be included into Compaction

5 participants

@YuJuncen @3pointer @iosmanthus @Tristan1900 @Leavrth