compact_log_backup: fix compact meta edit filter#18842
Merged
ti-chi-bot[bot] merged 5 commits intotikv:masterfrom Aug 26, 2025
Merged
compact_log_backup: fix compact meta edit filter#18842ti-chi-bot[bot] merged 5 commits intotikv:masterfrom
ti-chi-bot[bot] merged 5 commits intotikv:masterfrom
Conversation
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Contributor
Author
|
/check-issue-triage-complete |
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
YuJuncen
approved these changes
Aug 21, 2025
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
3pointer
reviewed
Aug 26, 2025
| self.destructed_self = self.destructed_self || em.destruct_self; | ||
| self.all_data_files_compacted = | ||
| self.all_data_files_compacted || em.all_data_files_compacted; | ||
| if self.destructed_self || self.all_data_files_compacted { |
Contributor
There was a problem hiding this comment.
if self.all_data_files_compacted is true while self.destructed_self is false.
should we clear the segments here?
3pointer
approved these changes
Aug 26, 2025
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: 3pointer, YuJuncen The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Contributor
Contributor
Author
|
/retest |
1 similar comment
Contributor
Author
|
/retest |
yibin87
pushed a commit
to yibin87/tikv
that referenced
this pull request
Sep 2, 2025
close tikv#18843 Merge the same meta edit from different migrations instead of replacing. Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
3AceShowHand
pushed a commit
to 3AceShowHand/tikv
that referenced
this pull request
Oct 13, 2025
close tikv#18843 Merge the same meta edit from different migrations instead of replacing. Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: 3AceShowHand <jinl1037@hotmail.com>
YuJuncen
pushed a commit
to YuJuncen/tikv
that referenced
this pull request
Dec 4, 2025
close tikv#18843 Merge the same meta edit from different migrations instead of replacing. Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com>
YuJuncen
added a commit
that referenced
this pull request
Dec 4, 2025
* br: batch download and merge download sst before ingest (#19062) close #19086 Add a new PRC method called batch-download to download batch SST. Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * make format Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * implement the base library for compacting logs (#17632) close #17631 Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * added `compact-log-bakcup` to `tikv-ctl` (#17845) close #17844 Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (#18085) close #18084 `min_input_ts` and `max_input_ts` will present in a log files compaction. Signed-off-by: hillium <yu745514916@live.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: fix typo (#18090) ref #15990 Fixed a typo: `Migartion` -> `Migration`. Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: filter out meta files by migration (#18123) close #18122 Now, `StreamMetaStorage` is able to filter out files by meta edits. Signed-off-by: hillium <yu745514916@live.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: added minimal compactions size (#18235) close #18234 Added `--minimal-compact-size` to `compact-log-backup`. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * log backup: fix several issues during compact log backup. (#18298) close #18308 log backup compact: fix several issues during compact a log backup Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: correct version assignment in subcompaction metadata (#18389) close #18390 Fixed a bug that caused the time range of compaction generated SSTs are too huge. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (#18837) close #18843 compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: use max ts among all storage checkpoint ts (#18848) close #18847 Now, `consistency` hook checks the storage checkpoint by the max value among them. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: fix compact meta edit filter (#18842) close #18843 Merge the same meta edit from different migrations instead of replacing. Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: offload reading meta to diff cpus (#18885) close #18884 This PR spawns read s3 file tasks to remote threads. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: read meta from checkpoint (#19068) close #19069 This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint. Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> --------- Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>
YuJuncen
added a commit
to YuJuncen/tikv
that referenced
this pull request
Dec 5, 2025
* br: batch download and merge download sst before ingest (tikv#19062) close tikv#19086 Add a new PRC method called batch-download to download batch SST. Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * make format Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * implement the base library for compacting logs (tikv#17632) close tikv#17631 Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * added `compact-log-bakcup` to `tikv-ctl` (tikv#17845) close tikv#17844 Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (tikv#18085) close tikv#18084 `min_input_ts` and `max_input_ts` will present in a log files compaction. Signed-off-by: hillium <yu745514916@live.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: fix typo (tikv#18090) ref tikv#15990 Fixed a typo: `Migartion` -> `Migration`. Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: filter out meta files by migration (tikv#18123) close tikv#18122 Now, `StreamMetaStorage` is able to filter out files by meta edits. Signed-off-by: hillium <yu745514916@live.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: added minimal compactions size (tikv#18235) close tikv#18234 Added `--minimal-compact-size` to `compact-log-backup`. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * log backup: fix several issues during compact log backup. (tikv#18298) close tikv#18308 log backup compact: fix several issues during compact a log backup Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: correct version assignment in subcompaction metadata (tikv#18389) close tikv#18390 Fixed a bug that caused the time range of compaction generated SSTs are too huge. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (tikv#18837) close tikv#18843 compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: use max ts among all storage checkpoint ts (tikv#18848) close tikv#18847 Now, `consistency` hook checks the storage checkpoint by the max value among them. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: fix compact meta edit filter (tikv#18842) close tikv#18843 Merge the same meta edit from different migrations instead of replacing. Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: offload reading meta to diff cpus (tikv#18885) close tikv#18884 This PR spawns read s3 file tasks to remote threads. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: read meta from checkpoint (tikv#19068) close tikv#19069 This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint. Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> --------- Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is changed and how it works?
Issue Number: Close #18843
What's Changed:
Related changes
pingcap/docs/pingcap/docs-cn:Check List
Tests
Side effects
Release note