implement the base library for compacting logs by YuJuncen · Pull Request #17632 · tikv/tikv

YuJuncen · 2024-10-11T08:08:24Z

What is changed and how it works?

Issue Number: Close #17631

What's Changed:

Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs.

The directory hierarchy:

./components/compact-log-backup/
├── Cargo.toml
└── src
    ├── compaction ← Things about compacting logs, including metadata and the logic of executing an compaction.
    │   ├── collector.rs ← Give some files, what compaction we will make?
    │   ├── exec.rs ← How we read and sort log entries and make them SSTs?
    │   ├── meta.rs ← How to describe a compaction?
    │   └── mod.rs
    ├── errors.rs ← Error definitions.
    ├── exec_hooks ← Common hooks.
    │   ├── checkpoint.rs ← Skip compactions that already done.
    │   ├── consistency.rs ← Lock the storage to keep it consistent.
    │   ├── mod.rs
    │   ├── observability.rs ← Expose metrics and print logs.
    │   └── save_meta.rs ← Save the compaction metadata to the external storage.
    ├── execute
    │   ├── hooking.rs ← What hooks we have?
    │   ├── mod.rs ← The controller of a compaction over time.
    │   └── test.rs
    ├── lib.rs
    ├── source.rs ← Data source, for now only logs are supported.
    ├── statistic.rs 
    ├── storage.rs ← Things helps to read the backup storage.
    ├── test_util.rs
    └── util.rs

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Release note

None

Signed-off-by: hillium <yujuncen@pingcap.com>

Leavrth · 2024-10-18T08:02:41Z

components/compact-log-backup/src/execute/test.rs

+};
+
+#[derive(Clone)]
+struct CompactionSpy(Sender<SubcompactionResult>);


should we move the CompactionSpy into mod test because it is only used in the test?

I guess the file itself is in the mod test. Do you meaning the test_utli mod?

components/compact-log-backup/src/storage.rs

Leavrth · 2024-10-21T02:57:16Z

components/compact-log-backup/src/compaction/collector.rs

+        self.items.drain().map(|(key, c)| {
+            // Hacking: update the statistic when we really yield the compaction.
+            // (At `poll_next`.)
+            c.form(&key, &self.cfg)


Need to update the self.stat here, too.

The stat will be updated in the poll_next, so the stat will only be updated when the user read the compaction from the stream.

Leavrth · 2024-10-21T03:13:59Z

components/compact-log-backup/src/exec_hooks/checkpoint.rs

+
+    fn before_a_subcompaction_start(&mut self, _cid: CId, cx: SubcompactionStartCtx<'_>) {
+        let hash = cx.subc.crc64();
+        if self.loaded.contains(&hash) {


What if two sub-compactions have the same crc64?

Then the second one cannot be executed. Thankfully its input will probably be saved as the final subcompaction won't be written.

Leavrth · 2024-10-21T04:30:52Z

components/compact-log-backup/src/compaction/collector.rs

+                let key = *o.key();
+                let u = o.get_mut();
+                u.add_file(file);
+                if u.size > self.cfg.subcompaction_size_threshold {


Will there be a lot of self.items to make it OOM if each entry of self.items are small.

Perhaps. We may add something like memory quota in the future. But for now in fact we cannot do better.

components/compact-log-backup/src/exec_hooks/observability.rs

Signed-off-by: hillium <yujuncen@pingcap.com>

3pointer

rest LGTM

components/compact-log-backup/src/compaction/collector.rs

3pointer · 2024-11-18T03:03:58Z

components/compact-log-backup/src/util.rs

+
+    /// Finishing one tiny task. This will yield the current carrier thread
+    /// when needed.
+    pub fn step(&mut self) -> Step {


Can we use a wrapper of tokio::task::yield_now().await; here?

components/compact-log-backup/src/compaction/exec.rs

ti-chi-bot · 2024-11-18T03:15:38Z

[LGTM Timeline notifier]

Timeline:

2024-10-21 05:43:37.192398691 +0000 UTC m=+243417.889189295: ☑️ agreed by Leavrth.
2024-11-18 03:15:37.603555487 +0000 UTC m=+844499.794424483: ☑️ agreed by 3pointer.

YuJuncen · 2024-11-18T03:19:22Z

/hold

Signed-off-by: hillium <yujuncen@pingcap.com>

YuJuncen · 2024-11-18T07:23:45Z

/unhold

ti-chi-bot · 2024-11-18T07:24:02Z

@YuJuncen: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2024-11-18T07:36:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3pointer, iosmanthus, Leavrth

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [iosmanthus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

close tikv#17631 Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com>

* br: batch download and merge download sst before ingest (#19062) close #19086 Add a new PRC method called batch-download to download batch SST. Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * make format Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * implement the base library for compacting logs (#17632) close #17631 Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * added `compact-log-bakcup` to `tikv-ctl` (#17845) close #17844 Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (#18085) close #18084 `min_input_ts` and `max_input_ts` will present in a log files compaction. Signed-off-by: hillium <yu745514916@live.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: fix typo (#18090) ref #15990 Fixed a typo: `Migartion` -> `Migration`. Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: filter out meta files by migration (#18123) close #18122 Now, `StreamMetaStorage` is able to filter out files by meta edits. Signed-off-by: hillium <yu745514916@live.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: added minimal compactions size (#18235) close #18234 Added `--minimal-compact-size` to `compact-log-backup`. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * log backup: fix several issues during compact log backup. (#18298) close #18308 log backup compact: fix several issues during compact a log backup Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: correct version assignment in subcompaction metadata (#18389) close #18390 Fixed a bug that caused the time range of compaction generated SSTs are too huge. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (#18837) close #18843 compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: use max ts among all storage checkpoint ts (#18848) close #18847 Now, `consistency` hook checks the storage checkpoint by the max value among them. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: fix compact meta edit filter (#18842) close #18843 Merge the same meta edit from different migrations instead of replacing. Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: offload reading meta to diff cpus (#18885) close #18884 This PR spawns read s3 file tasks to remote threads. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: read meta from checkpoint (#19068) close #19069 This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint. Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> --------- Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>

* br: batch download and merge download sst before ingest (tikv#19062) close tikv#19086 Add a new PRC method called batch-download to download batch SST. Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * make format Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * implement the base library for compacting logs (tikv#17632) close tikv#17631 Added a new crate named `compact-log-backup`. Now it can merge some log files generated by log backup and make them become SSTs. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * added `compact-log-bakcup` to `tikv-ctl` (tikv#17845) close tikv#17844 Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: record `min_input_ts` and `max_input_ts` in Compaction (tikv#18085) close tikv#18084 `min_input_ts` and `max_input_ts` will present in a log files compaction. Signed-off-by: hillium <yu745514916@live.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: fix typo (tikv#18090) ref tikv#15990 Fixed a typo: `Migartion` -> `Migration`. Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: filter out meta files by migration (tikv#18123) close tikv#18122 Now, `StreamMetaStorage` is able to filter out files by meta edits. Signed-off-by: hillium <yu745514916@live.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: added minimal compactions size (tikv#18235) close tikv#18234 Added `--minimal-compact-size` to `compact-log-backup`. Signed-off-by: hillium <yujuncen@pingcap.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * log backup: fix several issues during compact log backup. (tikv#18298) close tikv#18308 log backup compact: fix several issues during compact a log backup Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: correct version assignment in subcompaction metadata (tikv#18389) close tikv#18390 Fixed a bug that caused the time range of compaction generated SSTs are too huge. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering (tikv#18837) close tikv#18843 compact_log_backup: add new field to track fully compacted data KV files and fix metafile filtering Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: use max ts among all storage checkpoint ts (tikv#18848) close tikv#18847 Now, `consistency` hook checks the storage checkpoint by the max value among them. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: fix compact meta edit filter (tikv#18842) close tikv#18843 Merge the same meta edit from different migrations instead of replacing. Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: offload reading meta to diff cpus (tikv#18885) close tikv#18884 This PR spawns read s3 file tasks to remote threads. Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * compact_log_backup: read meta from checkpoint (tikv#19068) close tikv#19069 This PR makes `compact-log-backup` fills the migration with subcompactions skipped by checkpoint. Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> * fix build Signed-off-by: Juncen Yu <yujuncen@pingcap.com> --------- Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com> Signed-off-by: Juncen Yu <yujuncen@pingcap.com> Signed-off-by: hillium <yu745514916@live.com> Signed-off-by: 3pointer <luancheng@pingcap.com> Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: ris <79858083+RidRisR@users.noreply.github.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Jianjun Liao <36503113+Leavrth@users.noreply.github.com>

implement the base library for compacting logs

10feb74

Signed-off-by: hillium <yujuncen@pingcap.com>

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: yes Indicates the PR's author has signed the dco. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 11, 2024

YuJuncen added 3 commits October 11, 2024 16:37

make clippy happy

4205e20

Signed-off-by: hillium <yujuncen@pingcap.com>

make format

effc34e

Signed-off-by: hillium <yujuncen@pingcap.com>

added some comments, rename some methods

986ba6b

Signed-off-by: hillium <yujuncen@pingcap.com>

Leavrth approved these changes Oct 21, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Oct 21, 2024

address comments

0a134fd

Signed-off-by: hillium <yujuncen@pingcap.com>

3pointer approved these changes Nov 18, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 18, 2024

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 18, 2024

address comments

a4494fe

Signed-off-by: hillium <yujuncen@pingcap.com>

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 18, 2024

Merge branch 'master' into compact-log-backup-core

0b73d58

iosmanthus approved these changes Nov 18, 2024

View reviewed changes

ti-chi-bot bot added the approved label Nov 18, 2024

ti-chi-bot bot merged commit 9eb3eda into tikv:master Nov 18, 2024

ti-chi-bot bot added this to the Pool milestone Nov 18, 2024

YuJuncen mentioned this pull request Dec 5, 2025

tikv-ctl: pick compact-log-backup related stuff to release-8.5 #19169

Merged

9 tasks

Conversation

YuJuncen commented Oct 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is changed and how it works?

Check List

Release note

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

3pointer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ti-chi-bot bot commented Nov 18, 2024

[LGTM Timeline notifier]

Uh oh!

YuJuncen commented Nov 18, 2024

Uh oh!

YuJuncen commented Nov 18, 2024

Uh oh!

ti-chi-bot bot commented Nov 18, 2024

Uh oh!

ti-chi-bot bot commented Nov 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

YuJuncen commented Oct 11, 2024 •

edited

Loading