engine_rocks: introduce `force_partition_range` in compact guard by glorv · Pull Request #18866 · tikv/tikv

glorv · 2025-08-21T03:54:48Z

What is changed and how it works?

Issue Number: Close #18865

What's Changed:

- introduce `force_partition_range` in compact guard to force partition specific range to ensure SST ingestion under this range can be ingested to the bottom level.
- Add 2 new API in SST service to manage the `force_partition_range`. The `add_force_partition_range` will also trigger a manual compaction to ensure existing SST will be partitioned.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Release note

None

Signed-off-by: glorv <glorvs@163.com>

…tion-range

Signed-off-by: glorv <glorvs@163.com>

Connor1996

LGTM

v01dstar · 2025-08-23T06:03:57Z

components/raftstore/src/store/compaction_guard.rs

+                        &[]
+                    };
+
+                    tikv_util::info!("sst partition due to force partition ranges";


It seems that self.current_next_level_size >= self.max_compaction_size may also bring us here. I know it is just for logging, but do we need to distinguish it?

v01dstar · 2025-08-23T06:35:34Z

tests/integrations/import/test_sst_service.rs

+
+    // ingest a sst with a big range
+    let (mut meta, data) =
+        gen_sst_file_with_tidb_kvs(sst_path.clone(), &[(b"a", b"a"), (b"z", b"z")], None);


Insteading of ingest a generated SST, how about writing to TiKV with txn APIs. Because there is a key encoding logic in add_force_partition_range

Yes, this is a tricky workaround, because we need to write a large amount of data to let rocksdb trigger flush and generate sst files. So I just let gen_sst_file_with_tidb_kvs generate the rocksdb format data kv, there should be no difference.

v01dstar

rest LGTM

ti-chi-bot · 2025-08-23T06:35:59Z

[LGTM Timeline notifier]

Timeline:

2025-08-21 22:40:30.589940523 +0000 UTC m=+566038.533116029: ☑️ agreed by Connor1996.
2025-08-23 06:35:58.40648288 +0000 UTC m=+680966.349658397: ☑️ agreed by v01dstar.

hbisheng

LGTM

hbisheng · 2025-08-25T02:25:17Z

components/raftstore/src/store/compaction_guard.rs

+                let ttl = r.ttl.saturating_duration_since(now);
+                if ttl == Duration::ZERO {


nit: use a consistent style for time comparison

Suggested change

let ttl = r.ttl.saturating_duration_since(now);

if ttl == Duration::ZERO {

let expired = r.ttl < now;

if expired {

hbisheng · 2025-08-25T02:28:57Z

components/raftstore/src/store/compaction_guard.rs

+struct TtlRange {
+    start: Vec<u8>,
+    end: Vec<u8>,
+    ttl: Instant,


nit: TTL usually means a duration rather than a time point. expires_at is probably better.

Suggested change

ttl: Instant,

expires_at: Instant,

Signed-off-by: glorv <glorvs@163.com>

glorv · 2025-08-27T07:14:41Z

/hold waiting for pingcap/kvproto#1340

ti-chi-bot · 2025-08-28T17:30:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Connor1996, v01dstar, zhangjinpeng87

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [Connor1996,v01dstar,zhangjinpeng87]
~~src/config/OWNERS~~ [zhangjinpeng87]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

glorv · 2025-08-29T01:24:26Z

/retest

close #18912 Fix the incorrect assert of compaction guard introduced by #18866. Signed-off-by: glorv <glorvs@163.com>

…v#18866) close tikv#18865 - introduce `force_partition_range` in compact guard to force partition specific range to ensure SST ingestion under this range can be ingested to the bottom level. - Add 2 new API in SST service to manage the `force_partition_range`. The `add_force_partition_range` will also trigger a manual compaction to ensure existing SST will be partitioned. Signed-off-by: glorv <glorvs@163.com>

close tikv#18912 Fix the incorrect assert of compaction guard introduced by tikv#18866. Signed-off-by: glorv <glorvs@163.com>

close tikv#18865 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2025-09-08T02:40:12Z

In response to a cherrypick label: new pull request created to branch release-8.5: #18934.
But this PR has conflicts, please resolve them!

close tikv#18865 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2025-09-08T02:40:13Z

In response to a cherrypick label: new pull request created to branch release-8.1: #18935.
But this PR has conflicts, please resolve them!

close tikv#18865 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2025-09-08T02:40:14Z

In response to a cherrypick label: new pull request created to branch release-7.5: #18936.
But this PR has conflicts, please resolve them!

close tikv#18912 Fix the incorrect assert of compaction guard introduced by tikv#18866. Signed-off-by: glorv <glorvs@163.com>

) (#18934) close #18865 - introduce `force_partition_range` in compact guard to force partition specific range to ensure SST ingestion under this range can be ingested to the bottom level. - Add 2 new API in SST service to manage the `force_partition_range`. The `add_force_partition_range` will also trigger a manual compaction to ensure existing SST will be partitioned. Signed-off-by: glorv <glorvs@163.com> Signed-off-by: lucasliang <nkcs_lykx@hotmail.com> Co-authored-by: glorv <glorvs@163.com> Co-authored-by: lucasliang <nkcs_lykx@hotmail.com>

…v#18866) close tikv#18865 - introduce `force_partition_range` in compact guard to force partition specific range to ensure SST ingestion under this range can be ingested to the bottom level. - Add 2 new API in SST service to manage the `force_partition_range`. The `add_force_partition_range` will also trigger a manual compaction to ensure existing SST will be partitioned. Signed-off-by: glorv <glorvs@163.com> Signed-off-by: 3AceShowHand <jinl1037@hotmail.com>

close tikv#18912 Fix the incorrect assert of compaction guard introduced by tikv#18866. Signed-off-by: glorv <glorvs@163.com> Signed-off-by: 3AceShowHand <jinl1037@hotmail.com>

…v#18866) close tikv#18865 - introduce `force_partition_range` in compact guard to force partition specific range to ensure SST ingestion under this range can be ingested to the bottom level. - Add 2 new API in SST service to manage the `force_partition_range`. The `add_force_partition_range` will also trigger a manual compaction to ensure existing SST will be partitioned. Signed-off-by: glorv <glorvs@163.com>

close tikv#18912 Fix the incorrect assert of compaction guard introduced by tikv#18866. Signed-off-by: glorv <glorvs@163.com>

glorv added 4 commits August 21, 2025 10:30

support force sst partitioning on specific ranges

1164060

Signed-off-by: glorv <glorvs@163.com>

Merge branch 'master' of https://github.com/tikv/tikv into fore-parti…

390099d

…tion-range

add unit test

3039c55

Signed-off-by: glorv <glorvs@163.com>

fmt code

7227af6

Signed-off-by: glorv <glorvs@163.com>

glorv requested review from 3pointer, Connor1996, YuJuncen, hbisheng, hhwyt and v01dstar August 21, 2025 03:54

glorv added 4 commits August 21, 2025 12:14

bypass deny

2218cba

Signed-off-by: glorv <glorvs@163.com>

update kvproto

84c9ce7

Signed-off-by: glorv <glorvs@163.com>

fix clippy

9f55c02

Signed-off-by: glorv <glorvs@163.com>

fix clippy again

73cc40b

Signed-off-by: glorv <glorvs@163.com>

Connor1996 approved these changes Aug 21, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Aug 21, 2025

v01dstar reviewed Aug 23, 2025

View reviewed changes

v01dstar approved these changes Aug 23, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Aug 23, 2025

hbisheng reviewed Aug 25, 2025

View reviewed changes

fix comments

e522180

Signed-off-by: glorv <glorvs@163.com>

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 27, 2025

ti-chi-bot bot added the approved label Aug 28, 2025

ti-chi-bot bot merged commit 3899697 into tikv:master Aug 29, 2025
8 checks passed

ti-chi-bot bot added this to the Pool milestone Aug 29, 2025

glorv deleted the fore-partition-range branch August 29, 2025 01:51

ti-chi-bot bot pushed a commit that referenced this pull request Sep 1, 2025

rocksdb: fix compaction guard overlap check (#18913)

d29f835

close #18912 Fix the incorrect assert of compaction guard introduced by #18866. Signed-off-by: glorv <glorvs@163.com>

yibin87 pushed a commit to yibin87/tikv that referenced this pull request Sep 2, 2025

rocksdb: fix compaction guard overlap check (tikv#18913)

c7e3978

close tikv#18912 Fix the incorrect assert of compaction guard introduced by tikv#18866. Signed-off-by: glorv <glorvs@163.com>

glorv added needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. labels Sep 8, 2025

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Sep 8, 2025

This is an automated cherry-pick of tikv#18866

a06eb07

close tikv#18865 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Sep 8, 2025

This is an automated cherry-pick of tikv#18866

aa20ed1

close tikv#18865 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Sep 8, 2025

engine_rocks: introduce force_partition_range in compact guard (#18866) #18934

Merged

9 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this pull request Sep 8, 2025

This is an automated cherry-pick of tikv#18866

00137a5

close tikv#18865 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Sep 8, 2025

engine_rocks: introduce force_partition_range in compact guard (#18866) #18935

Open

9 tasks

ti-chi-bot mentioned this pull request Sep 8, 2025

engine_rocks: introduce force_partition_range in compact guard (#18866) #18936

Open

9 tasks

glorv added a commit to ti-chi-bot/tikv that referenced this pull request Sep 8, 2025

rocksdb: fix compaction guard overlap check (tikv#18913)

b005d54

close tikv#18912 Fix the incorrect assert of compaction guard introduced by tikv#18866. Signed-off-by: glorv <glorvs@163.com>

Leavrth pushed a commit to Leavrth/tikv that referenced this pull request Oct 21, 2025

rocksdb: fix compaction guard overlap check (tikv#18913)

f06716d

close tikv#18912 Fix the incorrect assert of compaction guard introduced by tikv#18866. Signed-off-by: glorv <glorvs@163.com>

glorv restored the fore-partition-range branch November 17, 2025 06:35

glorv deleted the fore-partition-range branch November 17, 2025 06:41

glorv mentioned this pull request Nov 18, 2025

the add_force_partition_range interface may trigger unnecessary compact #19120

Closed

		let ttl = r.ttl.saturating_duration_since(now);
		if ttl == Duration::ZERO {

Conversation

glorv commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is changed and how it works?

Related changes

Check List

Release note

Uh oh!

Connor1996 left a comment

Choose a reason for hiding this comment

Uh oh!

v01dstar Aug 23, 2025

Choose a reason for hiding this comment

Uh oh!

glorv Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

v01dstar Aug 23, 2025

Choose a reason for hiding this comment

Uh oh!

glorv Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

v01dstar left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Aug 23, 2025

[LGTM Timeline notifier]

Uh oh!

hbisheng left a comment

Choose a reason for hiding this comment

Uh oh!

hbisheng Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

hbisheng Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

glorv commented Aug 27, 2025

Uh oh!

ti-chi-bot bot commented Aug 28, 2025

Uh oh!

glorv commented Aug 29, 2025

Uh oh!

Uh oh!

ti-chi-bot commented Sep 8, 2025

Uh oh!

ti-chi-bot commented Sep 8, 2025

Uh oh!

ti-chi-bot commented Sep 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

glorv commented Aug 21, 2025 •

edited

Loading