flow-control: fix compaction slowdown caused by increased flow-control thresholds. by hhwyt · Pull Request #18710 · tikv/tikv

hhwyt · 2025-07-15T08:49:10Z

What is changed and how it works?

Issue Number: Close #18708

What's Changed:

This PR addresses performance stability issues caused by increasing
storage.flow-control.l0-file-threshold and
storage.flow-control.soft-pending-compaction-bytes-limit. Previously,
raising these values could reduce the effectiveness of RocksDB’s compaction
speed-up mechanism, because the RocksDB internal thresholds
(level0-slowdown-writes-trigger and soft-pending-compaction-bytes-limit)
would be overridden, delaying compaction acceleration.

Key improvements:
1. Conditional override of RocksDB thresholds:
  - level0-slowdown-writes-trigger is overridden by l0-file-threshold
only if it is smaller.
  - soft-pending-compaction-bytes-limit is overridden only if it is
smaller than storage.flow-control.soft-pending-compaction-bytes-limit.
This ensures that increasing flow-control settings does not weaken
compaction acceleration, while user-configured RocksDB thresholds that
are larger than the flow-control limits are overriden, allowing compaction
speed-up to trigger before write flow control.
flow control.
3. Updated write stall check:
  - ingest_maybe_slowdown_writes now uses level0-stop-writes-trigger
instead of level0-slowdown-writes-trigger to determine whether ingest
may trigger a write stall.
  - This keeps the original behavior, since `l0-file-threshold` overrides
`level0-stop-writes-trigger`, just like the previous behavior with
`level0-slowdown-writes-trigger`. Ideally, flow-control settings would
be used directly to determine write stalls, but
`ingest_maybe_slowdown_writes` cannot access the flow-control module
configuration because this function resides inside the Engine module。

After this change, write control effectively has three stages:
1. Compaction acceleration: triggered when RocksDB thresholds are reached.
2. Slowdown writes: triggered at storage.flow-control.l0-file-threshold and
storage.flow-control.soft-pending-compaction-bytes-limit.
4. Throttle writes: triggered at storage.flow-control.hard-pending-compaction-bytes-limit.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
TODO
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Release note

When storage.flow-control.enable is set to true, we only override rocksdb.level0_slowdown_writes_trigger to l0-file-threshold only when it is larger than l0-file-threshold and only override rocksdb.soft-pend-compaction-bytes-limit to storage.flow-control.soft-pending-compaction-bytes-limit only when it is larger than storage.flow-control.soft-pending-compaction-bytes-limit.

src/config/mod.rs

hhwyt · 2025-07-18T05:24:13Z

/retest

hhwyt · 2025-07-18T09:48:31Z

/hold

hbisheng · 2025-07-18T12:26:01Z

src/config/mod.rs

+                    );
+                    $cf_opts.level0_slowdown_writes_trigger = $cfg.l0_files_threshold as i32;
                }
+                // Keep override for level0_stop_writes_trigger to ensure ingest_maybe_stall


nit: this may highlight their differences?

Suggested change

// Keep override for level0_stop_writes_trigger to ensure ingest_maybe_stall

// If unset, `level0_stop_writes_trigger` defaults to `l0_files_threshold` (unlike `level0_slowdown_writes_trigger`, which defaults to 20) to ensure ingest_maybe_stall

ti-chi-bot · 2025-07-18T12:31:39Z

[LGTM Timeline notifier]

Timeline:

2025-07-18 07:02:34.891451316 +0000 UTC m=+2847207.614630284: ☑️ agreed by v01dstar.
2025-07-18 12:31:38.812580063 +0000 UTC m=+10969.814575899: ☑️ agreed by hbisheng.

Connor1996 · 2025-07-18T23:35:53Z

components/engine_rocks/src/misc.rs

+            // Use level0_stop_writes_trigger instead of level0_slowdown_writes_trigger
+            // to integrate with flow control while preserving RocksDB's compaction speed-up
+            // mechanism
+            let stop_trigger = options.get_level_zero_stop_writes_trigger();


this is confusing, why not use flow_control.l0-threshold directly?

+1. flow control parameters should be used here, rocksdb parameters should only be used to control the rocks internal compaction scheduling.

@Connor1996 @glorv
Absolutely, flow-control.l0-file-threshold is a prefect fit here. But the problem is — how can we access the flow-contrl configuration from the engine_rocks module? There isn't a proper way to do that, since flow-control is on the upper storage module.

Ideally, flow-control should be indendent module that can be accessed by both the storage module and engine_rocks, but that's not an easy thing.

I guess that's why the original code uses level0-slowdown-writes-trigger instead of l0-file-threshold, right?

glorv · 2025-07-21T02:33:04Z

src/config/mod.rs

            pub target_file_size_base: Option<ReadableSize>,
            pub level0_file_num_compaction_trigger: i32,
-            pub level0_slowdown_writes_trigger: Option<i32>,
+            pub level0_slowdown_writes_trigger: i32,


Why only change slowdown but not stop_writes? Is stop_writes still useful when write_stall is disabled?

@glorv

Why only change slowdown but not stop_writes?

The default stop_writes(36) is larger than l0-file-threshold(20). If we use this default, we cannot identify and log a warning when the user manually sets stop_writes to a larger value, as we did in the previous logic.

Is stop_writes still useful when write_stall is disabled?

No.

Ok. Please add a comment on level0_slowdown_writes_trigger as it is useless when disable_write_stall is on.

Sorry for the wrong reply.
When disable_write_stall is on, level0_slowdown_writes_trigger is used for compaction speed-up mechanism, and level0_stop_writes_triiger is used for determine wheter triggering a write stall.

Comments are already in fill_cf_opt!

hhwyt · 2025-09-20T12:40:19Z

/retest

Signed-off-by: hhwyt <hhwyt1@gmail.com> address comments Signed-off-by: hhwyt <hhwyt1@gmail.com> set default values Signed-off-by: hhwyt <hhwyt1@gmail.com> flow-control: remove override for level0-slowdown-writes-trigger and soft-pending-compaction-bytes-limit Signed-off-by: hhwyt <hhwyt1@gmail.com>

Signed-off-by: hhwyt <hhwyt1@gmail.com>

hhwyt · 2025-09-21T14:28:39Z

/unhold

hhwyt · 2025-09-21T17:10:59Z

/cc @hbisheng @glorv need a LGTM, PTAL again, thx~

glorv · 2025-09-22T02:30:17Z

/cc @zhangjinpeng87 @cfzjywxk PTAL for the config change

ti-chi-bot · 2025-09-22T02:30:23Z

@glorv: GitHub didn't allow me to request PR reviews from the following users: the, config, change, PTAL, for.

Note that only tikv members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

/cc @zhangjinpeng87 @cfzjywxk PTAL for the config change

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

zhangjinpeng87

LGTM

ti-chi-bot · 2025-09-22T16:29:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hbisheng, v01dstar, zhangjinpeng87

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [v01dstar,zhangjinpeng87]
~~src/config/OWNERS~~ [zhangjinpeng87]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-09-22T17:27:31Z

In response to a cherrypick label: new pull request created to branch release-8.5: #18994.

…l thresholds. (tikv#18710) close tikv#18708 This PR addresses performance stability issues caused by increasing storage.flow-control.l0-file-threshold and storage.flow-control.soft-pending-compaction-bytes-limit. Previously, raising these values could reduce the effectiveness of RocksDB’s compaction speed-up mechanism, because the RocksDB internal thresholds (level0-slowdown-writes-trigger and soft-pending-compaction-bytes-limit) would be overridden, delaying compaction acceleration. Key improvements: 1. Conditional override of RocksDB thresholds: - level0-slowdown-writes-trigger is overridden by l0-file-threshold only if it is smaller. - soft-pending-compaction-bytes-limit is overridden only if it is smaller than storage.flow-control.soft-pending-compaction-bytes-limit. This ensures that increasing flow-control settings does not weaken compaction acceleration, while user-configured RocksDB thresholds that are larger than the flow-control limits are overriden, allowing compaction speed-up to trigger before write flow control. flow control. 3. Updated write stall check: - ingest_maybe_slowdown_writes now uses level0-stop-writes-trigger instead of level0-slowdown-writes-trigger to determine whether ingest may trigger a write stall. - This keeps the original behavior, since `l0-file-threshold` overrides `level0-stop-writes-trigger`, just like the previous behavior with `level0-slowdown-writes-trigger`. Ideally, flow-control settings would be used directly to determine write stalls, but `ingest_maybe_slowdown_writes` cannot access the flow-control module configuration because this function resides inside the Engine module。 After this change, write control effectively has three stages: 1. Compaction acceleration: triggered when RocksDB thresholds are reached. 2. Flow control: triggered at storage.flow-control.l0-file-threshold and storage.flow-control.soft-pending-compaction-bytes-limit. 4. Stop writes: triggered at storage.flow-control.hard-pending-compaction-bytes-limit. Signed-off-by: hhwyt <hhwyt1@gmail.com> Signed-off-by: 3AceShowHand <jinl1037@hotmail.com>

…l thresholds. (#18710) (#18994) close #18708 This PR addresses performance stability issues caused by increasing storage.flow-control.l0-file-threshold and storage.flow-control.soft-pending-compaction-bytes-limit. Previously, raising these values could reduce the effectiveness of RocksDB’s compaction speed-up mechanism, because the RocksDB internal thresholds (level0-slowdown-writes-trigger and soft-pending-compaction-bytes-limit) would be overridden, delaying compaction acceleration. Key improvements: 1. Conditional override of RocksDB thresholds: - level0-slowdown-writes-trigger is overridden by l0-file-threshold only if it is smaller. - soft-pending-compaction-bytes-limit is overridden only if it is smaller than storage.flow-control.soft-pending-compaction-bytes-limit. This ensures that increasing flow-control settings does not weaken compaction acceleration, while user-configured RocksDB thresholds that are larger than the flow-control limits are overriden, allowing compaction speed-up to trigger before write flow control. flow control. 3. Updated write stall check: - ingest_maybe_slowdown_writes now uses level0-stop-writes-trigger instead of level0-slowdown-writes-trigger to determine whether ingest may trigger a write stall. - This keeps the original behavior, since `l0-file-threshold` overrides `level0-stop-writes-trigger`, just like the previous behavior with `level0-slowdown-writes-trigger`. Ideally, flow-control settings would be used directly to determine write stalls, but `ingest_maybe_slowdown_writes` cannot access the flow-control module configuration because this function resides inside the Engine module。 After this change, write control effectively has three stages: 1. Compaction acceleration: triggered when RocksDB thresholds are reached. 2. Flow control: triggered at storage.flow-control.l0-file-threshold and storage.flow-control.soft-pending-compaction-bytes-limit. 4. Stop writes: triggered at storage.flow-control.hard-pending-compaction-bytes-limit. Signed-off-by: hhwyt <hhwyt1@gmail.com> Co-authored-by: hhwyt <hhwyt1@gmail.com>

ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has signed the dco. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 15, 2025

v01dstar reviewed Jul 15, 2025

View reviewed changes

src/config/mod.rs Show resolved Hide resolved

hhwyt force-pushed the decouple-l0-file-threshold branch from 7de2145 to 3be7d14 Compare July 16, 2025 07:18

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 17, 2025

hhwyt changed the title ~~flow-control: remove override for level0-slowdown-writes-trigger and …~~ flow-control: fix flow-control override to enhance compaction stability Jul 17, 2025

hhwyt changed the title ~~flow-control: fix flow-control override to enhance compaction stability~~ flow-control: fix compaction slowdown caused by increased flow-control thresholds. Jul 18, 2025

hhwyt requested review from Connor1996 and glorv July 18, 2025 03:36

v01dstar approved these changes Jul 18, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jul 18, 2025

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 18, 2025

hbisheng approved these changes Jul 18, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 18, 2025

Connor1996 reviewed Jul 18, 2025

View reviewed changes

glorv reviewed Jul 21, 2025

View reviewed changes

hhwyt added 2 commits September 20, 2025 21:25

fix unit test

af7a83a

Signed-off-by: hhwyt <hhwyt1@gmail.com>

hhwyt force-pushed the decouple-l0-file-threshold branch from 77278e6 to af7a83a Compare September 21, 2025 05:19

hhwyt added 3 commits September 21, 2025 13:29

format

4389729

Signed-off-by: hhwyt <hhwyt1@gmail.com>

add comments

d6d82fc

Signed-off-by: hhwyt <hhwyt1@gmail.com>

fix unit test

78ff1e1

Signed-off-by: hhwyt <hhwyt1@gmail.com>

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 21, 2025

ti-chi-bot bot requested review from cfzjywxk and zhangjinpeng87 September 22, 2025 02:30

zhangjinpeng87 approved these changes Sep 22, 2025

View reviewed changes

ti-chi-bot bot added approved needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. labels Sep 22, 2025

ti-chi-bot bot merged commit 9c443de into tikv:master Sep 22, 2025
9 checks passed

ti-chi-bot bot added this to the Pool milestone Sep 22, 2025

ti-chi-bot mentioned this pull request Sep 22, 2025

flow-control: fix compaction slowdown caused by increased flow-control thresholds. (#18710) #18994

Merged

9 tasks

ti-chi-bot bot removed the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Dec 3, 2025

	// Keep override for level0_stop_writes_trigger to ensure ingest_maybe_stall
	// If unset, `level0_stop_writes_trigger` defaults to `l0_files_threshold` (unlike `level0_slowdown_writes_trigger`, which defaults to 20) to ensure ingest_maybe_stall

Conversation

hhwyt commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is changed and how it works?

Related changes

Check List

Release note

Uh oh!

Uh oh!

hhwyt commented Jul 18, 2025

Uh oh!

hhwyt commented Jul 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Jul 18, 2025

[LGTM Timeline notifier]

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hhwyt Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hhwyt Sep 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hhwyt commented Sep 20, 2025

Uh oh!

hhwyt commented Sep 21, 2025

Uh oh!

hhwyt commented Sep 21, 2025

Uh oh!

glorv commented Sep 22, 2025

Uh oh!

ti-chi-bot bot commented Sep 22, 2025

Uh oh!

zhangjinpeng87 left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Sep 22, 2025

Uh oh!

Uh oh!

ti-chi-bot commented Sep 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

hhwyt commented Jul 15, 2025 •

edited

Loading

hhwyt Jul 21, 2025 •

edited

Loading

hhwyt Sep 21, 2025 •

edited

Loading