ddl: introduce a new system variable to control the `store-write-bwlimit` when ingesting (#57145) by ti-chi-bot · Pull Request #57377 · pingcap/tidb

ti-chi-bot · 2024-11-14T05:55:11Z

This is an automated cherry-pick of #57145

What problem does this PR solve?

Issue Number: close #57156

Problem Summary:

See the issue description

What changed and how does it work?

Introduce a new system variable to control the store-write-bwlimit when ingesting

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Manual test:

bootstrap a cluster
prepare 1M rows data
Turn on tidb_ddl_enable_fast_reorg, and with tidb_enable_dist_task on or off, do the following test:
1. set the tidb_ddl_reorg_max_write_speed to 0
2. add an index for a table, the speed is fast
3. after setting the tidb_ddl_reorg_max_write_speed to a lower value, the speed of adding index decreases

TiDB root@127.0.0.1:test> select @@global.tidb_ddl_reorg_max_write_speed
+-----------------------------------------+
| @@global.tidb_ddl_reorg_max_write_speed |
+-----------------------------------------+
| 0                                       |
+-----------------------------------------+
1 row in set
Time: 0.006s
TiDB root@127.0.0.1:test> select @@global.tidb_enable_dist_task
+--------------------------------+
| @@global.tidb_enable_dist_task |
+--------------------------------+
| 1                              |
+--------------------------------+
1 row in set
Time: 0.007s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.265s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 5.554s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '64mb';
Query OK, 0 rows affected
Time: 0.017s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.284s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 22.273s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '32mb';
Query OK, 0 rows affected
Time: 0.025s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.269s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 22.390s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '16mb';
Query OK, 0 rows affected
Time: 0.023s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.283s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 76.556s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '8mb';
Query OK, 0 rows affected
Time: 0.023s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.289s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 76.525s
TiDB root@127.0.0.1:test> set @@global.tidb_enable_dist_task = 0;
Query OK, 0 rows affected
Time: 0.015s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = 0;
Query OK, 0 rows affected
Time: 0.023s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.280s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 5.284s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '64mb';
Query OK, 0 rows affected
Time: 0.031s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.278s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 15.354s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '32mb';
Query OK, 0 rows affected
Time: 0.020s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.276s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 21.400s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '16mb';
Query OK, 0 rows affected
Time: 0.008s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.291s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 39.388s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '8mb';
Query OK, 0 rows affected
Time: 0.014s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.282s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 75.431s

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Add a system variable tidb_ddl_reorg_max_write_speed to limit the speed of ingesting when adding index
增加系统变量 tidb_ddl_reorg_max_write_speed 来限制加索引时 ingesting 的速度上限

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

codecov · 2024-11-14T07:47:01Z

Codecov Report

Attention: Patch coverage is 65.38462% with 9 lines in your changes missing coverage. Please review.

Please upload report for BASE (release-8.1@1a581e1). Learn more about missing BASE report.

Additional details and impacted files

@@               Coverage Diff                @@
##             release-8.1     #57377   +/-   ##
================================================
  Coverage               ?   72.4951%           
================================================
  Files                  ?       1468           
  Lines                  ?     436988           
  Branches               ?          0           
================================================
  Hits                   ?     316795           
  Misses                 ?      99945           
  Partials               ?      20248

Flag	Coverage Δ
unit	`72.4951% <65.3846%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.9656% <0.0000%> (?)`
parser	`∅ <0.0000%> (?)`
br	`41.5312% <0.0000%> (?)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ti-chi-bot · 2025-06-23T08:52:07Z

[LGTM Timeline notifier]

Timeline:

2024-12-10 07:09:11.664509846 +0000 UTC m=+335941.753312376: ☑️ agreed by CbcWestwolf.
2025-06-23 08:52:06.156367611 +0000 UTC m=+693778.879546594: ☑️ agreed by fzzf678.

yudongusa

please also open a document PR

ti-chi-bot · 2025-06-23T16:25:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CbcWestwolf, fzzf678, tangenta, yudongusa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [CbcWestwolf,tangenta]
~~pkg/ddl/OWNERS~~ [tangenta]
~~pkg/sessionctx/variable/OWNERS~~ [yudongusa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This is an automated cherry-pick of pingcap#57145

503d49e

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. type/cherry-pick-for-release-8.1 This PR is cherry-picked to release-8.1 from a source PR. labels Nov 14, 2024

ti-chi-bot mentioned this pull request Nov 14, 2024

ddl: introduce a new system variable to control the store-write-bwlimit when ingesting #57145

Merged

13 tasks

ti-chi-bot assigned CbcWestwolf Nov 14, 2024

CbcWestwolf force-pushed the cherry-pick-57145-to-release-8.1 branch from a24fe2f to 75b5d24 Compare November 14, 2024 07:05

ti-chi-bot bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 14, 2024

fix

99ec7bc

CbcWestwolf force-pushed the cherry-pick-57145-to-release-8.1 branch from 75b5d24 to 99ec7bc Compare November 14, 2024 07:13

fix bazel

229c0cc

CbcWestwolf approved these changes Dec 10, 2024

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. do-not-merge/cherry-pick-not-approved and removed cherry-pick-approved Cherry pick PR approved by release team. labels Dec 10, 2024

ti-chi-bot bot added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved labels Dec 26, 2024

ti-chi-bot bot added do-not-merge/cherry-pick-not-approved and removed cherry-pick-approved Cherry pick PR approved by release team. labels Mar 4, 2025

ti-chi-bot bot added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved labels Apr 23, 2025

fzzf678 approved these changes Jun 23, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 23, 2025

tangenta approved these changes Jun 23, 2025

View reviewed changes

yudongusa approved these changes Jun 23, 2025

View reviewed changes

ti-chi-bot bot added the approved label Jun 23, 2025

ti-chi-bot bot merged commit ae2ded3 into pingcap:release-8.1 Jun 23, 2025
9 checks passed

CbcWestwolf deleted the cherry-pick-57145-to-release-8.1 branch June 24, 2025 01:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: introduce a new system variable to control the `store-write-bwlimit` when ingesting (#57145)#57377

ddl: introduce a new system variable to control the `store-write-bwlimit` when ingesting (#57145)#57377
ti-chi-bot[bot] merged 3 commits intopingcap:release-8.1from
ti-chi-bot:cherry-pick-57145-to-release-8.1

ti-chi-bot commented Nov 14, 2024 •

edited by CbcWestwolf

Loading

Uh oh!

codecov bot commented Nov 14, 2024 •

edited

Loading

Uh oh!

ti-chi-bot bot commented Jun 23, 2025

Uh oh!

yudongusa left a comment

Uh oh!

ti-chi-bot bot commented Jun 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

ti-chi-bot commented Nov 14, 2024 • edited by CbcWestwolf Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

codecov bot commented Nov 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ti-chi-bot bot commented Jun 23, 2025

[LGTM Timeline notifier]

Uh oh!

yudongusa left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Jun 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ti-chi-bot commented Nov 14, 2024 •

edited by CbcWestwolf

Loading

codecov bot commented Nov 14, 2024 •

edited

Loading