Skip to content

ddl: introduce a new system variable to control the store-write-bwlimit when ingesting (#57145)#57375

Closed
ti-chi-bot wants to merge 1 commit intopingcap:release-7.1from
ti-chi-bot:cherry-pick-57145-to-release-7.1
Closed

ddl: introduce a new system variable to control the store-write-bwlimit when ingesting (#57145)#57375
ti-chi-bot wants to merge 1 commit intopingcap:release-7.1from
ti-chi-bot:cherry-pick-57145-to-release-7.1

Conversation

@ti-chi-bot
Copy link
Member

@ti-chi-bot ti-chi-bot commented Nov 14, 2024 •
edited by CbcWestwolf
Loading

This is an automated cherry-pick of #57145

What problem does this PR solve?

Issue Number: close #57156

Problem Summary:

See the issue description

What changed and how does it work?

Introduce a new system variable to control the store-write-bwlimit when ingesting

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Manual test:

  1. bootstrap a cluster
  2. prepare 1M rows data
  3. Turn on tidb_ddl_enable_fast_reorg, and with tidb_enable_dist_task on or off, do the following test:
    1. set the tidb_ddl_reorg_max_write_speed to 0
    2. add an index for a table, the speed is fast
    3. after setting the tidb_ddl_reorg_max_write_speed to a lower value, the speed of adding index decreases
TiDB root@127.0.0.1:test> select @@global.tidb_enable_dist_task
+--------------------------------+
| @@global.tidb_enable_dist_task |
+--------------------------------+
| 0                              |
+--------------------------------+
1 row in set
Time: 0.009s
TiDB root@127.0.0.1:test> select @@global.tidb_ddl_reorg_max_write_speed
+-----------------------------------------+
| @@global.tidb_ddl_reorg_max_write_speed |
+-----------------------------------------+
| 0                                       |
+-----------------------------------------+
1 row in set
Time: 0.005s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 5.168s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '64mb';
Query OK, 0 rows affected
Time: 0.022s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.285s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 13.132s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '32mb';
Query OK, 0 rows affected
Time: 0.028s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.289s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 19.922s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '16mb';
Query OK, 0 rows affected
Time: 0.021s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.272s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 37.066s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '8mb';
Query OK, 0 rows affected
Time: 0.019s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.264s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 71.897s
TiDB root@127.0.0.1:test> set @@global.tidb_enable_dist_task = 1
Query OK, 0 rows affected
Time: 0.014s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = 0;
Query OK, 0 rows affected
Time: 0.019s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.292s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 12.792s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '64mb';
Query OK, 0 rows affected
Time: 0.017s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.281s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 23.271s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '32mb';
Query OK, 0 rows affected
Time: 0.017s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.328s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 65.301s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '16mb';
Query OK, 0 rows affected
Time: 0.017s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.294s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 75.756s
TiDB root@127.0.0.1:test> set @@global.tidb_ddl_reorg_max_write_speed = '8mb';
Query OK, 0 rows affected
Time: 0.008s
TiDB root@127.0.0.1:test> alter table t1 drop index idx_data;
Query OK, 0 rows affected
Time: 0.279s
TiDB root@127.0.0.1:test> CREATE INDEX idx_data ON test.t1 (data)
Query OK, 0 rows affected
Time: 143.587s
image

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Add a system variable tidb_ddl_reorg_max_write_speed to limit the speed of ingesting when adding index
增加系统变量 tidb_ddl_reorg_max_write_speed 来限制加索引时 ingesting 的速度上限

@ti-chi-bot ti-chi-bot requested a review from a team as a code owner November 14, 2024 05:53
@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. type/cherry-pick-for-release-7.1 This PR is cherry-picked to release-7.1 from a source PR. labels Nov 14, 2024
@ti-chi-bot ti-chi-bot mentioned this pull request Nov 14, 2024
Merged
13 tasks
@ti-chi-bot ti-chi-bot bot added do-not-merge/cherry-pick-not-approved size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 14, 2024
@CbcWestwolf CbcWestwolf force-pushed the cherry-pick-57145-to-release-7.1 branch from d7c63f4 to 48335a6 Compare November 14, 2024 08:53
@ti-chi-bot ti-chi-bot bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 14, 2024
@CbcWestwolf
Copy link
Member

CbcWestwolf commented Nov 14, 2024

/retest

1 similar comment
@CbcWestwolf
Copy link
Member

CbcWestwolf commented Nov 14, 2024

/retest

@codecov
Copy link

codecov bot commented Nov 14, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 58.82353% with 7 lines in your changes missing coverage. Please review.

Please upload report for BASE (release-7.1@a276364). Learn more about missing BASE report.

Additional details and impacted files 
@@               Coverage Diff                @@
##             release-7.1     #57375   +/-   ##
================================================
  Coverage               ?   73.4592%           
================================================
  Files                  ?       1213           
  Lines                  ?     382062           
  Branches               ?          0           
================================================
  Hits                   ?     280660           
  Misses                 ?      83476           
  Partials               ?      17926
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ti-chi-bot ti-chi-bot bot added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved labels Nov 21, 2024
@ti-chi-bot
Copy link

ti-chi-bot bot commented Dec 10, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CbcWestwolf

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Dec 10, 2024
@ti-chi-bot
Copy link

ti-chi-bot bot commented Dec 10, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-12-10 08:13:24.011921998 +0000 UTC m=+339794.100724536: ☑️ agreed by CbcWestwolf.

@ti-chi-bot ti-chi-bot bot added the approved label Dec 10, 2024
@ti-chi-bot ti-chi-bot bot removed the cherry-pick-approved Cherry pick PR approved by release team. label Mar 4, 2025
@ti-chi-bot
Copy link

ti-chi-bot bot commented Mar 4, 2025

This cherry pick PR is for a release branch and has not yet been approved by triage owners.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick:

  1. It must be approved by the approvers firstly.
  2. AFTER it has been approved by approvers, please wait for the cherry-pick merging approval from triage owners.
Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot closed this Apr 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CbcWestwolf CbcWestwolf CbcWestwolf approved these changes

Assignees

@CbcWestwolf CbcWestwolf

Labels

approved do-not-merge/cherry-pick-not-approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. type/cherry-pick-for-release-7.1 This PR is cherry-picked to release-7.1 from a source PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ti-chi-bot @CbcWestwolf