Skip to content

Make import.num_threads be able to config online#17665

Merged
ti-chi-bot[bot] merged 32 commits intotikv:masterfrom
RidRisR:onlineConfig
Nov 5, 2024
Merged

Make import.num_threads be able to config online#17665
ti-chi-bot[bot] merged 32 commits intotikv:masterfrom
RidRisR:onlineConfig

Conversation

@RidRisR
Copy link
Contributor

@RidRisR RidRisR commented Oct 16, 2024 •
edited
Loading

What is changed and how it works?

Issue Number: Close #17572

What's Changed:

We add a new thread pool in utils named ResizableRuntime which is able to resize itself without killing the running tasks.
With this new thread pool, import.num_threads is able to change online without restart the tikv server.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code
[root@tikv]# ps -TC tikv-server -o comm | uniq -c
      #...
      8 import
      #...
[root@tikv]# tikv-ctl --host 127.0.0.1:20160 modify-tikv-config -n import.num_threads -v 4
success
[root@tikv]# ps -TC tikv-server -o comm | uniq -c
      #...
      4 import
      #...

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Release note

Make `import.num_threads` be able to change online without restart the tikv-server.

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. contribution This PR is from a community contributor. labels Oct 16, 2024
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Oct 16, 2024

Hi @RidRisR. Thanks for your PR.

I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot bot added needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. dco-signoff: no Indicates the PR's author has not signed dco. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 16, 2024
@RidRisR RidRisR force-pushed the onlineConfig branch 2 times, most recently from 8f1d2a4 to ea65110 Compare October 21, 2024 09:40
@ti-chi-bot ti-chi-bot bot added dco-signoff: yes Indicates the PR's author has signed the dco. dco-signoff: no Indicates the PR's author has not signed dco. and removed dco-signoff: no Indicates the PR's author has not signed dco. dco-signoff: yes Indicates the PR's author has signed the dco. labels Oct 21, 2024
@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 21, 2024
@RidRisR
merge
6046723 
Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
@ti-chi-bot ti-chi-bot bot added dco-signoff: yes Indicates the PR's author has signed the dco. and removed dco-signoff: no Indicates the PR's author has not signed dco. labels Oct 22, 2024
@RidRisR
Copy link
Contributor Author

RidRisR commented Oct 22, 2024

/test pull-unit-test

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Oct 22, 2024

@RidRisR: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/test pull-unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@Leavrth
Copy link
Contributor

Leavrth commented Nov 1, 2024

/unhold

@ti-chi-bot ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 1, 2024
@Leavrth
Copy link
Contributor

Leavrth commented Nov 2, 2024

/ok-to-test

@ti-chi-bot ti-chi-bot bot added ok-to-test Indicates a PR is ready to be tested. and removed needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Nov 2, 2024
ti-chi-bot bot and others added 3 commits November 3, 2024 09:25
@ti-chi-bot
Merge branch 'master' into onlineConfig
630e595
@ti-chi-bot
Merge branch 'master' into onlineConfig
8472449
@RidRisR
lint
218c5af 
Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
@ti-chi-bot ti-chi-bot bot added dco-signoff: no Indicates the PR's author has not signed dco. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed dco-signoff: yes Indicates the PR's author has signed the dco. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 4, 2024
RidRisR and others added 10 commits November 4, 2024 03:47
@RidRisR
Revert "lint"
49a726c 
This reverts commit 218c5af.

Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
@RidRisR
lint
f6972d0 
Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
@ti-chi-bot
Merge branch 'master' into onlineConfig
6f16f57
@RidRisR
unused import
ca77b0d 
Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
@RidRisR
remove unused func
190faf3 
Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
@RidRisR
mod test
c813f43 
Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
@ti-chi-bot
Merge branch 'master' into onlineConfig
b238732
@RidRisR
handle closure
4e8af91 
Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
@RidRisR
lint
587aec6 
Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
@ti-chi-bot
Merge branch 'master' into onlineConfig
eefe452
@RidRisR
Copy link
Contributor Author

RidRisR commented Nov 5, 2024

/test pull-unit-test

@BornChanger
Copy link
Contributor

BornChanger commented Nov 5, 2024

/retest

ti-chi-bot bot and others added 2 commits November 5, 2024 07:36
@ti-chi-bot
Merge branch 'master' into onlineConfig
67edcdd
@RidRisR
longer timeout
c69d4d6 
Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
@RidRisR
Copy link
Contributor Author

RidRisR commented Nov 5, 2024

/test pull-unit-test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhangjinpeng87 zhangjinpeng87 zhangjinpeng87 approved these changes

@YuJuncen YuJuncen YuJuncen approved these changes

+1 more reviewer

@Leavrth Leavrth Leavrth approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved contribution This PR is from a community contributor. dco-signoff: yes Indicates the PR's author has signed the dco. lgtm ok-to-test Indicates a PR is ready to be tested. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Milestone

Pool

Development

Successfully merging this pull request may close these issues.

br: make import.num-threads dynamic

5 participants

@RidRisR @YuJuncen @Leavrth @BornChanger @zhangjinpeng87