-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[refactor] Optimize threads usage mode in BE #4440
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
morningman
previously approved these changes
Aug 28, 2020
Contributor
morningman
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Contributor
|
Hi @acelyc111 please resolve the conflict |
BE can not graceful exit because some threads are running in endless loop. This patch do the following optimization: - Use the well encapsulated Thread and ThreadPool instead of std::thread and std::vector<std::thread> - Use CountDownLatch in thread's loop condition to avoid endless loop - Introduce a new class Daemon for daemon works, like tcmalloc_gc, memory_maintenance and calculate_metrics - Decouple statistics type TaskWorkerPool and StorageEngine notification by submit tasks to TaskWorkerPool's queue - Reorder objects' stop and deconstruct in main(), i.e. stop network services at first, then internal services - Use libevent in pthreads mode, by calling evthread_use_pthreads(), then EvHttpServer can exit gracefully in multi-threads - Call brpc::Server's Stop() and ClearServices() explicitly
c5fd736 to
9413ab4
Compare
Member
Author
Resolved |
chaoyli
requested changes
Sep 2, 2020
chaoyli
approved these changes
Sep 2, 2020
chaoyli
approved these changes
Sep 2, 2020
morningman
pushed a commit
to morningman/doris
that referenced
this pull request
Sep 7, 2020
The number of thread initialized in task worker pool is not right. This bug is introduced from apache#4440
1 task
morningman
added a commit
that referenced
this pull request
Sep 8, 2020
The number of thread initialized in task worker pool is not right. This bug is introduced from #4440
morningman
pushed a commit
to morningman/doris
that referenced
this pull request
Sep 14, 2020
The tablet and disk information reporting threads need to report to the FE periodically. At the same time these two reporting threads will also be triggered by certain events. The modification in PR apache#4440 caused these two threads to be triggered only by events, and could not report regularly.
1 task
morningman
added a commit
that referenced
this pull request
Sep 20, 2020
The tablet and disk information reporting threads need to report to the FE periodically. At the same time these two reporting threads will also be triggered by certain events. The modification in PR #4440 caused these two threads to be triggered only by events, and could not report regularly.
Closed
w41ter
pushed a commit
to w41ter/incubator-doris
that referenced
this pull request
Oct 17, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
BE can not graceful exit because some threads are running in endless
loop. This patch do the following optimization:
and std::vectorstd::thread
memory_maintenance and calculate_metrics
by submit tasks to TaskWorkerPool's queue
services at first, then internal services
then EvHttpServer can exit gracefully in multi-threads
Types of changes
What types of changes does your code introduce to Doris?
Put an
xin the boxes that applyChecklist
Put an
xin the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.Further comments
none