[Compaction] Use separate thread pool for base and cumulative compaction #5618
Conversation
| CONF_mInt32(min_base_compaction_threads, "10"); | ||
| CONF_mInt32(max_base_compaction_threads, "10"); | ||
| CONF_mInt32(min_cumulative_compaction_threads, "10"); | ||
| CONF_mInt32(max_cumulative_compaction_threads, "10"); |
There was a problem hiding this comment.
They are not mutable, use CONF_Int32 instead.
There was a problem hiding this comment.
They are not mutable, use CONF_Int32 instead.
OK, thanks.
| std::set<TTabletId>::iterator it_tablet = | ||
| (*tablet_submitted_base_compaction).find(tablet_ptr->tablet_id()); | ||
| if (it_tablet != (*tablet_submitted_base_compaction).end()) { | ||
| continue; | ||
| } | ||
| it_tablet = (*tablet_submitted_cumulative_compaction).find(tablet_ptr->tablet_id()); | ||
| if (it_tablet != (*tablet_submitted_cumulative_compaction).end()) { | ||
| continue; | ||
| } | ||
|
|
There was a problem hiding this comment.
You can use ContainsKey in be/src/gutil/map-util.h instead to simplify code.
There was a problem hiding this comment.
You can use
ContainsKeyin be/src/gutil/map-util.h instead to simplify code.
We need to judge whether an element exists in setrather than map.
| std::unique_ptr<ThreadPool> _base_compaction_thread_pool; | ||
| std::unique_ptr<ThreadPool> _cumulative_compaction_thread_pool; | ||
|
|
||
| CompactionPermitLimiter _permit_limiter; | ||
|
|
||
| std::mutex _tablet_submitted_compaction_mutex; | ||
| std::map<DataDir*, vector<TTabletId>> _tablet_submitted_compaction; | ||
| RWMutex _tablet_submitted_base_compaction_mutex; | ||
| std::map<DataDir*, std::set<TTabletId>> _tablet_submitted_base_compaction; | ||
| RWMutex _tablet_submitted_cumulative_compaction_mutex; | ||
| std::map<DataDir*, std::set<TTabletId>> _tablet_submitted_cumulative_compaction; |
There was a problem hiding this comment.
How about use a two elements array, one for base compaction, the other for cumulative compaction, to simplify code?
There was a problem hiding this comment.
How about use a two elements array, one for base compaction, the other for cumulative compaction, to simplify code?
OK, Thank you.
…or query tvf apache#57745 (apache#5618) cherry-picks from apache#57745 Related to apache#5568 Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Proposed changes
We hope that
cumulative compactioncan be implemented as soon as possible, butbase compactiontasks are likely to execute for a long time and occupy most of the threads in thread pool, which may result in high CPU load.This patch submits
cumulative compactionandbase compactioninto separate thread pool. So we have more flexibility to adjust the execution ofcumulative compactionandbase compactionaccording to the status of cluster.Types of changes
What types of changes does your code introduce to Doris?
Put an
xin the boxes that apply