[feature](profilev2) Preliminary support for profilev2. #24881
Conversation
|
run buildall |
| DCHECK(parent_counter_name == ROOT_COUNTER || | ||
| _counter_map.find(parent_counter_name) != _counter_map.end()); | ||
| Counter* counter = _pool->add(new Counter(type, 0)); | ||
| Counter* counter = _pool->add(new Counter(type, 0, level)); |
There was a problem hiding this comment.
warning: variable 'counter' is not initialized [cppcoreguidelines-init-variables]
| Counter* counter = _pool->add(new Counter(type, 0, level)); | |
| Counter* counter = nullptr = _pool->add(new Counter(type, 0, level)); |
| _metadata = md; | ||
| } | ||
|
|
||
| bool is_set_metadata() const { return _is_set_metadata; } |
There was a problem hiding this comment.
warning: function 'is_set_metadata' should be marked [[nodiscard]] [modernize-use-nodiscard]
| bool is_set_metadata() const { return _is_set_metadata; } | |
| [[nodiscard]] bool is_set_metadata() const { return _is_set_metadata; } |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
gensrc/thrift/RuntimeProfile.thrift
Outdated
| 1: required string name | ||
| 2: required Metrics.TUnit type | ||
| 3: required i64 value | ||
| 4: required i64 level |
There was a problem hiding this comment.
should always be optional, never use required.
be/src/util/runtime_profile.h
Outdated
| return add_counter(name, type, ""); | ||
| } | ||
|
|
||
| Counter* add_counter_with_levle(const std::string& name, TUnit::type type, int64_t level) { |
|
run buildall |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
| output.append(prefix).append("SHORT-CIRCUIT"); | ||
| } | ||
|
|
||
| return output.toString(); |
| return builder.toString(); | ||
| } | ||
|
|
||
| public String getSimpleString() { |
There was a problem hiding this comment.
rename to getLevel1Profile
| return; | ||
| } | ||
| if (node.isSetMetadata()) { | ||
| this.nodeid = (int) node.getMetadata(); |
There was a problem hiding this comment.
I think we should not depend on it.
Add a specific field in profile object named plan_node_id.
metadata is too normal.
|
run buildall |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
run buildall |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
TeamCity be ut coverage result: |
| std::lock_guard<std::mutex> l(_children_lock); | ||
| DCHECK(_child_map.find(name) == _child_map.end()); | ||
| RuntimeProfile* child = _pool->add(new RuntimeProfile(name)); | ||
| if (this->is_set_metadata()) { |
There was a problem hiding this comment.
这样做可能是错的，因为如果meta data 里是node id，那么child 会有自己的node id
|
run buildall |
|
run buildall |
|
TeamCity be ut coverage result: |
|
(From new machine)TeamCity pipeline, clickbench performance test result: |
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
You can set the level of counters on the backend using ADD_COUNTER_WITH_LEVEL/ADD_TIMER_WITH_LEVEL. The profile can then merge counters with level 1.
set profile_level = 1;
such as
sql
select count(*) from customer join item on c_customer_sk = i_item_sk
profile
Simple profile
PLAN FRAGMENT 0
OUTPUT EXPRS:
count(*)
PARTITION: UNPARTITIONED
VRESULT SINK
MYSQL_PROTOCAL
7:VAGGREGATE (merge finalize)
| output: count(partial_count(*))[apache#44]
| group by:
| cardinality=1
| TotalTime: avg 725.608us, max 725.608us, min 725.608us
| RowsReturned: 1
|
6:VEXCHANGE
offset: 0
TotalTime: avg 52.411us, max 52.411us, min 52.411us
RowsReturned: 8
PLAN FRAGMENT 1
PARTITION: HASH_PARTITIONED: c_customer_sk
STREAM DATA SINK
EXCHANGE ID: 06
UNPARTITIONED
TotalTime: avg 106.263us, max 118.38us, min 81.403us
BlocksSent: 8
5:VAGGREGATE (update serialize)
| output: partial_count(*)[apache#43]
| group by:
| cardinality=1
| TotalTime: avg 679.296us, max 739.395us, min 554.904us
| BuildTime: avg 33.198us, max 48.387us, min 28.880us
| ExecTime: avg 27.633us, max 40.278us, min 24.537us
| RowsReturned: 8
|
4:VHASH JOIN
| join op: INNER JOIN(PARTITIONED)[]
| equal join conjunct: c_customer_sk = i_item_sk
| runtime filters: RF000[bloom] <- i_item_sk(18000/16384/1048576)
| cardinality=17,740
| vec output tuple id: 3
| vIntermediate tuple ids: 2
| hash output slot ids: 22
| RowsReturned: 18.0K (18000)
| ProbeRows: 18.0K (18000)
| ProbeTime: avg 862.308us, max 1.576ms, min 666.28us
| BuildRows: 18.0K (18000)
| BuildTime: avg 3.8ms, max 3.860ms, min 2.317ms
|
|----1:VEXCHANGE
| offset: 0
| TotalTime: avg 48.822us, max 67.459us, min 30.380us
| RowsReturned: 18.0K (18000)
|
3:VEXCHANGE
offset: 0
TotalTime: avg 33.162us, max 39.480us, min 28.854us
RowsReturned: 18.0K (18000)
PLAN FRAGMENT 2
PARTITION: HASH_PARTITIONED: c_customer_id
STREAM DATA SINK
EXCHANGE ID: 03
HASH_PARTITIONED: c_customer_sk
TotalTime: avg 753.954us, max 1.210ms, min 499.470us
BlocksSent: 64
2:VOlapScanNode
TABLE: default_cluster:tpcds.customer(customer), PREAGGREGATION: ON
runtime filters: RF000[bloom] -> c_customer_sk
partitions=1/1, tablets=12/12, tabletList=1550745,1550747,1550749 ...
cardinality=100000, avgRowSize=0.0, numNodes=1
pushAggOp=NONE
TotalTime: avg 18.417us, max 41.319us, min 10.189us
RowsReturned: 18.0K (18000)
---------
Co-authored-by: yiguolei <676222867@qq.com>
Proposed changes
You can set the level of counters on the backend using ADD_COUNTER_WITH_LEVEL/ADD_TIMER_WITH_LEVEL. The profile can then merge counters with level 1.
set profile_level = 1;
such as
sql
select count(*) from customer join item on c_customer_sk = i_item_sk
profile
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...