[improve](metrics)Display garbage collector type #27408

liugddx · 2023-11-22T08:34:49Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

liugddx · 2023-11-22T08:35:01Z

run buildall

liugddx · 2023-11-22T09:10:07Z

run buildall

liugddx · 2023-11-22T10:39:27Z

run p0

liugddx · 2023-11-22T10:39:45Z

run pipelinex_p0

liugddx · 2023-11-22T10:43:35Z

@morningman

doris-robot · 2023-11-22T11:09:49Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.23 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17099035299 Bytes

doris-robot · 2023-11-22T11:43:03Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 8b0f6f2ef40bfda886f6312ba8190a13871aaae2, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4898	4592	4581	4581
q2	359	140	144	140
q3	2026	1907	1873	1873
q4	1376	1271	1251	1251
q5	3962	3956	3992	3956
q6	245	133	136	133
q7	1449	891	901	891
q8	2706	2769	2758	2758
q9	9822	9755	9473	9473
q10	3468	3524	3514	3514
q11	377	256	243	243
q12	445	293	289	289
q13	4555	3848	3808	3808
q14	318	283	287	283
q15	590	547	521	521
q16	671	586	578	578
q17	1137	960	936	936
q18	7822	7366	7323	7323
q19	1656	1659	1674	1659
q20	533	306	306	306
q21	4395	3952	3926	3926
q22	478	368	369	368
Total cold run time: 53288 ms
Total hot run time: 48810 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4563	4572	4571	4571
q2	337	233	275	233
q3	4025	3987	3989	3987
q4	2686	2676	2697	2676
q5	9619	9567	9545	9545
q6	241	126	126	126
q7	3007	2457	2503	2457
q8	4442	4439	4439	4439
q9	13193	13131	13084	13084
q10	4121	4215	4167	4167
q11	798	635	719	635
q12	987	809	812	809
q13	4252	3570	3590	3570
q14	387	357	345	345
q15	583	535	520	520
q16	716	663	700	663
q17	3934	3905	3856	3856
q18	9457	8932	8854	8854
q19	1779	1767	1767	1767
q20	2394	2100	2075	2075
q21	8686	8599	8591	8591
q22	894	768	793	768
Total cold run time: 81101 ms
Total hot run time: 77738 ms

liugddx · 2023-11-23T02:00:47Z

run buildall

doris-robot · 2023-11-23T04:08:06Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit bfbb81d1e03cd13471348d485b54e24d64d8cc9d, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4906	4639	4647	4639
q2	357	177	161	161
q3	2042	1891	1888	1888
q4	1403	1260	1244	1244
q5	3984	3924	4031	3924
q6	252	128	130	128
q7	1447	889	892	889
q8	2787	2804	2808	2804
q9	9807	9597	9400	9400
q10	3495	3540	3515	3515
q11	384	250	242	242
q12	434	292	305	292
q13	4537	3795	3788	3788
q14	334	291	290	290
q15	585	531	534	531
q16	656	582	584	582
q17	1145	952	952	952
q18	7933	7475	7458	7458
q19	1691	1681	1701	1681
q20	579	312	302	302
q21	4450	3998	4011	3998
q22	476	376	373	373
Total cold run time: 53684 ms
Total hot run time: 49081 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4570	4599	4549	4549
q2	349	229	266	229
q3	4027	4003	3997	3997
q4	2702	2687	2697	2687
q5	9787	9756	9722	9722
q6	241	125	124	124
q7	3018	2475	2456	2456
q8	4544	4565	4538	4538
q9	12989	12936	12864	12864
q10	4104	4184	4173	4173
q11	760	649	636	636
q12	977	809	815	809
q13	4297	3617	3562	3562
q14	401	361	341	341
q15	579	523	518	518
q16	763	677	691	677
q17	3929	3913	3879	3879
q18	9664	8968	9131	8968
q19	1813	1789	1797	1789
q20	2408	2053	2035	2035
q21	8813	8637	8548	8548
q22	897	790	829	790
Total cold run time: 81632 ms
Total hot run time: 77891 ms

doris-robot · 2023-11-23T04:14:30Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.01 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17099430587 Bytes

morningman

LGTM

github-actions · 2023-11-23T15:17:03Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-11-23T15:17:06Z

PR approved by anyone and no changes requested.

swjtu-zhanglei · 2023-11-24T00:51:00Z

Can you add a regression case ?

liugddx · 2023-11-24T01:59:58Z

Can you add a regression case ?

Done. thanks for your reminder.

swjtu-zhanglei · 2023-11-24T02:24:19Z

Can you add a regression case ?

Done. thanks for your reminder.

Hi, thanks for your contribution, ut and regression test is both necessary, maybe you can refer test_fe_metrics.groovy to add a regression test for covering json and promethes api

liugddx · 2023-11-24T03:23:01Z

Can you add a regression case ?

Done. thanks for your reminder.

Hi, thanks for your contribution, ut and regression test is both necessary, maybe you can refer test_fe_metrics.groovy to add a regression test for covering json and promethes api

Done. thank you for your review

liugddx · 2023-11-24T03:29:22Z

run buildall

doris-robot · 2023-11-24T05:26:18Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.05 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17100028715 Bytes

doris-robot · 2023-11-24T05:27:32Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit ce6a928d258b3ae5e7773ee597daa66576853d04, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4904	4671	4664	4664
q2	372	150	158	150
q3	2008	1953	1900	1900
q4	1392	1251	1219	1219
q5	3919	3908	3985	3908
q6	251	130	135	130
q7	1419	887	890	887
q8	2753	2778	2759	2759
q9	39294	12636	9392	9392
q10	10572	3529	3518	3518
q11	398	233	244	233
q12	626	297	294	294
q13	4638	3821	3789	3789
q14	316	291	279	279
q15	581	531	526	526
q16	674	592	587	587
q17	1123	946	919	919
q18	7767	7335	7442	7335
q19	1693	1661	1669	1661
q20	541	310	302	302
q21	5086	4051	3978	3978
q22	473	381	375	375
Total cold run time: 90800 ms
Total hot run time: 48805 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4605	4595	4570	4570
q2	341	225	240	225
q3	3984	3992	3993	3992
q4	2702	2697	2685	2685
q5	9629	9631	9661	9631
q6	245	125	127	125
q7	3024	2481	2504	2481
q8	4414	4437	4430	4430
q9	12906	12754	12913	12754
q10	4062	4153	4160	4153
q11	794	678	638	638
q12	972	806	825	806
q13	4259	3564	3587	3564
q14	374	361	359	359
q15	566	516	523	516
q16	738	731	662	662
q17	3843	3893	3926	3893
q18	9436	9028	8910	8910
q19	1773	1763	1765	1763
q20	2421	2070	2065	2065
q21	8774	8443	8318	8318
q22	880	788	813	788
Total cold run time: 80742 ms
Total hot run time: 77328 ms

morningman

LGTM

github-actions · 2023-11-27T01:34:09Z

PR approved by at least one committer and no changes requested.

hf200012

LGTM

add gc name

fb31cc4

liugddx changed the title ~~[improve](metrics) Display garbage collector type~~ [improve](metrics)Display garbage collector type Nov 22, 2023

liugddx added 3 commits November 22, 2023 16:38

Merge branch 'master' into improve-1

7a0ca8d

1

bb047fa

Merge remote-tracking branch 'origin/improve-1' into improve-1

8b0f6f2

liugddx added 3 commits November 23, 2023 07:45

add gc name

b1e1ad0

1

55ec643

add gc name

bfbb81d

morningman previously approved these changes Nov 23, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 23, 2023

github-actions bot added the reviewed label Nov 23, 2023

1

3486b26

liugddx dismissed morningman’s stale review via 3486b26 November 24, 2023 01:58

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 24, 2023

1

ce6a928

liugddx requested a review from morningman November 24, 2023 12:07

morningman approved these changes Nov 27, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 27, 2023

swjtu-zhanglei approved these changes Nov 27, 2023

View reviewed changes

hf200012 approved these changes Nov 27, 2023

View reviewed changes

morningman merged commit 5bdfaf6 into apache:master Nov 27, 2023

seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023

[improve](metrics)Display garbage collector type (apache#27408)

6e28a99

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[improve](metrics)Display garbage collector type (apache#27408)

e3d4efa

swjtu-zhanglei mentioned this pull request Dec 21, 2023

[Fix](metrics) correct fe collector type for jvm_gc #28784

Merged

[improve](metrics)Display garbage collector type #27408

[improve](metrics)Display garbage collector type #27408

Uh oh!

Conversation

liugddx commented Nov 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

liugddx commented Nov 22, 2023

Uh oh!

liugddx commented Nov 22, 2023

Uh oh!

liugddx commented Nov 22, 2023

Uh oh!

liugddx commented Nov 22, 2023

Uh oh!

liugddx commented Nov 22, 2023

Uh oh!

doris-robot commented Nov 22, 2023

Uh oh!

doris-robot commented Nov 22, 2023

Uh oh!

liugddx commented Nov 23, 2023

Uh oh!

doris-robot commented Nov 23, 2023

Uh oh!

doris-robot commented Nov 23, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 23, 2023

Uh oh!

github-actions bot commented Nov 23, 2023

Uh oh!

swjtu-zhanglei commented Nov 24, 2023

Uh oh!

liugddx commented Nov 24, 2023

Uh oh!

swjtu-zhanglei commented Nov 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liugddx commented Nov 24, 2023

Uh oh!

liugddx commented Nov 24, 2023

Uh oh!

doris-robot commented Nov 24, 2023

Uh oh!

doris-robot commented Nov 24, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 27, 2023

Uh oh!

hf200012 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

liugddx commented Nov 22, 2023 •

edited

Loading

swjtu-zhanglei commented Nov 24, 2023 •

edited

Loading