Skip to content

[Bug](dead lock) Fix dead lock in Tablet Stat Mgr #46959

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xy720 merged 8 commits into from
Jan 20, 2025
Merged

[Bug](dead lock) Fix dead lock in Tablet Stat Mgr #46959

xy720 merged 8 commits into from
Jan 20, 2025

Conversation

@xy720
Copy link
Member

@xy720 xy720 commented Jan 14, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #28608

Problem Summary:

In TabletStatMgr, We use stream().parallel() or parallelStream() in a ForkJoinTask，when they are called, the stream will allocate the ForEach task to multiple threads. However, when the stream is within a ForkJoinTask, it will attempt to steal threads from the ForkJoinPool. When the number of threads in the ForkJoinPool is small, thread competition is very likely to occur, ultimately leading to a deadlock.

Here is a deadlock stack of 4-core Fe with 3 Be:

Dead Lock Stack:

The address 0x00000005cc189d48 is the ForkJoinPool in TabletStatMgr.

"tablet stat mgr" #28 daemon prio=5 os_prio=0 cpu=12322.96ms elapsed=2159051.95s allocated=8527M defined_classes=5 tid=0x00007f4d241d6800 nid=0x24b6 in Object.wait()  [0x00007f4cfb37a000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(java.base@11.0.24/Native Method)
        - waiting on <no object reference available>
        at java.util.concurrent.ForkJoinTask.externalAwaitDone(java.base@11.0.24/ForkJoinTask.java:330)
        - waiting to re-lock in wait() <0x00000005debf6e00> (a java.util.concurrent.ForkJoinTask$AdaptedRunnableAction)
        at java.util.concurrent.ForkJoinTask.doJoin(java.base@11.0.24/ForkJoinTask.java:398)
        at java.util.concurrent.ForkJoinTask.join(java.base@11.0.24/ForkJoinTask.java:721)
        at org.apache.doris.catalog.TabletStatMgr.runAfterCatalogReady(TabletStatMgr.java:85)
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58)
        at org.apache.doris.common.util.Daemon.run(Daemon.java:119)

   Locked ownable synchronizers:
        - None

"ForkJoinPool-2-worker-9" #444184 daemon prio=5 os_prio=0 cpu=2.16ms elapsed=175074.53s allocated=1076K defined_classes=0 tid=0x00007f4d60dc6000 nid=0xd4a06 waiting on condition  [0x00007f4ce315f000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.24/Native Method)
        - parking to wait for  <0x00000005cc189d48> (a java.util.concurrent.ForkJoinPool)
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.24/LockSupport.java:194)
        at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.24/ForkJoinPool.java:1628)
        at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.24/ForkJoinWorkerThread.java:183)

"ForkJoinPool-2-worker-11" #444199 daemon prio=5 os_prio=0 cpu=1.27ms elapsed=175014.53s allocated=555K defined_classes=0 tid=0x00007f4d802a1800 nid=0xd4cd6 waiting on condition  [0x00007f4cdc32e000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.24/Native Method)
        - parking to wait for  <0x00000005cc189d48> (a java.util.concurrent.ForkJoinPool)
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.24/LockSupport.java:194)
        at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.24/ForkJoinPool.java:1628)
        at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.24/ForkJoinWorkerThread.java:183)

"ForkJoinPool-2-worker-13" #444212 daemon prio=5 os_prio=0 cpu=0.09ms elapsed=174954.53s allocated=32784B defined_classes=0 tid=0x00007f4d7cedd000 nid=0xd4fa5 waiting on condition  [0x00007f4cd9520000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.24/Native Method)
        - parking to wait for  <0x00000005cc189d48> (a java.util.concurrent.ForkJoinPool)
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.24/LockSupport.java:194)
        at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.24/ForkJoinPool.java:1628)
        at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.24/ForkJoinWorkerThread.java:183)

This commit will abandon ForkJoinPool and use a regular thread pool instead.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xy720
Copy link
Member Author

xy720 commented Jan 14, 2025

run buildall

1 similar comment
@xy720
Copy link
Member Author

xy720 commented Jan 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 14, 2025

TPC-H: Total hot run time: 33261 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a8054f4298570d1e14194f25cbac15ad96878705, data reload: false

------ Round 1 ----------------------------------
q1	17589	6275	6139	6139
q2	2056	336	174	174
q3	10479	1424	740	740
q4	10307	879	443	443
q5	7711	2260	2000	2000
q6	212	184	150	150
q7	911	757	613	613
q8	9255	1399	1227	1227
q9	5305	5021	4998	4998
q10	6780	2315	1875	1875
q11	472	290	290	290
q12	346	362	221	221
q13	17778	3742	3132	3132
q14	225	235	217	217
q15	581	504	532	504
q16	622	617	590	590
q17	585	876	345	345
q18	7212	6567	6452	6452
q19	2149	955	556	556
q20	305	319	194	194
q21	2914	2220	2078	2078
q22	374	333	323	323
Total cold run time: 104168 ms
Total hot run time: 33261 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6355	6496	6301	6301
q2	246	331	246	246
q3	2264	2669	2287	2287
q4	1428	1812	1439	1439
q5	4357	4733	5014	4733
q6	188	178	140	140
q7	2105	1970	1862	1862
q8	2700	2814	2685	2685
q9	7214	7243	7159	7159
q10	3041	3316	2850	2850
q11	609	536	496	496
q12	680	764	643	643
q13	3541	4054	3308	3308
q14	271	294	276	276
q15	607	539	520	520
q16	646	676	660	660
q17	1247	1751	1258	1258
q18	7734	7504	7182	7182
q19	811	1223	1088	1088
q20	2022	2057	1906	1906
q21	5653	5262	4870	4870
q22	615	589	548	548
Total cold run time: 54334 ms
Total hot run time: 52457 ms

@doris-robot
Copy link

doris-robot commented Jan 14, 2025

TPC-H: Total hot run time: 32704 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 356b2e212fe08259b2a9787e453fa6fc969cfb96, data reload: false

------ Round 1 ----------------------------------
q1	17616	6169	6071	6071
q2	2051	311	172	172
q3	10554	1259	737	737
q4	10264	868	431	431
q5	8268	2198	1918	1918
q6	211	181	149	149
q7	880	761	604	604
q8	9252	1371	1197	1197
q9	5287	4852	4872	4852
q10	6742	2338	1843	1843
q11	487	276	255	255
q12	347	359	216	216
q13	17762	3716	3075	3075
q14	237	233	207	207
q15	574	523	527	523
q16	634	613	599	599
q17	572	858	325	325
q18	7218	6542	6472	6472
q19	1593	956	539	539
q20	307	324	186	186
q21	2911	2166	2014	2014
q22	381	341	319	319
Total cold run time: 104148 ms
Total hot run time: 32704 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6307	6282	6252	6252
q2	244	335	243	243
q3	2269	2718	2288	2288
q4	1436	1845	1359	1359
q5	4349	4785	4814	4785
q6	193	178	149	149
q7	2133	2014	1815	1815
q8	2640	2797	2708	2708
q9	7244	7350	7055	7055
q10	2992	3380	2858	2858
q11	579	528	515	515
q12	711	781	640	640
q13	3528	3876	3263	3263
q14	285	317	293	293
q15	574	549	521	521
q16	652	696	668	668
q17	1222	1763	1261	1261
q18	7987	7559	7541	7541
q19	836	1164	1035	1035
q20	2038	2093	1865	1865
q21	5716	5309	5036	5036
q22	610	619	583	583
Total cold run time: 54545 ms
Total hot run time: 52733 ms

@doris-robot
Copy link

doris-robot commented Jan 14, 2025

TPC-DS: Total hot run time: 196231 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 356b2e212fe08259b2a9787e453fa6fc969cfb96, data reload: false

query1	1297	980	943	943
query2	7002	2166	2061	2061
query3	11079	4388	4439	4388
query4	35185	24089	23320	23320
query5	3880	671	491	491
query6	311	213	199	199
query7	4008	544	316	316
query8	293	255	239	239
query9	9550	2707	2688	2688
query10	447	323	261	261
query11	16210	15244	15131	15131
query12	167	113	105	105
query13	1563	547	405	405
query14	12695	7158	7894	7158
query15	253	215	191	191
query16	8503	730	493	493
query17	1637	796	561	561
query18	1895	422	316	316
query19	211	199	173	173
query20	123	114	112	112
query21	207	138	106	106
query22	4457	4499	4370	4370
query23	34802	33654	33461	33461
query24	6979	2326	2350	2326
query25	479	512	406	406
query26	776	295	158	158
query27	2300	491	335	335
query28	6098	2534	2472	2472
query29	566	570	465	465
query30	216	193	158	158
query31	965	917	846	846
query32	72	68	59	59
query33	494	395	317	317
query34	763	923	544	544
query35	847	851	763	763
query36	1042	1093	944	944
query37	117	109	74	74
query38	4350	4422	4307	4307
query39	1494	1460	1481	1460
query40	236	126	106	106
query41	52	51	52	51
query42	125	116	105	105
query43	545	551	512	512
query44	1437	860	868	860
query45	196	188	181	181
query46	949	1132	695	695
query47	1903	1901	1836	1836
query48	382	424	323	323
query49	727	514	414	414
query50	692	727	419	419
query51	7011	7014	6873	6873
query52	101	102	93	93
query53	232	275	189	189
query54	491	506	418	418
query55	84	80	112	80
query56	273	261	240	240
query57	1174	1136	1100	1100
query58	249	232	240	232
query59	3038	3134	3037	3037
query60	288	280	256	256
query61	117	114	114	114
query62	869	810	698	698
query63	215	193	189	189
query64	2904	1032	647	647
query65	3282	3156	3148	3148
query66	781	431	320	320
query67	15822	15810	15615	15615
query68	5099	862	547	547
query69	496	293	278	278
query70	1221	1149	1129	1129
query71	388	307	262	262
query72	5941	3991	4026	3991
query73	662	795	375	375
query74	10227	9311	9093	9093
query75	3343	3270	2676	2676
query76	3132	1276	821	821
query77	468	389	281	281
query78	10197	10004	9593	9593
query79	2592	852	644	644
query80	709	553	498	498
query81	506	286	239	239
query82	239	157	121	121
query83	174	175	148	148
query84	242	96	81	81
query85	733	387	299	299
query86	385	328	319	319
query87	4501	4528	4343	4343
query88	4827	2234	2224	2224
query89	423	338	298	298
query90	1872	197	193	193
query91	136	137	107	107
query92	76	60	54	54
query93	2327	908	542	542
query94	648	431	304	304
query95	338	271	252	252
query96	493	665	276	276
query97	2859	2927	2776	2776
query98	223	205	197	197
query99	1457	1523	1397	1397
Total cold run time: 294442 ms
Total hot run time: 196231 ms

@wm1581066 wm1581066 requested a review from dataroaring January 14, 2025 05:42
@doris-robot
Copy link

doris-robot commented Jan 14, 2025

ClickBench: Total hot run time: 31.45 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 356b2e212fe08259b2a9787e453fa6fc969cfb96, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.06	0.07
query4	1.62	0.10	0.11
query5	0.49	0.43	0.40
query6	1.19	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.59	0.50	0.49
query10	0.56	0.55	0.55
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.61	0.60
query14	2.71	2.75	2.86
query15	0.91	0.84	0.85
query16	0.39	0.38	0.39
query17	1.00	1.06	1.01
query18	0.22	0.21	0.19
query19	2.00	1.83	1.99
query20	0.01	0.01	0.01
query21	15.35	0.89	0.56
query22	0.75	0.71	0.68
query23	15.39	1.45	0.53
query24	2.78	1.20	1.74
query25	0.17	0.26	0.08
query26	0.25	0.15	0.13
query27	0.07	0.04	0.04
query28	13.95	1.52	1.05
query29	12.59	3.98	3.32
query30	0.25	0.10	0.06
query31	2.82	0.61	0.39
query32	3.23	0.55	0.46
query33	3.00	2.99	3.02
query34	16.88	5.23	4.54
query35	4.58	4.46	4.54
query36	0.65	0.48	0.47
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.18	0.14	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.23 s
Total hot run time: 31.45 s

@xy720
Copy link
Member Author

xy720 commented Jan 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 14, 2025

TPC-H: Total hot run time: 32848 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 76a2505c90906e9446878ce00505cc1a3ba012fa, data reload: false

------ Round 1 ----------------------------------
q1	17617	6175	6089	6089
q2	2045	312	182	182
q3	10393	1249	706	706
q4	10208	878	445	445
q5	7537	2213	1990	1990
q6	208	181	149	149
q7	904	758	605	605
q8	9245	1395	1145	1145
q9	5267	4921	4941	4921
q10	6749	2288	1857	1857
q11	485	278	256	256
q12	348	388	223	223
q13	17769	3703	3137	3137
q14	233	230	210	210
q15	580	527	531	527
q16	634	626	588	588
q17	575	873	325	325
q18	7211	6627	6347	6347
q19	1211	970	576	576
q20	323	344	197	197
q21	3111	2209	2052	2052
q22	377	332	321	321
Total cold run time: 103030 ms
Total hot run time: 32848 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6326	6521	6296	6296
q2	247	336	240	240
q3	2273	2676	2369	2369
q4	1415	1823	1361	1361
q5	4397	4749	4811	4749
q6	198	190	148	148
q7	2111	2038	1861	1861
q8	2644	2852	2752	2752
q9	7297	7307	7144	7144
q10	3047	3353	2848	2848
q11	615	542	513	513
q12	705	797	607	607
q13	3507	4005	3375	3375
q14	291	313	278	278
q15	603	539	525	525
q16	675	672	667	667
q17	1244	1764	1304	1304
q18	7770	7573	7404	7404
q19	857	1138	1145	1138
q20	2011	2044	1856	1856
q21	5588	5404	5084	5084
q22	611	605	574	574
Total cold run time: 54432 ms
Total hot run time: 53093 ms

@doris-robot
Copy link

doris-robot commented Jan 14, 2025

TPC-DS: Total hot run time: 195490 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 76a2505c90906e9446878ce00505cc1a3ba012fa, data reload: false

query1	1309	946	927	927
query2	6361	2063	2036	2036
query3	10977	4560	4422	4422
query4	32891	23592	23272	23272
query5	3610	609	466	466
query6	275	190	185	185
query7	3973	486	311	311
query8	293	227	217	217
query9	9494	2750	2743	2743
query10	467	317	237	237
query11	17762	15579	15318	15318
query12	154	105	102	102
query13	1528	514	381	381
query14	9777	7512	7418	7418
query15	256	222	186	186
query16	7255	659	479	479
query17	1626	740	587	587
query18	1815	404	316	316
query19	203	193	173	173
query20	123	124	117	117
query21	209	129	111	111
query22	4456	4696	4222	4222
query23	34654	34061	33947	33947
query24	6615	2324	2265	2265
query25	521	449	372	372
query26	1149	281	156	156
query27	2400	474	329	329
query28	5926	2521	2495	2495
query29	676	557	422	422
query30	214	186	163	163
query31	952	876	829	829
query32	93	57	58	57
query33	493	368	311	311
query34	798	855	514	514
query35	842	830	734	734
query36	1018	1001	954	954
query37	128	102	77	77
query38	4265	4298	4221	4221
query39	1485	1471	1418	1418
query40	211	121	101	101
query41	50	50	48	48
query42	120	104	102	102
query43	510	522	504	504
query44	1429	867	839	839
query45	184	177	166	166
query46	914	1073	650	650
query47	1892	1928	1884	1884
query48	401	408	327	327
query49	788	488	391	391
query50	679	663	397	397
query51	7044	7169	6921	6921
query52	103	103	97	97
query53	228	259	195	195
query54	506	514	412	412
query55	86	78	78	78
query56	279	271	257	257
query57	1204	1226	1121	1121
query58	240	228	226	226
query59	3176	3093	2902	2902
query60	273	263	257	257
query61	114	117	117	117
query62	831	751	724	724
query63	217	189	184	184
query64	4331	1010	638	638
query65	3271	3166	3151	3151
query66	901	397	302	302
query67	16239	15813	15738	15738
query68	7316	819	563	563
query69	479	294	264	264
query70	1214	1111	1102	1102
query71	406	285	284	284
query72	5889	3846	3747	3747
query73	663	769	367	367
query74	10052	9223	8922	8922
query75	3168	3136	2638	2638
query76	3190	1171	748	748
query77	460	341	273	273
query78	10193	10042	9270	9270
query79	4155	809	593	593
query80	1916	523	467	467
query81	560	278	237	237
query82	1172	152	122	122
query83	265	170	150	150
query84	235	92	75	75
query85	803	350	296	296
query86	481	305	309	305
query87	4421	4416	4352	4352
query88	5037	2195	2192	2192
query89	453	326	286	286
query90	1779	189	187	187
query91	127	133	106	106
query92	65	57	52	52
query93	3013	884	543	543
query94	764	396	289	289
query95	332	269	261	261
query96	489	616	291	291
query97	2813	2859	2727	2727
query98	229	198	197	197
query99	1443	1502	1365	1365
Total cold run time: 296846 ms
Total hot run time: 195490 ms

@doris-robot
Copy link

doris-robot commented Jan 14, 2025

ClickBench: Total hot run time: 30.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 76a2505c90906e9446878ce00505cc1a3ba012fa, data reload: false

query1	0.03	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.08	0.07
query4	1.61	0.11	0.11
query5	0.44	0.42	0.42
query6	1.14	0.66	0.65
query7	0.03	0.01	0.02
query8	0.04	0.04	0.03
query9	0.58	0.52	0.51
query10	0.54	0.56	0.54
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.61	0.60
query14	2.73	2.88	2.77
query15	0.90	0.82	0.84
query16	0.39	0.39	0.38
query17	1.01	1.07	1.08
query18	0.22	0.22	0.21
query19	1.89	1.99	1.86
query20	0.01	0.01	0.01
query21	15.38	0.96	0.59
query22	0.77	0.87	0.62
query23	15.26	1.44	0.54
query24	3.24	0.48	1.45
query25	0.26	0.25	0.15
query26	0.30	0.13	0.13
query27	0.06	0.04	0.05
query28	13.53	1.55	1.05
query29	12.63	3.89	3.27
query30	0.25	0.09	0.06
query31	2.82	0.60	0.39
query32	3.24	0.55	0.46
query33	2.98	3.02	3.10
query34	16.85	5.15	4.42
query35	4.47	4.55	4.48
query36	0.63	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.95 s
Total hot run time: 30.85 s

@xy720
Copy link
Member Author

xy720 commented Jan 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 14, 2025

TPC-H: Total hot run time: 32710 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e81bf3f12c440e940ba58df55830573f378dfe18, data reload: false

------ Round 1 ----------------------------------
q1	17585	6348	6055	6055
q2	2047	329	167	167
q3	10394	1278	706	706
q4	10220	864	434	434
q5	7529	2205	2001	2001
q6	211	187	149	149
q7	888	761	634	634
q8	9243	1362	1171	1171
q9	5286	4942	4862	4862
q10	6739	2276	1881	1881
q11	491	275	257	257
q12	348	355	215	215
q13	17755	3664	3097	3097
q14	233	238	210	210
q15	577	514	523	514
q16	619	618	586	586
q17	586	854	322	322
q18	7077	6585	6399	6399
q19	1196	964	567	567
q20	309	328	194	194
q21	2881	2219	1988	1988
q22	376	323	301	301
Total cold run time: 102590 ms
Total hot run time: 32710 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6242	6249	6236	6236
q2	244	329	228	228
q3	2311	2656	2325	2325
q4	1397	1838	1351	1351
q5	4324	4784	4834	4784
q6	185	176	145	145
q7	2094	1987	1861	1861
q8	2598	2760	2694	2694
q9	7231	7222	7252	7222
q10	3027	3319	2835	2835
q11	596	517	498	498
q12	648	798	636	636
q13	3464	3888	3233	3233
q14	309	294	284	284
q15	575	536	524	524
q16	631	663	649	649
q17	1234	1716	1248	1248
q18	7876	7428	7479	7428
q19	815	1164	1086	1086
q20	2042	2080	1878	1878
q21	5691	5252	4969	4969
q22	618	624	596	596
Total cold run time: 54152 ms
Total hot run time: 52710 ms

@doris-robot
Copy link

doris-robot commented Jan 14, 2025

TPC-DS: Total hot run time: 188227 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e81bf3f12c440e940ba58df55830573f378dfe18, data reload: false

query1	955	375	373	373
query2	6534	2055	1992	1992
query3	6715	216	219	216
query4	33564	23542	23517	23517
query5	4400	653	477	477
query6	290	202	191	191
query7	4637	485	316	316
query8	288	256	229	229
query9	9580	2749	2747	2747
query10	479	320	259	259
query11	18238	15350	15038	15038
query12	170	108	111	108
query13	1680	508	391	391
query14	10719	7298	6621	6621
query15	229	199	191	191
query16	7920	604	443	443
query17	1581	720	539	539
query18	2109	386	293	293
query19	216	172	150	150
query20	115	108	119	108
query21	210	119	102	102
query22	4177	4385	4207	4207
query23	34237	33436	33169	33169
query24	6389	2212	2363	2212
query25	515	445	376	376
query26	1036	275	161	161
query27	1987	455	328	328
query28	5264	2505	2458	2458
query29	716	539	409	409
query30	234	184	153	153
query31	950	882	803	803
query32	91	58	62	58
query33	513	355	294	294
query34	753	848	518	518
query35	797	822	719	719
query36	979	994	926	926
query37	115	101	78	78
query38	4163	4106	4052	4052
query39	1434	1425	1413	1413
query40	215	119	108	108
query41	59	61	58	58
query42	123	105	103	103
query43	503	543	490	490
query44	1416	844	837	837
query45	183	184	175	175
query46	854	1041	641	641
query47	1851	1876	1771	1771
query48	390	404	328	328
query49	791	511	409	409
query50	659	646	365	365
query51	6812	6835	6823	6823
query52	103	103	90	90
query53	217	248	189	189
query54	483	482	413	413
query55	88	83	81	81
query56	276	272	244	244
query57	1166	1164	1096	1096
query58	257	240	240	240
query59	3026	3045	2900	2900
query60	279	285	256	256
query61	118	114	113	113
query62	864	746	694	694
query63	224	194	190	190
query64	4308	1047	651	651
query65	3233	3121	3163	3121
query66	1002	435	310	310
query67	15848	15597	15523	15523
query68	4290	834	550	550
query69	478	284	251	251
query70	1201	1069	1062	1062
query71	380	345	259	259
query72	5744	3758	3780	3758
query73	665	755	366	366
query74	10360	8959	8807	8807
query75	3168	3118	2621	2621
query76	3181	1185	794	794
query77	578	374	306	306
query78	10082	10025	9337	9337
query79	3044	811	609	609
query80	731	526	433	433
query81	518	267	238	238
query82	223	149	122	122
query83	176	182	156	156
query84	240	159	80	80
query85	789	341	290	290
query86	399	311	310	310
query87	4381	4469	4436	4436
query88	4912	2193	2168	2168
query89	380	339	296	296
query90	1846	191	185	185
query91	135	131	107	107
query92	66	61	53	53
query93	2479	942	541	541
query94	660	363	303	303
query95	322	260	254	254
query96	487	621	275	275
query97	2878	2858	2815	2815
query98	227	204	196	196
query99	1413	1500	1395	1395
Total cold run time: 286760 ms
Total hot run time: 188227 ms

@doris-robot
Copy link

doris-robot commented Jan 14, 2025

ClickBench: Total hot run time: 31.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e81bf3f12c440e940ba58df55830573f378dfe18, data reload: false

query1	0.04	0.04	0.06
query2	0.08	0.04	0.03
query3	0.24	0.07	0.06
query4	1.60	0.11	0.10
query5	0.42	0.42	0.41
query6	1.14	0.66	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.50	0.50
query10	0.56	0.56	0.56
query11	0.15	0.10	0.11
query12	0.13	0.11	0.10
query13	0.61	0.60	0.59
query14	2.72	2.83	2.83
query15	0.91	0.83	0.82
query16	0.37	0.38	0.38
query17	1.02	0.99	1.06
query18	0.22	0.20	0.21
query19	1.90	1.88	1.95
query20	0.02	0.01	0.01
query21	15.38	0.92	0.58
query22	0.76	0.92	0.74
query23	15.07	1.41	0.57
query24	3.50	0.92	0.91
query25	0.18	0.21	0.13
query26	0.50	0.15	0.14
query27	0.07	0.04	0.05
query28	12.84	1.52	1.04
query29	12.57	4.00	3.21
query30	0.24	0.10	0.06
query31	2.82	0.59	0.38
query32	3.23	0.54	0.47
query33	2.95	3.01	3.11
query34	16.85	5.04	4.46
query35	4.52	4.45	4.53
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.02	0.02
Total cold run time: 105.4 s
Total hot run time: 31.26 s

@xy720 xy720 force-pushed the fix-dead-lock-tablet-stat branch from e81bf3f to 4d48c00 Compare January 15, 2025 09:00
@xy720
Copy link
Member Author

xy720 commented Jan 15, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 15, 2025

TPC-H: Total hot run time: 33078 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4d48c000e6ce6e454e52066cb4f31b54d2a53369, data reload: false

------ Round 1 ----------------------------------
q1	17603	6202	6067	6067
q2	2048	311	167	167
q3	10417	1256	710	710
q4	10206	1017	550	550
q5	7515	2379	2205	2205
q6	208	181	150	150
q7	909	767	604	604
q8	9240	1381	1204	1204
q9	5227	4953	4876	4876
q10	6819	2381	1914	1914
q11	464	288	263	263
q12	346	357	220	220
q13	17770	3743	3105	3105
q14	242	232	217	217
q15	573	526	539	526
q16	631	619	582	582
q17	564	870	332	332
q18	6982	6556	6328	6328
q19	1226	967	554	554
q20	325	332	202	202
q21	2904	2234	1985	1985
q22	376	346	317	317
Total cold run time: 102595 ms
Total hot run time: 33078 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6217	6243	6269	6243
q2	238	328	240	240
q3	2318	2666	2359	2359
q4	1421	1852	1366	1366
q5	4391	4747	4855	4747
q6	199	180	147	147
q7	2083	1974	1850	1850
q8	2607	2800	2698	2698
q9	7225	7188	7143	7143
q10	3033	3330	2782	2782
q11	613	528	506	506
q12	693	755	635	635
q13	3463	3954	3236	3236
q14	276	293	271	271
q15	563	529	520	520
q16	685	676	638	638
q17	1240	1747	1239	1239
q18	7717	7490	7327	7327
q19	805	1140	1141	1140
q20	2050	2018	1893	1893
q21	5690	5171	4914	4914
q22	609	629	585	585
Total cold run time: 54136 ms
Total hot run time: 52479 ms

@doris-robot
Copy link

doris-robot commented Jan 15, 2025

TPC-DS: Total hot run time: 189038 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4d48c000e6ce6e454e52066cb4f31b54d2a53369, data reload: false

query1	971	372	379	372
query2	6531	2141	2005	2005
query3	6921	217	219	217
query4	33628	23594	23120	23120
query5	4387	620	492	492
query6	293	186	185	185
query7	4615	484	305	305
query8	295	240	227	227
query9	9610	2771	2761	2761
query10	491	301	250	250
query11	18383	15189	14962	14962
query12	160	111	106	106
query13	1655	527	396	396
query14	10789	7302	6927	6927
query15	235	197	186	186
query16	7778	643	472	472
query17	1613	743	584	584
query18	1898	400	308	308
query19	237	183	160	160
query20	119	110	110	110
query21	222	120	102	102
query22	4201	4384	4195	4195
query23	34173	33195	33675	33195
query24	6393	2274	2222	2222
query25	463	474	374	374
query26	732	233	162	162
query27	1985	465	336	336
query28	5234	2432	2420	2420
query29	538	495	412	412
query30	233	185	168	168
query31	949	912	793	793
query32	71	61	60	60
query33	508	348	292	292
query34	768	839	494	494
query35	816	810	723	723
query36	993	1024	982	982
query37	123	97	78	78
query38	4267	4188	4163	4163
query39	1479	1426	1389	1389
query40	204	127	102	102
query41	53	52	52	52
query42	126	104	103	103
query43	529	540	485	485
query44	1372	813	804	804
query45	183	169	162	162
query46	854	1039	638	638
query47	1812	1845	1773	1773
query48	384	403	320	320
query49	738	481	404	404
query50	617	672	394	394
query51	6899	6949	6897	6897
query52	102	103	92	92
query53	227	253	191	191
query54	488	480	418	418
query55	87	80	85	80
query56	263	281	252	252
query57	1146	1158	1088	1088
query58	242	248	236	236
query59	2973	3075	2983	2983
query60	279	277	277	277
query61	131	121	120	120
query62	857	776	707	707
query63	231	198	196	196
query64	3246	1023	659	659
query65	3252	3145	3178	3145
query66	799	419	328	328
query67	15828	15612	15410	15410
query68	4738	833	550	550
query69	481	288	260	260
query70	1174	1105	1138	1105
query71	382	284	266	266
query72	5818	3829	3919	3829
query73	647	751	361	361
query74	10043	9144	8931	8931
query75	3170	3191	2654	2654
query76	3075	1178	783	783
query77	489	374	282	282
query78	10062	10092	9717	9717
query79	3124	794	618	618
query80	852	524	525	524
query81	532	275	242	242
query82	665	150	125	125
query83	169	170	149	149
query84	240	96	75	75
query85	794	350	302	302
query86	421	315	295	295
query87	4449	4578	4370	4370
query88	4951	2196	2179	2179
query89	389	333	295	295
query90	1771	204	193	193
query91	132	134	146	134
query92	67	58	54	54
query93	2013	858	536	536
query94	795	403	269	269
query95	325	269	256	256
query96	487	616	279	279
query97	2758	2855	2781	2781
query98	223	209	198	198
query99	1407	1571	1405	1405
Total cold run time: 285322 ms
Total hot run time: 189038 ms

@doris-robot
Copy link

doris-robot commented Jan 15, 2025

ClickBench: Total hot run time: 31.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4d48c000e6ce6e454e52066cb4f31b54d2a53369, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.04	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.11
query5	0.43	0.39	0.41
query6	1.17	0.65	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.50	0.49
query10	0.55	0.57	0.54
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.61	0.60
query14	2.72	2.81	2.69
query15	0.89	0.82	0.81
query16	0.39	0.39	0.39
query17	1.04	1.05	1.04
query18	0.24	0.21	0.21
query19	1.94	1.96	1.86
query20	0.02	0.01	0.01
query21	15.38	0.94	0.59
query22	0.77	0.89	0.71
query23	15.17	1.46	0.54
query24	3.42	1.39	1.86
query25	0.18	0.13	0.12
query26	0.21	0.16	0.13
query27	0.09	0.06	0.05
query28	14.89	1.45	1.05
query29	12.61	3.92	3.25
query30	0.26	0.10	0.06
query31	2.82	0.59	0.37
query32	3.22	0.54	0.46
query33	2.96	3.09	3.06
query34	16.83	5.14	4.52
query35	4.55	4.55	4.49
query36	0.64	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.37 s
Total hot run time: 31.75 s

@xy720
Copy link
Member Author

xy720 commented Jan 15, 2025

run buildall

1 similar comment
@xy720
Copy link
Member Author

xy720 commented Jan 15, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 15, 2025

TPC-H: Total hot run time: 32279 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6d1a3677391bcbf553a61c5145fdd5f7b62d558d, data reload: false

------ Round 1 ----------------------------------
q1	17590	5478	5370	5370
q2	2054	316	169	169
q3	10404	1285	712	712
q4	10217	961	536	536
q5	7524	2413	2287	2287
q6	203	173	136	136
q7	904	786	617	617
q8	9252	1395	1162	1162
q9	5205	4941	4926	4926
q10	6827	2331	1872	1872
q11	460	267	262	262
q12	340	356	218	218
q13	17765	3689	3060	3060
q14	232	230	213	213
q15	523	467	465	465
q16	640	617	590	590
q17	555	862	319	319
q18	7008	6415	6574	6415
q19	1204	956	515	515
q20	308	323	182	182
q21	2790	2190	1937	1937
q22	379	334	316	316
Total cold run time: 102384 ms
Total hot run time: 32279 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5431	5437	5725	5437
q2	242	326	238	238
q3	2265	2685	2333	2333
q4	1397	1815	1403	1403
q5	4334	4785	4630	4630
q6	162	159	126	126
q7	2040	1938	1832	1832
q8	2656	2835	2717	2717
q9	7295	7287	7334	7287
q10	2963	3228	2803	2803
q11	572	519	493	493
q12	695	753	613	613
q13	3492	3925	3310	3310
q14	294	289	273	273
q15	508	478	471	471
q16	660	711	672	672
q17	1248	1748	1258	1258
q18	7721	7431	7401	7401
q19	775	1117	1092	1092
q20	2001	2038	1942	1942
q21	5532	5215	5108	5108
q22	627	616	592	592
Total cold run time: 52910 ms
Total hot run time: 52031 ms

@doris-robot
Copy link

doris-robot commented Jan 15, 2025

TPC-DS: Total hot run time: 194698 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6d1a3677391bcbf553a61c5145fdd5f7b62d558d, data reload: false

query1	1319	966	932	932
query2	6105	2079	2046	2046
query3	11090	4528	4511	4511
query4	60550	28974	23076	23076
query5	5643	609	465	465
query6	440	212	210	210
query7	5537	505	292	292
query8	335	244	245	244
query9	8338	2722	2717	2717
query10	466	313	270	270
query11	17363	15440	15645	15440
query12	158	107	103	103
query13	1438	570	417	417
query14	11534	7338	6666	6666
query15	212	198	192	192
query16	7065	699	490	490
query17	1105	730	550	550
query18	1647	398	314	314
query19	196	200	164	164
query20	116	117	108	108
query21	215	122	103	103
query22	4572	5012	4570	4570
query23	34034	33357	33251	33251
query24	5604	2365	2322	2322
query25	485	469	393	393
query26	692	288	157	157
query27	2188	479	337	337
query28	3890	2519	2462	2462
query29	556	550	416	416
query30	224	184	156	156
query31	926	906	822	822
query32	79	62	60	60
query33	452	375	309	309
query34	746	876	519	519
query35	808	854	771	771
query36	999	1034	945	945
query37	118	111	83	83
query38	4227	4379	4332	4332
query39	1526	1425	1460	1425
query40	211	122	104	104
query41	62	57	51	51
query42	125	104	102	102
query43	536	541	496	496
query44	1316	842	822	822
query45	187	176	174	174
query46	895	1066	672	672
query47	1912	1937	1896	1896
query48	407	396	319	319
query49	766	485	394	394
query50	660	706	407	407
query51	7019	6891	6767	6767
query52	102	102	95	95
query53	227	246	180	180
query54	483	487	419	419
query55	84	83	80	80
query56	261	275	244	244
query57	1190	1188	1107	1107
query58	250	226	241	226
query59	3078	3111	2968	2968
query60	281	257	252	252
query61	146	144	150	144
query62	725	718	663	663
query63	231	193	190	190
query64	1370	1016	666	666
query65	3242	3181	3116	3116
query66	779	392	299	299
query67	16173	15846	15690	15690
query68	2459	833	557	557
query69	417	296	264	264
query70	1171	1147	1057	1057
query71	338	282	260	260
query72	6109	3821	3952	3821
query73	633	748	366	366
query74	9774	9159	9034	9034
query75	3166	3164	2665	2665
query76	1914	1162	790	790
query77	332	356	277	277
query78	10134	10130	9340	9340
query79	1031	882	612	612
query80	1760	543	440	440
query81	576	281	239	239
query82	385	154	122	122
query83	275	171	151	151
query84	286	89	78	78
query85	784	342	299	299
query86	424	315	309	309
query87	4450	4456	4371	4371
query88	4324	2188	2173	2173
query89	403	327	289	289
query90	1837	188	184	184
query91	143	140	105	105
query92	61	60	54	54
query93	1523	896	544	544
query94	690	386	309	309
query95	373	275	250	250
query96	484	611	276	276
query97	2768	2891	2789	2789
query98	222	214	198	198
query99	1319	1346	1292	1292
Total cold run time: 306281 ms
Total hot run time: 194698 ms

@doris-robot
Copy link

doris-robot commented Jan 15, 2025

ClickBench: Total hot run time: 31.02 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6d1a3677391bcbf553a61c5145fdd5f7b62d558d, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.25	0.07	0.07
query4	1.63	0.11	0.11
query5	0.44	0.43	0.40
query6	1.13	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.50	0.52
query10	0.56	0.56	0.55
query11	0.15	0.10	0.10
query12	0.14	0.10	0.11
query13	0.60	0.62	0.59
query14	2.82	2.93	2.88
query15	0.90	0.82	0.82
query16	0.38	0.38	0.37
query17	1.03	1.03	1.05
query18	0.22	0.22	0.21
query19	1.96	1.88	2.05
query20	0.02	0.01	0.01
query21	15.37	0.97	0.60
query22	0.74	0.88	0.63
query23	15.23	1.35	0.61
query24	3.44	0.99	1.37
query25	0.19	0.30	0.12
query26	0.33	0.16	0.14
query27	0.06	0.06	0.05
query28	13.88	0.96	0.42
query29	12.57	3.91	3.24
query30	0.26	0.09	0.07
query31	2.82	0.60	0.40
query32	3.22	0.55	0.46
query33	3.00	3.08	3.00
query34	16.65	5.20	4.56
query35	4.66	4.56	4.53
query36	0.65	0.51	0.49
query37	0.10	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.52 s
Total hot run time: 31.02 s

@xy720
Copy link
Member Author

xy720 commented Jan 16, 2025

run p0

@xy720 xy720 force-pushed the fix-dead-lock-tablet-stat branch from 6d1a367 to 3419dc5 Compare January 16, 2025 08:54
@xy720
Copy link
Member Author

xy720 commented Jan 16, 2025

run buildall

@xy720
Copy link
Member Author

xy720 commented Jan 16, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 16, 2025

TPC-H: Total hot run time: 32554 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 26da004c2c1efc1f310ce42c8dabfa6b51835867, data reload: false

------ Round 1 ----------------------------------
q1	17575	5472	5446	5446
q2	2040	304	167	167
q3	10417	1288	704	704
q4	10223	997	524	524
q5	7495	2413	2186	2186
q6	193	176	140	140
q7	927	768	619	619
q8	9257	1370	1177	1177
q9	5348	4999	4862	4862
q10	6846	2323	1904	1904
q11	477	261	249	249
q12	340	359	217	217
q13	17767	3662	3125	3125
q14	235	233	213	213
q15	511	487	469	469
q16	623	624	577	577
q17	564	869	331	331
q18	7014	6521	6531	6521
q19	1212	959	531	531
q20	312	322	191	191
q21	3033	2285	2089	2089
q22	362	340	312	312
Total cold run time: 102771 ms
Total hot run time: 32554 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5509	5512	5521	5512
q2	242	330	242	242
q3	2281	2624	2362	2362
q4	1428	1837	1397	1397
q5	4331	4758	4625	4625
q6	167	157	125	125
q7	2083	1984	1898	1898
q8	2636	2839	2660	2660
q9	7364	7380	7233	7233
q10	3026	3274	2810	2810
q11	578	507	505	505
q12	698	785	643	643
q13	3453	3863	3244	3244
q14	278	293	273	273
q15	514	472	471	471
q16	655	671	654	654
q17	1226	1729	1262	1262
q18	7666	7455	7344	7344
q19	790	929	1148	929
q20	1965	2075	1898	1898
q21	5793	5440	5140	5140
q22	584	630	603	603
Total cold run time: 53267 ms
Total hot run time: 51830 ms

@doris-robot
Copy link

doris-robot commented Jan 16, 2025

TPC-DS: Total hot run time: 186986 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 26da004c2c1efc1f310ce42c8dabfa6b51835867, data reload: false

query1	986	391	403	391
query2	6536	2070	2023	2023
query3	6791	213	220	213
query4	33248	23294	23108	23108
query5	4307	608	462	462
query6	285	201	193	193
query7	4595	496	306	306
query8	303	241	218	218
query9	9685	2675	2669	2669
query10	475	320	257	257
query11	18054	15181	14842	14842
query12	158	106	106	106
query13	1704	525	400	400
query14	9199	7266	6503	6503
query15	272	199	190	190
query16	7890	641	456	456
query17	1587	694	552	552
query18	2053	387	298	298
query19	219	176	178	176
query20	117	116	107	107
query21	210	121	102	102
query22	4504	4592	4228	4228
query23	34559	32927	33061	32927
query24	6483	2213	2261	2213
query25	487	449	374	374
query26	1231	270	151	151
query27	2395	443	326	326
query28	5395	2425	2413	2413
query29	696	526	407	407
query30	231	188	160	160
query31	948	902	774	774
query32	81	61	58	58
query33	512	351	291	291
query34	762	854	496	496
query35	780	801	721	721
query36	1020	1022	972	972
query37	122	100	77	77
query38	4289	4220	4121	4121
query39	1431	1439	1400	1400
query40	207	111	100	100
query41	62	51	48	48
query42	117	103	103	103
query43	516	520	489	489
query44	1316	832	844	832
query45	174	172	162	162
query46	854	1026	636	636
query47	1780	1836	1797	1797
query48	388	415	324	324
query49	775	478	417	417
query50	626	650	387	387
query51	6843	6843	6815	6815
query52	105	101	91	91
query53	220	245	183	183
query54	480	486	422	422
query55	80	80	82	80
query56	267	256	255	255
query57	1173	1108	1077	1077
query58	287	227	235	227
query59	3086	3246	3030	3030
query60	280	260	258	258
query61	114	113	112	112
query62	803	715	631	631
query63	223	185	189	185
query64	4409	976	642	642
query65	3287	3122	3142	3122
query66	1012	416	312	312
query67	15845	15704	15440	15440
query68	5061	818	536	536
query69	466	296	258	258
query70	1199	1057	1127	1057
query71	386	285	259	259
query72	5853	3867	3827	3827
query73	659	737	359	359
query74	9957	9190	8764	8764
query75	3167	3146	2662	2662
query76	3161	1156	778	778
query77	468	350	284	284
query78	9983	10035	9370	9370
query79	2918	822	601	601
query80	704	532	428	428
query81	528	268	239	239
query82	443	155	131	131
query83	180	169	155	155
query84	243	103	70	70
query85	782	344	295	295
query86	407	305	347	305
query87	4444	4653	4368	4368
query88	4814	2208	2154	2154
query89	404	325	293	293
query90	1849	193	186	186
query91	141	139	107	107
query92	70	58	56	56
query93	2313	894	531	531
query94	690	420	298	298
query95	337	266	262	262
query96	495	627	280	280
query97	2797	2931	2743	2743
query98	226	201	198	198
query99	1271	1385	1274	1274
Total cold run time: 286345 ms
Total hot run time: 186986 ms

@doris-robot
Copy link

doris-robot commented Jan 16, 2025

ClickBench: Total hot run time: 30.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 26da004c2c1efc1f310ce42c8dabfa6b51835867, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.23	0.07	0.07
query4	1.61	0.10	0.10
query5	0.41	0.40	0.41
query6	1.17	0.64	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.49	0.51
query10	0.56	0.56	0.54
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.60	0.61	0.58
query14	2.75	2.73	2.73
query15	0.91	0.83	0.82
query16	0.37	0.37	0.38
query17	0.95	1.07	1.06
query18	0.24	0.20	0.21
query19	1.92	1.86	1.92
query20	0.01	0.02	0.01
query21	15.37	0.96	0.61
query22	0.76	0.76	0.69
query23	15.27	1.41	0.52
query24	3.17	1.50	0.72
query25	0.15	0.20	0.11
query26	0.29	0.15	0.14
query27	0.05	0.05	0.05
query28	13.96	0.98	0.42
query29	12.54	3.96	3.28
query30	0.24	0.08	0.06
query31	2.83	0.59	0.39
query32	3.23	0.55	0.46
query33	2.99	2.98	2.99
query34	16.62	5.25	4.57
query35	4.56	4.55	4.53
query36	0.82	0.50	0.49
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.11 s
Total hot run time: 30.51 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 17, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jan 17, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 17, 2025

PR approved by anyone and no changes requested.

@xy720
Copy link
Member Author

xy720 commented Jan 17, 2025

cc @swjtu-zhanglei @gavinchou

@xy720 xy720 merged commit f9cb405 into apache:master Jan 20, 2025
25 of 26 checks passed
@xy720 xy720 mentioned this pull request Jan 20, 2025
Merged
16 tasks
github-actions bot pushed a commit that referenced this pull request Jan 24, 2025
@xy720
[Bug](dead lock) Fix dead lock in Tablet Stat Mgr (#46959)
491c6a5 
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #28608

Problem Summary:

In TabletStatMgr, We use stream().parallel() or parallelStream() in a
ForkJoinTask，when they are called, the stream will allocate the
`ForEach` task to multiple threads. However, when the stream is within a
ForkJoinTask, it will attempt to steal threads from the ForkJoinPool.
When the number of threads in the ForkJoinPool is small, thread
competition is very likely to occur, ultimately leading to a deadlock.

This commit will abandon ForkJoinPool and use a regular thread pool
instead.
@github-actions github-actions bot mentioned this pull request Jan 24, 2025
Merged
dataroaring pushed a commit that referenced this pull request Jan 26, 2025
@github-actions @xy720
branch-3.0: [Bug](dead lock) Fix dead lock in Tablet Stat Mgr #46959 (#…
cc38fad 
…47418)

Cherry-picked from #46959

Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com>
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@xy720 @lzyy2024
[Bug](dead lock) Fix dead lock in Tablet Stat Mgr (apache#46959)
710dbb8 
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: apache#28608

Problem Summary:

In TabletStatMgr, We use stream().parallel() or parallelStream() in a
ForkJoinTask，when they are called, the stream will allocate the
`ForEach` task to multiple threads. However, when the stream is within a
ForkJoinTask, it will attempt to steal threads from the ForkJoinPool.
When the number of threads in the ForkJoinPool is small, thread
competition is very likely to occur, ultimately leading to a deadlock.

This commit will abandon ForkJoinPool and use a regular thread pool
instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou left review comments

@cambyzju cambyzju cambyzju approved these changes

@liutang123 liutang123 liutang123 approved these changes

@dataroaring dataroaring Awaiting requested review from dataroaring

+1 more reviewer

@swjtu-zhanglei swjtu-zhanglei swjtu-zhanglei left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.4-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

9 participants

@xy720 @Thearas @doris-robot @gavinchou @cambyzju @liutang123 @swjtu-zhanglei @wm1581066 @dataroaring