Skip to content

[Log](topn) Log query id in topn two read and change the timeout to uery timeout #52717

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jul 23, 2025
Merged

[Log](topn) Log query id in topn two read and change the timeout to uery timeout #52717

dataroaring merged 1 commit into from
Jul 23, 2025

Conversation

@HappenLee
Copy link
Contributor

@HappenLee HappenLee commented Jul 3, 2025 •
edited
Loading

cherry pick #53450

@HappenLee HappenLee requested a review from dataroaring as a code owner July 3, 2025 07:45
@Thearas
Copy link
Contributor

Thearas commented Jul 3, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@HappenLee
Copy link
Contributor Author

HappenLee commented Jul 3, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 3, 2025

TPC-H: Total hot run time: 39429 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a42fb0adaf380dbb40b38732e32a81873fb09523, data reload: false

------ Round 1 ----------------------------------
q1	17718	7106	6567	6567
q2	2074	177	184	177
q3	10891	1085	1156	1085
q4	10477	747	694	694
q5	7759	2821	2739	2739
q6	211	131	129	129
q7	972	600	606	600
q8	9355	1899	1996	1899
q9	6693	6386	6406	6386
q10	7048	2238	2282	2238
q11	483	270	244	244
q12	387	206	206	206
q13	17830	2989	2962	2962
q14	228	203	206	203
q15	507	459	453	453
q16	491	392	377	377
q17	973	539	550	539
q18	7214	6714	6705	6705
q19	1438	1053	1083	1053
q20	492	195	197	195
q21	3863	2987	3071	2987
q22	1147	1007	991	991
Total cold run time: 108251 ms
Total hot run time: 39429 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6605	6572	6543	6543
q2	330	237	229	229
q3	2862	2765	2803	2765
q4	2026	1779	1782	1779
q5	5742	5718	5701	5701
q6	207	128	131	128
q7	2160	1802	1740	1740
q8	3377	3523	3534	3523
q9	8876	8709	8945	8709
q10	3565	3535	3495	3495
q11	603	495	508	495
q12	827	567	569	567
q13	9593	3098	3121	3098
q14	314	300	300	300
q15	521	457	456	456
q16	491	423	448	423
q17	1812	1611	1606	1606
q18	8101	7848	7678	7678
q19	1680	1481	1388	1388
q20	2101	1910	1809	1809
q21	4925	4978	4973	4973
q22	1138	1054	1052	1052
Total cold run time: 67856 ms
Total hot run time: 58457 ms

@doris-robot
Copy link

doris-robot commented Jul 3, 2025

TPC-DS: Total hot run time: 196429 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a42fb0adaf380dbb40b38732e32a81873fb09523, data reload: false

query1	1285	914	897	897
query2	6403	1898	1864	1864
query3	10938	4451	4373	4373
query4	62036	29378	23593	23593
query5	5149	459	452	452
query6	399	195	175	175
query7	5455	312	308	308
query8	316	229	227	227
query9	8446	2557	2559	2557
query10	452	263	265	263
query11	17565	15164	15706	15164
query12	164	100	109	100
query13	1441	458	445	445
query14	9748	6641	7436	6641
query15	206	181	176	176
query16	7134	494	513	494
query17	1148	620	597	597
query18	1947	315	316	315
query19	208	166	159	159
query20	121	117	107	107
query21	203	105	103	103
query22	4741	4451	4685	4451
query23	34797	34545	34442	34442
query24	6164	2918	2889	2889
query25	550	427	414	414
query26	654	170	172	170
query27	1829	365	362	362
query28	4042	2184	2113	2113
query29	735	491	429	429
query30	231	152	167	152
query31	980	814	805	805
query32	64	57	59	57
query33	433	314	293	293
query34	949	519	522	519
query35	869	736	738	736
query36	1081	937	952	937
query37	107	68	70	68
query38	3980	3986	3990	3986
query39	1534	1483	1487	1483
query40	205	96	95	95
query41	49	50	50	50
query42	110	97	95	95
query43	505	478	500	478
query44	1211	818	820	818
query45	190	172	176	172
query46	1146	709	719	709
query47	2050	1899	1923	1899
query48	482	378	393	378
query49	707	383	382	382
query50	846	424	414	414
query51	7360	7266	7319	7266
query52	98	92	90	90
query53	260	177	190	177
query54	550	446	471	446
query55	75	75	76	75
query56	249	267	255	255
query57	1308	1218	1224	1218
query58	233	218	217	217
query59	3216	3086	2951	2951
query60	275	253	260	253
query61	136	124	105	105
query62	757	656	668	656
query63	217	188	184	184
query64	1371	634	611	611
query65	3295	3153	3205	3153
query66	689	297	295	295
query67	15808	15530	15532	15530
query68	4276	568	562	562
query69	408	258	258	258
query70	1162	1063	1157	1063
query71	350	252	245	245
query72	6413	3956	4019	3956
query73	745	344	364	344
query74	10483	8970	9341	8970
query75	3366	2615	2631	2615
query76	1986	1028	1097	1028
query77	488	260	278	260
query78	10574	9639	9555	9555
query79	2044	599	586	586
query80	1358	426	420	420
query81	524	216	218	216
query82	1271	91	84	84
query83	275	142	165	142
query84	283	84	72	72
query85	1029	307	290	290
query86	394	300	296	296
query87	4487	4250	4199	4199
query88	3644	2362	2325	2325
query89	417	289	295	289
query90	1970	185	188	185
query91	184	146	146	146
query92	64	49	53	49
query93	2736	539	543	539
query94	786	285	303	285
query95	353	254	257	254
query96	615	282	277	277
query97	3320	3168	3164	3164
query98	216	199	196	196
query99	1588	1293	1271	1271
Total cold run time: 315711 ms
Total hot run time: 196429 ms

@doris-robot
Copy link

doris-robot commented Jul 3, 2025

ClickBench: Total hot run time: 29.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a42fb0adaf380dbb40b38732e32a81873fb09523, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.63	0.10	0.11
query5	0.53	0.50	0.49
query6	1.15	0.73	0.72
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.57	0.50	0.49
query10	0.55	0.56	0.55
query11	0.13	0.10	0.10
query12	0.14	0.14	0.12
query13	0.62	0.61	0.60
query14	0.77	0.78	0.80
query15	0.84	0.82	0.82
query16	0.37	0.39	0.39
query17	1.08	1.04	0.96
query18	0.22	0.20	0.21
query19	1.86	1.88	1.85
query20	0.02	0.01	0.00
query21	15.40	0.59	0.57
query22	2.59	2.51	1.46
query23	17.27	0.90	0.87
query24	3.11	0.35	1.82
query25	0.29	0.11	0.06
query26	0.44	0.13	0.13
query27	0.04	0.05	0.04
query28	10.40	0.50	0.44
query29	12.57	3.23	3.16
query30	0.25	0.07	0.06
query31	2.85	0.38	0.39
query32	3.24	0.45	0.46
query33	2.98	2.96	3.01
query34	17.14	4.47	4.51
query35	4.60	4.52	4.64
query36	0.67	0.47	0.47
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.02	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.16 s
Total hot run time: 29.01 s

@doris-robot
Copy link

doris-robot commented Jul 3, 2025

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.23% (10942/26538)
Line Coverage 32.02% (93693/292629)
Region Coverage 31.11% (48312/155293)
Branch Coverage 27.56% (24740/89778)

gavinchou
gavinchou previously approved these changes Jul 5, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 5, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jul 5, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 5, 2025

PR approved by anyone and no changes requested.

@gavinchou gavinchou changed the title [compile](arm) Add ARM_MARCH to specify the ARM architecture instruction set [compile](arm) Add ARM_MARCH to specify the ARM architecture instruction set #52678 Jul 5, 2025
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 22, 2025
@HappenLee HappenLee changed the title [compile](arm) Add ARM_MARCH to specify the ARM architecture instruction set #52678 [Log](topn) Log query id in topn two read and change the timeout to uery timeout Jul 22, 2025
@HappenLee
Copy link
Contributor Author

HappenLee commented Jul 22, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 22, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.89% (11135/26580)
Line Coverage 32.42% (95353/294129)
Region Coverage 31.56% (49257/156069)
Branch Coverage 27.98% (25237/90186)

@doris-robot
Copy link

doris-robot commented Jul 22, 2025

TPC-H: Total hot run time: 40081 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4890d238a02607e32b0b8422b3562809794e5171, data reload: false

------ Round 1 ----------------------------------
q1	17907	6970	6702	6702
q2	2082	165	155	155
q3	10686	1176	1187	1176
q4	10451	767	748	748
q5	7742	3336	2819	2819
q6	223	134	132	132
q7	1006	613	612	612
q8	9362	1962	2015	1962
q9	6759	6418	6409	6409
q10	6989	2327	2322	2322
q11	450	264	261	261
q12	405	214	217	214
q13	17798	2972	3003	2972
q14	231	210	209	209
q15	509	475	470	470
q16	479	405	390	390
q17	978	602	542	542
q18	7348	6803	6691	6691
q19	1429	1112	1013	1013
q20	492	200	209	200
q21	3882	3143	3118	3118
q22	1071	995	964	964
Total cold run time: 108279 ms
Total hot run time: 40081 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6627	6611	6583	6583
q2	330	234	234	234
q3	3028	2993	2947	2947
q4	2089	1830	1877	1830
q5	5767	5755	5794	5755
q6	210	133	136	133
q7	2229	1853	1833	1833
q8	3426	3612	3561	3561
q9	8880	9022	8953	8953
q10	3553	3567	3529	3529
q11	597	506	492	492
q12	823	587	594	587
q13	7440	3183	3172	3172
q14	303	269	265	265
q15	512	478	461	461
q16	483	454	430	430
q17	1858	1642	1620	1620
q18	8303	7814	7734	7734
q19	1660	1564	1520	1520
q20	2100	1890	1895	1890
q21	5237	5092	5121	5092
q22	1113	1060	1018	1018
Total cold run time: 66568 ms
Total hot run time: 59639 ms

@doris-robot
Copy link

doris-robot commented Jul 22, 2025

TPC-DS: Total hot run time: 197177 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4890d238a02607e32b0b8422b3562809794e5171, data reload: false

query1	1282	898	898	898
query2	6254	1941	1978	1941
query3	10974	4447	4601	4447
query4	33291	23451	23875	23451
query5	4340	456	431	431
query6	301	192	193	192
query7	4009	315	327	315
query8	286	233	248	233
query9	9667	2596	2593	2593
query10	495	271	257	257
query11	17839	15570	15504	15504
query12	147	101	99	99
query13	1541	429	414	414
query14	9237	6926	6675	6675
query15	264	177	179	177
query16	7921	481	506	481
query17	1630	616	601	601
query18	2171	323	323	323
query19	321	166	174	166
query20	128	119	114	114
query21	209	111	116	111
query22	4670	4447	4314	4314
query23	35105	34180	34180	34180
query24	11029	2977	2896	2896
query25	630	431	439	431
query26	743	172	173	172
query27	2206	378	369	369
query28	6446	2169	2170	2169
query29	812	468	463	463
query30	255	162	165	162
query31	1053	828	831	828
query32	93	58	62	58
query33	753	293	301	293
query34	934	531	527	527
query35	894	737	752	737
query36	1129	979	979	979
query37	110	66	76	66
query38	4119	3996	4022	3996
query39	1513	1523	1449	1449
query40	194	107	104	104
query41	48	47	50	47
query42	134	107	105	105
query43	519	502	494	494
query44	1272	837	829	829
query45	192	164	171	164
query46	1174	727	742	727
query47	2029	1890	1908	1890
query48	471	389	400	389
query49	918	392	402	392
query50	849	455	436	436
query51	7474	7330	7383	7330
query52	102	90	88	88
query53	257	186	184	184
query54	1181	480	481	480
query55	75	83	79	79
query56	268	263	250	250
query57	1322	1192	1173	1173
query58	224	226	225	225
query59	3405	3021	3144	3021
query60	281	251	256	251
query61	155	115	111	111
query62	844	691	696	691
query63	230	199	196	196
query64	5455	692	665	665
query65	3358	3253	3288	3253
query66	1239	304	298	298
query67	16101	15568	15503	15503
query68	4583	605	594	594
query69	437	261	268	261
query70	1195	1152	1075	1075
query71	352	261	251	251
query72	6236	4053	4047	4047
query73	766	340	352	340
query74	10369	8980	9292	8980
query75	3359	2613	2678	2613
query76	2414	1068	1038	1038
query77	393	276	280	276
query78	10560	9583	9497	9497
query79	1720	600	597	597
query80	1066	415	428	415
query81	550	220	219	219
query82	915	91	89	89
query83	245	167	142	142
query84	234	78	80	78
query85	1242	309	291	291
query86	409	293	294	293
query87	4389	4265	4204	4204
query88	3762	2433	2395	2395
query89	415	293	297	293
query90	1942	184	199	184
query91	184	144	149	144
query92	65	52	47	47
query93	2331	555	554	554
query94	828	284	298	284
query95	369	259	258	258
query96	623	276	275	275
query97	3295	3073	3146	3073
query98	226	198	203	198
query99	1488	1291	1285	1285
Total cold run time: 301534 ms
Total hot run time: 197177 ms

@doris-robot
Copy link

doris-robot commented Jul 22, 2025

ClickBench: Total hot run time: 29.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4890d238a02607e32b0b8422b3562809794e5171, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.11
query5	0.50	0.50	0.52
query6	1.12	0.74	0.74
query7	0.02	0.01	0.02
query8	0.05	0.03	0.04
query9	0.57	0.50	0.49
query10	0.55	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.60
query14	0.76	0.80	0.78
query15	0.84	0.83	0.83
query16	0.37	0.37	0.41
query17	0.99	1.04	1.05
query18	0.24	0.22	0.22
query19	1.88	1.78	1.76
query20	0.01	0.01	0.01
query21	15.39	0.58	0.58
query22	2.49	2.06	1.79
query23	17.01	0.81	0.69
query24	2.95	0.56	1.79
query25	0.17	0.23	0.14
query26	0.28	0.15	0.13
query27	0.05	0.04	0.04
query28	10.22	0.53	0.46
query29	12.60	3.21	3.22
query30	0.25	0.06	0.05
query31	2.86	0.38	0.39
query32	3.23	0.46	0.46
query33	2.94	2.99	3.01
query34	17.09	4.52	4.52
query35	4.61	4.55	4.52
query36	0.64	0.47	0.47
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.03
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.02	0.03
Total cold run time: 103.99 s
Total hot run time: 29.65 s

dataroaring
dataroaring approved these changes Jul 23, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit e66150e into apache:branch-3.0 Jul 23, 2025
25 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou gavinchou left review comments

Assignees

No one assigned

Labels

reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@HappenLee @Thearas @doris-robot @hello-stephen @gavinchou @dataroaring