Skip to content

[opt](mtmv) optimize mtmv rewrite performance #49514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 15 commits into from
May 16, 2025
Merged

[opt](mtmv) optimize mtmv rewrite performance #49514

morrySnow merged 15 commits into from
May 16, 2025

Conversation

@seawinde
Copy link
Contributor

@seawinde seawinde commented Mar 26, 2025 •
edited
Loading

What problem does this PR solve?

  • Obtaining available materialized view adjustments after partition pruning RBO rewrite rules. This reduces partition version comparison overhead for partitioned materializations, thereby improving performance.
  • Set maximum time threshold for transparent query rewriting with default value of 1000ms. Rewrite attempts will be terminated if exceeding this threshold.
    set materialized_view_rewrite_duration_threshold_ms = 1000
  • Optimize code structure in transparent rewriting framework:
    1. Replace Lambda expressions with conventional for-loops
    2. Remove redundant member variables
  • Cache available partitions of materialized views. Since calculating available partitions for materialized views is time-consuming, caching can significantly enhance performance.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@seawinde
Copy link
Contributor Author

seawinde commented Mar 26, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-H: Total hot run time: 34130 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7c83462441c26e0ba56a0615120574c2bb0457d5, data reload: false

------ Round 1 ----------------------------------
q1	26033	5113	5077	5077
q2	2067	291	162	162
q3	10439	1309	699	699
q4	10223	1020	515	515
q5	7543	2404	2347	2347
q6	197	170	133	133
q7	931	741	603	603
q8	9450	1317	1138	1138
q9	7047	5177	5148	5148
q10	6863	2318	1902	1902
q11	485	280	254	254
q12	356	358	221	221
q13	17783	3673	3084	3084
q14	226	234	214	214
q15	553	481	481	481
q16	631	626	605	605
q17	596	864	357	357
q18	7668	7344	7067	7067
q19	1947	975	582	582
q20	324	339	195	195
q21	3841	2588	2368	2368
q22	1108	1021	978	978
Total cold run time: 116311 ms
Total hot run time: 34130 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5311	5228	5185	5185
q2	239	324	234	234
q3	2157	2646	2296	2296
q4	1409	1827	1472	1472
q5	4482	4478	4406	4406
q6	221	173	128	128
q7	1983	1930	1769	1769
q8	2617	2630	2568	2568
q9	7326	7140	7144	7140
q10	3005	3174	2779	2779
q11	578	509	510	509
q12	693	760	596	596
q13	3546	3964	3365	3365
q14	285	288	284	284
q15	503	482	476	476
q16	656	698	667	667
q17	1170	1589	1397	1397
q18	7663	7572	7425	7425
q19	883	864	892	864
q20	1907	2021	1849	1849
q21	5477	4817	4902	4817
q22	1065	1053	1031	1031
Total cold run time: 53176 ms
Total hot run time: 51257 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-DS: Total hot run time: 193510 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7c83462441c26e0ba56a0615120574c2bb0457d5, data reload: false

query1	1405	1073	1042	1042
query2	6315	1929	1926	1926
query3	11174	4528	4802	4528
query4	25813	23783	23231	23231
query5	4157	673	493	493
query6	298	202	197	197
query7	3985	499	271	271
query8	304	257	232	232
query9	8487	2622	2599	2599
query10	480	310	260	260
query11	15418	15013	15011	15011
query12	162	107	108	107
query13	1561	533	381	381
query14	8860	6220	6199	6199
query15	209	181	166	166
query16	7175	609	499	499
query17	1185	689	568	568
query18	2002	423	297	297
query19	199	198	159	159
query20	129	123	119	119
query21	206	121	111	111
query22	4521	4423	4562	4423
query23	34274	33687	33553	33553
query24	8678	2445	2441	2441
query25	533	451	400	400
query26	1200	278	145	145
query27	3140	511	324	324
query28	4640	2436	2437	2436
query29	760	596	433	433
query30	274	218	190	190
query31	940	893	771	771
query32	75	69	62	62
query33	558	377	315	315
query34	879	889	502	502
query35	858	840	783	783
query36	962	1015	940	940
query37	127	104	80	80
query38	4293	4285	4262	4262
query39	1504	1482	1459	1459
query40	204	124	105	105
query41	100	52	53	52
query42	120	105	111	105
query43	510	525	483	483
query44	1398	817	847	817
query45	184	171	169	169
query46	902	1069	648	648
query47	1828	1861	1801	1801
query48	376	434	301	301
query49	765	547	409	409
query50	741	764	428	428
query51	4367	4307	4321	4307
query52	108	107	99	99
query53	231	267	181	181
query54	517	530	448	448
query55	94	85	86	85
query56	308	305	303	303
query57	1212	1202	1145	1145
query58	263	275	236	236
query59	2683	2891	2785	2785
query60	299	295	281	281
query61	127	130	127	127
query62	779	746	686	686
query63	223	183	184	183
query64	4319	1059	691	691
query65	4463	4321	4375	4321
query66	1091	410	309	309
query67	16539	15750	15594	15594
query68	8300	896	506	506
query69	486	300	257	257
query70	1170	1116	1116	1116
query71	449	304	281	281
query72	5253	4621	4648	4621
query73	661	568	355	355
query74	8977	9032	8663	8663
query75	3887	3234	2715	2715
query76	3628	1205	783	783
query77	800	378	281	281
query78	9887	10145	9370	9370
query79	1212	812	572	572
query80	610	526	449	449
query81	465	256	222	222
query82	193	125	90	90
query83	173	179	157	157
query84	243	93	71	71
query85	733	362	310	310
query86	326	312	271	271
query87	4371	4589	4482	4482
query88	2878	2224	2242	2224
query89	389	303	273	273
query90	2057	208	216	208
query91	142	143	114	114
query92	71	59	58	58
query93	1190	1052	581	581
query94	661	415	295	295
query95	349	314	262	262
query96	495	571	275	275
query97	3106	3264	3130	3130
query98	225	213	204	204
query99	1460	1400	1300	1300
Total cold run time: 277338 ms
Total hot run time: 193510 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

ClickBench: Total hot run time: 31.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7c83462441c26e0ba56a0615120574c2bb0457d5, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.10
query3	0.25	0.19	0.19
query4	1.59	0.19	0.19
query5	0.59	0.57	0.58
query6	1.18	0.71	0.71
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.53
query10	0.59	0.58	0.57
query11	0.16	0.10	0.11
query12	0.15	0.11	0.12
query13	0.61	0.61	0.60
query14	2.80	2.70	2.86
query15	0.90	0.86	0.85
query16	0.38	0.38	0.38
query17	1.06	1.01	1.04
query18	0.21	0.20	0.20
query19	1.90	1.94	1.79
query20	0.01	0.01	0.01
query21	15.37	0.90	0.57
query22	0.73	1.14	0.74
query23	14.91	1.36	0.67
query24	6.98	1.70	1.01
query25	0.46	0.19	0.16
query26	0.50	0.17	0.14
query27	0.05	0.05	0.05
query28	9.71	0.85	0.42
query29	12.56	4.07	3.41
query30	0.24	0.08	0.06
query31	2.82	0.59	0.39
query32	3.25	0.55	0.46
query33	3.03	3.10	3.05
query34	15.79	5.15	4.56
query35	4.55	4.51	4.56
query36	0.70	0.50	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.31 s
Total hot run time: 31.66 s

@seawinde
Copy link
Contributor Author

seawinde commented Mar 26, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-H: Total hot run time: 34475 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 30a86b8330ca6bf4228a1f6d415125e4ad75d7c4, data reload: false

------ Round 1 ----------------------------------
q1	26755	5089	5111	5089
q2	2087	308	164	164
q3	10384	1281	706	706
q4	10217	1001	527	527
q5	7543	2356	2412	2356
q6	186	166	135	135
q7	946	755	608	608
q8	9331	1301	1112	1112
q9	6927	5188	5144	5144
q10	6850	2323	1903	1903
q11	480	294	257	257
q12	343	355	216	216
q13	17753	3726	3122	3122
q14	235	233	221	221
q15	536	483	496	483
q16	622	629	597	597
q17	576	856	369	369
q18	7644	7318	7304	7304
q19	1231	967	562	562
q20	345	334	192	192
q21	3979	2625	2440	2440
q22	1071	1015	968	968
Total cold run time: 116041 ms
Total hot run time: 34475 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5162	5156	5138	5138
q2	242	334	225	225
q3	2172	2636	2249	2249
q4	1464	1817	1515	1515
q5	4539	4497	4404	4404
q6	214	171	133	133
q7	1949	1943	1775	1775
q8	2613	2610	2577	2577
q9	7307	7231	7181	7181
q10	2988	3175	2753	2753
q11	564	506	485	485
q12	730	754	600	600
q13	3456	3909	3335	3335
q14	302	293	288	288
q15	517	478	477	477
q16	650	685	633	633
q17	1155	1469	1385	1385
q18	7718	7551	7510	7510
q19	819	808	937	808
q20	2000	2050	1878	1878
q21	5184	4712	4694	4694
q22	1024	1021	1002	1002
Total cold run time: 52769 ms
Total hot run time: 51045 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-DS: Total hot run time: 187498 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 30a86b8330ca6bf4228a1f6d415125e4ad75d7c4, data reload: false

query1	1025	522	473	473
query2	6552	1902	1919	1902
query3	6798	224	215	215
query4	26647	23781	23157	23157
query5	4395	673	508	508
query6	305	211	210	210
query7	4612	501	286	286
query8	309	247	249	247
query9	8633	2633	2609	2609
query10	456	312	253	253
query11	15799	15404	14887	14887
query12	164	112	110	110
query13	1654	541	402	402
query14	9395	6518	6346	6346
query15	216	193	172	172
query16	7320	660	467	467
query17	1208	717	578	578
query18	2088	407	299	299
query19	193	190	163	163
query20	125	119	117	117
query21	210	120	101	101
query22	4200	4183	4027	4027
query23	34221	33266	33100	33100
query24	8375	2378	2392	2378
query25	532	484	408	408
query26	1230	271	153	153
query27	2731	498	328	328
query28	4321	2456	2412	2412
query29	754	575	447	447
query30	292	226	193	193
query31	964	890	788	788
query32	74	68	66	66
query33	566	380	320	320
query34	786	852	505	505
query35	825	806	749	749
query36	962	1000	904	904
query37	120	104	79	79
query38	4217	4149	4147	4147
query39	1489	1609	1437	1437
query40	215	142	103	103
query41	54	58	52	52
query42	120	109	102	102
query43	504	511	478	478
query44	1343	809	795	795
query45	182	171	171	171
query46	834	1027	637	637
query47	1769	1771	1720	1720
query48	399	417	303	303
query49	798	529	418	418
query50	680	722	420	420
query51	4212	4255	4145	4145
query52	111	105	104	104
query53	231	261	187	187
query54	498	495	445	445
query55	84	81	84	81
query56	301	278	268	268
query57	1142	1155	1066	1066
query58	244	244	253	244
query59	2547	2775	2724	2724
query60	315	281	264	264
query61	134	137	134	134
query62	808	745	667	667
query63	221	186	185	185
query64	4345	1010	753	753
query65	4344	4284	4296	4284
query66	1130	406	302	302
query67	15849	15554	15466	15466
query68	7962	878	507	507
query69	476	306	269	269
query70	1144	1190	1154	1154
query71	456	308	277	277
query72	5652	4832	4955	4832
query73	714	663	343	343
query74	9134	9396	8781	8781
query75	3984	3258	2750	2750
query76	3588	1187	774	774
query77	789	393	290	290
query78	10166	10186	9374	9374
query79	1952	812	572	572
query80	663	520	456	456
query81	575	258	220	220
query82	206	126	100	100
query83	179	174	156	156
query84	240	97	76	76
query85	756	358	312	312
query86	385	322	315	315
query87	4537	4463	4553	4463
query88	2972	2252	2316	2252
query89	388	315	282	282
query90	1929	212	220	212
query91	149	151	113	113
query92	81	63	58	58
query93	2378	1075	575	575
query94	633	428	299	299
query95	354	279	269	269
query96	499	564	281	281
query97	3156	3276	3153	3153
query98	228	211	202	202
query99	1365	1442	1320	1320
Total cold run time: 275867 ms
Total hot run time: 187498 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

ClickBench: Total hot run time: 31.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 30a86b8330ca6bf4228a1f6d415125e4ad75d7c4, data reload: false

query1	0.05	0.04	0.03
query2	0.12	0.10	0.10
query3	0.25	0.20	0.20
query4	1.64	0.19	0.19
query5	0.61	0.57	0.60
query6	1.20	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.58	0.54	0.52
query10	0.59	0.60	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.59
query14	2.67	2.79	2.69
query15	0.94	0.87	0.85
query16	0.40	0.38	0.38
query17	1.06	1.02	1.00
query18	0.22	0.20	0.20
query19	1.96	1.99	1.76
query20	0.01	0.01	0.01
query21	15.36	0.93	0.56
query22	0.76	1.19	0.75
query23	14.86	1.42	0.69
query24	6.81	1.34	1.03
query25	0.53	0.21	0.07
query26	0.56	0.16	0.13
query27	0.06	0.05	0.05
query28	9.84	0.87	0.43
query29	12.54	4.05	3.33
query30	0.26	0.09	0.07
query31	2.81	0.58	0.39
query32	3.22	0.55	0.48
query33	3.03	3.09	3.09
query34	15.69	5.07	4.48
query35	4.54	4.57	4.52
query36	0.66	0.51	0.49
query37	0.09	0.06	0.06
query38	0.05	0.05	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.12
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.29 s
Total hot run time: 31.52 s

@seawinde
Copy link
Contributor Author

seawinde commented Mar 26, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-H: Total hot run time: 34049 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 227b8ac4366322eb0087ca500b6cdc1c36d38ec8, data reload: false

------ Round 1 ----------------------------------
q1	26083	4997	4961	4961
q2	2061	285	174	174
q3	10392	1220	681	681
q4	10225	982	521	521
q5	7817	2411	2311	2311
q6	184	161	135	135
q7	918	737	623	623
q8	9324	1300	1091	1091
q9	6754	5128	5150	5128
q10	6823	2317	1882	1882
q11	465	273	255	255
q12	347	357	226	226
q13	17774	3673	3085	3085
q14	235	227	217	217
q15	550	489	479	479
q16	622	626	585	585
q17	579	842	344	344
q18	7523	7312	7203	7203
q19	1708	933	562	562
q20	310	316	186	186
q21	3843	3326	2418	2418
q22	1027	1017	982	982
Total cold run time: 115564 ms
Total hot run time: 34049 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5143	5222	5074	5074
q2	231	322	229	229
q3	2152	2678	2285	2285
q4	1412	1789	1420	1420
q5	4501	4426	4399	4399
q6	217	171	129	129
q7	1964	1920	1784	1784
q8	2573	2595	2559	2559
q9	7376	7263	7080	7080
q10	2963	3210	2762	2762
q11	572	498	501	498
q12	694	769	612	612
q13	3552	3908	3394	3394
q14	313	305	269	269
q15	527	476	468	468
q16	656	687	638	638
q17	1146	1554	1386	1386
q18	7686	7609	7442	7442
q19	824	792	854	792
q20	1924	1984	1848	1848
q21	5412	4775	4894	4775
q22	1089	1051	1057	1051
Total cold run time: 52927 ms
Total hot run time: 50894 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-DS: Total hot run time: 193511 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 227b8ac4366322eb0087ca500b6cdc1c36d38ec8, data reload: false

query1	1410	1065	1059	1059
query2	6129	1903	1874	1874
query3	10995	4639	4421	4421
query4	55400	25409	23420	23420
query5	5109	542	477	477
query6	358	211	193	193
query7	4921	487	276	276
query8	331	260	240	240
query9	5867	2525	2543	2525
query10	416	322	280	280
query11	15144	15088	14871	14871
query12	162	113	104	104
query13	1076	520	396	396
query14	10212	6218	6276	6218
query15	197	200	182	182
query16	7097	660	478	478
query17	1069	723	558	558
query18	1589	397	308	308
query19	196	195	173	173
query20	126	122	119	119
query21	208	140	113	113
query22	4514	4630	4362	4362
query23	33924	33714	33621	33621
query24	6616	2481	2414	2414
query25	461	469	401	401
query26	681	269	141	141
query27	2182	507	340	340
query28	3294	2445	2439	2439
query29	599	569	457	457
query30	279	234	202	202
query31	903	904	840	840
query32	80	68	67	67
query33	478	379	314	314
query34	754	855	525	525
query35	809	863	782	782
query36	943	1000	901	901
query37	124	107	79	79
query38	4205	4311	4370	4311
query39	1511	1475	1467	1467
query40	213	132	116	116
query41	60	96	56	56
query42	129	107	108	107
query43	553	499	489	489
query44	1283	808	815	808
query45	179	179	166	166
query46	865	1043	655	655
query47	1901	1918	1812	1812
query48	381	422	317	317
query49	693	507	428	428
query50	719	754	415	415
query51	4318	4292	4264	4264
query52	110	106	100	100
query53	232	278	187	187
query54	499	502	424	424
query55	79	80	83	80
query56	272	275	269	269
query57	1196	1203	1127	1127
query58	257	246	244	244
query59	2558	2890	2757	2757
query60	318	287	260	260
query61	160	135	128	128
query62	728	724	664	664
query63	221	191	183	183
query64	1489	1139	683	683
query65	4395	4413	4366	4366
query66	803	413	292	292
query67	16210	15836	15589	15589
query68	7282	880	501	501
query69	539	297	262	262
query70	1193	1115	1109	1109
query71	496	294	265	265
query72	5909	4788	4838	4788
query73	1442	684	352	352
query74	8964	9006	8822	8822
query75	3732	3230	2743	2743
query76	4205	1191	759	759
query77	602	373	274	274
query78	10141	10044	9312	9312
query79	4960	805	545	545
query80	821	514	440	440
query81	496	258	220	220
query82	549	128	100	100
query83	311	182	160	160
query84	293	103	81	81
query85	788	462	304	304
query86	404	312	276	276
query87	4593	4517	4390	4390
query88	3576	2219	2275	2219
query89	443	339	277	277
query90	1772	207	203	203
query91	141	140	111	111
query92	71	58	58	58
query93	3492	1056	576	576
query94	668	402	317	317
query95	349	272	262	262
query96	483	574	265	265
query97	3190	3220	3109	3109
query98	228	229	203	203
query99	1463	1380	1274	1274
Total cold run time: 304786 ms
Total hot run time: 193511 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

ClickBench: Total hot run time: 31.36 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 227b8ac4366322eb0087ca500b6cdc1c36d38ec8, data reload: false

query1	0.05	0.04	0.03
query2	0.12	0.11	0.10
query3	0.26	0.19	0.19
query4	1.60	0.18	0.19
query5	0.59	0.57	0.60
query6	1.20	0.73	0.71
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.57	0.52	0.52
query10	0.58	0.58	0.58
query11	0.15	0.12	0.11
query12	0.14	0.11	0.12
query13	0.62	0.60	0.60
query14	2.67	2.70	2.82
query15	0.91	0.86	0.85
query16	0.38	0.39	0.39
query17	1.02	1.06	1.04
query18	0.22	0.20	0.20
query19	1.88	1.91	1.93
query20	0.01	0.01	0.01
query21	15.36	0.90	0.53
query22	0.76	1.27	0.63
query23	14.89	1.35	0.62
query24	6.89	1.84	0.94
query25	0.50	0.09	0.23
query26	0.60	0.15	0.14
query27	0.05	0.05	0.04
query28	9.80	0.84	0.44
query29	12.53	4.02	3.35
query30	0.26	0.09	0.07
query31	2.81	0.60	0.38
query32	3.22	0.55	0.47
query33	3.08	3.06	3.14
query34	15.72	5.13	4.51
query35	4.52	4.47	4.53
query36	0.66	0.52	0.48
query37	0.09	0.06	0.07
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.17 s
Total hot run time: 31.36 s

@seawinde
Copy link
Contributor Author

seawinde commented Mar 27, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

TPC-H: Total hot run time: 34060 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 42685e1e81fe963e419f0f92460ae82184321af5, data reload: false

------ Round 1 ----------------------------------
q1	26246	5145	5024	5024
q2	2082	289	169	169
q3	10399	1250	694	694
q4	10230	1018	530	530
q5	7542	2373	2322	2322
q6	182	162	129	129
q7	898	743	602	602
q8	9306	1256	1079	1079
q9	6853	5138	5106	5106
q10	6873	2319	1901	1901
q11	476	279	257	257
q12	356	351	225	225
q13	17775	3650	3123	3123
q14	223	239	202	202
q15	535	474	486	474
q16	638	598	605	598
q17	569	859	328	328
q18	7620	7319	7206	7206
q19	1390	961	551	551
q20	305	338	189	189
q21	3848	2604	2354	2354
q22	1066	1033	997	997
Total cold run time: 115412 ms
Total hot run time: 34060 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5177	5156	5175	5156
q2	236	325	232	232
q3	2172	2639	2279	2279
q4	1397	1865	1396	1396
q5	4428	4397	4385	4385
q6	216	175	130	130
q7	2011	1905	1782	1782
q8	2592	2543	2443	2443
q9	7282	7220	7056	7056
q10	3019	3192	2731	2731
q11	588	503	503	503
q12	679	789	679	679
q13	3519	3930	3236	3236
q14	286	299	296	296
q15	540	515	486	486
q16	674	685	653	653
q17	1152	1567	1345	1345
q18	7728	7501	7425	7425
q19	812	813	856	813
q20	1984	2062	1865	1865
q21	5185	4862	4813	4813
q22	1074	1038	1019	1019
Total cold run time: 52751 ms
Total hot run time: 50723 ms

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

TPC-DS: Total hot run time: 193034 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 42685e1e81fe963e419f0f92460ae82184321af5, data reload: false

query1	1382	1072	1029	1029
query2	6440	1901	1860	1860
query3	11025	4606	4538	4538
query4	54476	25526	23727	23727
query5	5146	503	467	467
query6	351	196	181	181
query7	4906	488	265	265
query8	304	247	247	247
query9	5687	2544	2546	2544
query10	408	314	253	253
query11	15119	15033	14796	14796
query12	155	105	101	101
query13	1061	521	376	376
query14	10121	6232	6317	6232
query15	222	198	180	180
query16	7058	659	509	509
query17	1083	749	597	597
query18	1553	444	338	338
query19	200	198	166	166
query20	131	131	117	117
query21	207	123	119	119
query22	4404	4431	4364	4364
query23	34036	33309	33477	33309
query24	6619	2430	2431	2430
query25	492	499	457	457
query26	719	266	144	144
query27	2285	505	333	333
query28	3376	2468	2430	2430
query29	573	550	436	436
query30	269	214	193	193
query31	848	857	803	803
query32	76	63	72	63
query33	445	415	298	298
query34	756	866	502	502
query35	798	865	768	768
query36	1007	1009	908	908
query37	124	99	79	79
query38	4141	4163	4120	4120
query39	1489	1513	1470	1470
query40	209	118	105	105
query41	52	52	51	51
query42	124	108	111	108
query43	487	502	484	484
query44	1333	810	809	809
query45	183	177	169	169
query46	842	1041	666	666
query47	1829	1909	1809	1809
query48	391	429	318	318
query49	688	514	414	414
query50	709	739	431	431
query51	4281	4306	4188	4188
query52	106	112	100	100
query53	225	256	181	181
query54	503	503	431	431
query55	83	81	83	81
query56	322	284	281	281
query57	1175	1180	1138	1138
query58	252	259	232	232
query59	2760	2737	2697	2697
query60	296	279	270	270
query61	127	131	127	127
query62	695	781	670	670
query63	224	183	185	183
query64	2020	1025	699	699
query65	4467	4348	4418	4348
query66	778	393	299	299
query67	15893	15778	15463	15463
query68	7502	887	518	518
query69	545	297	272	272
query70	1205	1131	1109	1109
query71	492	292	256	256
query72	5986	4920	4852	4852
query73	1405	679	355	355
query74	9024	9020	8724	8724
query75	3777	3200	2693	2693
query76	4170	1328	750	750
query77	644	380	276	276
query78	9973	10280	9361	9361
query79	2287	802	571	571
query80	626	512	436	436
query81	487	252	224	224
query82	425	129	94	94
query83	314	171	159	159
query84	281	93	76	76
query85	813	427	333	333
query86	370	322	288	288
query87	4379	4465	4290	4290
query88	3133	2196	2189	2189
query89	387	316	288	288
query90	1949	202	206	202
query91	143	154	117	117
query92	72	57	56	56
query93	1161	1055	591	591
query94	704	418	306	306
query95	344	266	264	264
query96	482	561	266	266
query97	3127	3316	3141	3141
query98	221	201	203	201
query99	1438	1398	1275	1275
Total cold run time: 298412 ms
Total hot run time: 193034 ms

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

ClickBench: Total hot run time: 30.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 42685e1e81fe963e419f0f92460ae82184321af5, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.10	0.11
query3	0.24	0.19	0.19
query4	1.59	0.20	0.19
query5	0.60	0.56	0.57
query6	1.19	0.72	0.71
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.52	0.50
query10	0.59	0.59	0.57
query11	0.15	0.11	0.10
query12	0.15	0.11	0.11
query13	0.61	0.61	0.61
query14	2.82	2.79	2.85
query15	0.91	0.86	0.84
query16	0.38	0.37	0.38
query17	1.00	1.00	1.03
query18	0.21	0.19	0.20
query19	1.87	1.94	1.77
query20	0.01	0.01	0.02
query21	15.36	0.90	0.56
query22	0.74	1.18	0.60
query23	15.02	1.36	0.59
query24	6.80	1.66	0.65
query25	0.50	0.14	0.07
query26	0.58	0.15	0.12
query27	0.05	0.05	0.06
query28	9.67	0.87	0.41
query29	12.56	3.94	3.28
query30	0.25	0.09	0.06
query31	2.85	0.59	0.37
query32	3.22	0.56	0.46
query33	3.09	3.13	3.06
query34	15.76	5.11	4.49
query35	4.56	4.51	4.52
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.25 s
Total hot run time: 30.76 s

@seawinde seawinde force-pushed the mtmv_rewrite_performance_pick_to_master branch from 42685e1 to cc62e25 Compare March 27, 2025 12:49
@seawinde
Copy link
Contributor Author

seawinde commented Apr 1, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 1, 2025

TPC-H: Total hot run time: 33828 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cc62e258fde48b0353d882d8542143ddb2aaf5a5, data reload: false

------ Round 1 ----------------------------------
q1	26094	5079	5007	5007
q2	2072	286	167	167
q3	10497	1221	677	677
q4	10236	1039	531	531
q5	7853	2396	2343	2343
q6	191	161	130	130
q7	889	724	623	623
q8	9317	1283	1109	1109
q9	6797	5101	5115	5101
q10	6848	2291	1887	1887
q11	472	273	262	262
q12	353	356	212	212
q13	17770	3608	3027	3027
q14	219	232	213	213
q15	530	484	479	479
q16	621	609	599	599
q17	595	855	341	341
q18	7696	7214	7054	7054
q19	1568	971	572	572
q20	313	317	194	194
q21	3928	2588	2351	2351
q22	1064	991	949	949
Total cold run time: 115923 ms
Total hot run time: 33828 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5195	5126	5121	5121
q2	242	328	234	234
q3	2138	2650	2291	2291
q4	1436	1778	1365	1365
q5	4427	4406	4356	4356
q6	218	170	128	128
q7	1989	1922	1728	1728
q8	2625	2517	2555	2517
q9	7248	7232	6943	6943
q10	3052	3179	2725	2725
q11	573	498	484	484
q12	688	779	600	600
q13	3444	3852	3326	3326
q14	281	285	272	272
q15	538	485	477	477
q16	629	688	656	656
q17	1158	1540	1418	1418
q18	7877	7669	7588	7588
q19	810	841	905	841
q20	1956	1952	1842	1842
q21	5359	4887	4758	4758
q22	1053	1060	977	977
Total cold run time: 52936 ms
Total hot run time: 50647 ms

@doris-robot
Copy link

doris-robot commented Apr 1, 2025

TPC-DS: Total hot run time: 193132 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cc62e258fde48b0353d882d8542143ddb2aaf5a5, data reload: false

query1	1394	1059	1040	1040
query2	6200	1870	1874	1870
query3	11030	4424	4539	4424
query4	53427	25505	22994	22994
query5	5124	564	482	482
query6	332	192	193	192
query7	4869	494	276	276
query8	301	236	228	228
query9	5633	2596	2588	2588
query10	424	312	262	262
query11	15121	15114	14780	14780
query12	160	111	111	111
query13	1049	523	425	425
query14	10124	6343	6446	6343
query15	206	187	190	187
query16	7065	673	481	481
query17	1072	742	599	599
query18	1557	429	316	316
query19	202	200	174	174
query20	128	126	134	126
query21	210	127	119	119
query22	4339	4440	4431	4431
query23	34034	33293	33232	33232
query24	6569	2497	2509	2497
query25	472	456	400	400
query26	701	276	150	150
query27	2367	527	345	345
query28	3244	2464	2435	2435
query29	620	579	440	440
query30	271	218	189	189
query31	857	864	808	808
query32	75	64	65	64
query33	469	368	322	322
query34	765	855	521	521
query35	808	835	758	758
query36	991	988	898	898
query37	121	99	75	75
query38	4121	4369	4192	4192
query39	1493	1440	1442	1440
query40	216	124	105	105
query41	59	57	51	51
query42	136	118	113	113
query43	508	523	485	485
query44	1339	861	822	822
query45	182	185	175	175
query46	870	1043	661	661
query47	1832	1879	1837	1837
query48	401	423	344	344
query49	694	512	431	431
query50	728	754	432	432
query51	4283	4324	4288	4288
query52	113	113	112	112
query53	232	266	183	183
query54	513	511	431	431
query55	82	85	86	85
query56	288	298	282	282
query57	1160	1211	1166	1166
query58	258	253	233	233
query59	2642	2890	2662	2662
query60	297	307	285	285
query61	131	130	170	130
query62	740	741	685	685
query63	233	182	194	182
query64	1819	1076	723	723
query65	4416	4303	4249	4249
query66	723	394	296	296
query67	16113	15550	15644	15550
query68	8808	845	512	512
query69	528	299	265	265
query70	1163	1093	1126	1093
query71	499	299	274	274
query72	5959	4919	4953	4919
query73	1479	713	349	349
query74	9315	8884	8914	8884
query75	4308	3249	2744	2744
query76	4407	1191	748	748
query77	717	368	376	368
query78	10006	10145	9315	9315
query79	2955	808	560	560
query80	678	517	437	437
query81	478	268	215	215
query82	449	126	100	100
query83	275	179	169	169
query84	287	94	77	77
query85	786	356	396	356
query86	331	292	292	292
query87	4416	4435	4265	4265
query88	2981	2220	2261	2220
query89	398	302	283	283
query90	1965	207	205	205
query91	150	146	115	115
query92	78	63	57	57
query93	1694	1032	587	587
query94	689	419	321	321
query95	361	277	304	277
query96	488	560	273	273
query97	3206	3218	3131	3131
query98	232	199	209	199
query99	1399	1386	1310	1310
Total cold run time: 300438 ms
Total hot run time: 193132 ms

@doris-robot
Copy link

doris-robot commented Apr 1, 2025

ClickBench: Total hot run time: 30.6 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cc62e258fde48b0353d882d8542143ddb2aaf5a5, data reload: false

query1	0.04	0.04	0.03
query2	0.14	0.11	0.10
query3	0.24	0.19	0.20
query4	1.59	0.19	0.20
query5	0.59	0.58	0.60
query6	1.22	0.72	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.56	0.52	0.53
query10	0.58	0.58	0.56
query11	0.16	0.10	0.11
query12	0.15	0.11	0.11
query13	0.62	0.59	0.59
query14	2.67	2.69	2.80
query15	0.92	0.84	0.86
query16	0.39	0.38	0.37
query17	1.03	1.03	1.05
query18	0.22	0.20	0.23
query19	1.96	2.00	1.78
query20	0.02	0.01	0.01
query21	15.38	0.88	0.55
query22	0.75	1.42	0.68
query23	14.74	1.39	0.64
query24	7.06	1.64	0.33
query25	0.29	0.09	0.12
query26	0.68	0.16	0.15
query27	0.05	0.06	0.04
query28	8.79	0.86	0.42
query29	12.66	4.00	3.35
query30	0.25	0.08	0.06
query31	2.85	0.61	0.38
query32	3.23	0.56	0.47
query33	3.04	3.02	3.03
query34	15.78	5.11	4.48
query35	4.56	4.54	4.50
query36	0.66	0.49	0.48
query37	0.08	0.07	0.06
query38	0.06	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.42 s
Total hot run time: 30.6 s

@seawinde
Copy link
Contributor Author

seawinde commented Apr 1, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 1, 2025

TPC-H: Total hot run time: 35350 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit db6808aba751117d15f73cf389578eaa9af4bbc6, data reload: false

------ Round 1 ----------------------------------
q1	26310	5197	5108	5108
q2	2084	290	192	192
q3	10479	1258	732	732
q4	10256	1011	538	538
q5	8130	2395	2441	2395
q6	193	162	134	134
q7	916	775	617	617
q8	9333	1318	1087	1087
q9	6812	5181	5145	5145
q10	6936	2318	1931	1931
q11	489	289	276	276
q12	361	363	228	228
q13	17785	3682	3138	3138
q14	224	230	208	208
q15	538	480	486	480
q16	638	616	601	601
q17	627	860	408	408
q18	7544	7349	7045	7045
q19	1289	951	573	573
q20	350	361	240	240
q21	4524	3814	3297	3297
q22	1040	1027	977	977
Total cold run time: 116858 ms
Total hot run time: 35350 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5227	5173	5233	5173
q2	241	321	233	233
q3	2165	2643	2253	2253
q4	1549	1978	1481	1481
q5	4538	4406	4366	4366
q6	221	170	127	127
q7	2006	1883	1768	1768
q8	2620	2603	2562	2562
q9	7158	7165	7163	7163
q10	2970	3174	2760	2760
q11	573	505	488	488
q12	673	773	621	621
q13	3570	3884	3321	3321
q14	273	304	283	283
q15	520	485	480	480
q16	636	689	642	642
q17	1166	1512	1485	1485
q18	7869	7578	7540	7540
q19	825	820	817	817
q20	2037	2009	1846	1846
q21	5193	4799	4604	4604
q22	1053	1048	995	995
Total cold run time: 53083 ms
Total hot run time: 51008 ms

@doris-robot
Copy link

doris-robot commented Apr 1, 2025

TPC-DS: Total hot run time: 185492 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit db6808aba751117d15f73cf389578eaa9af4bbc6, data reload: false

query1	1035	477	467	467
query2	6580	1854	1853	1853
query3	6748	224	223	223
query4	26183	23216	23107	23107
query5	4340	623	444	444
query6	294	203	188	188
query7	4625	494	276	276
query8	296	232	228	228
query9	8617	2572	2556	2556
query10	486	341	261	261
query11	15338	15089	14823	14823
query12	171	113	108	108
query13	1644	504	400	400
query14	8983	6149	6101	6101
query15	217	182	167	167
query16	7169	660	487	487
query17	1193	734	545	545
query18	1962	401	309	309
query19	183	180	165	165
query20	123	117	110	110
query21	208	126	106	106
query22	4123	4265	4008	4008
query23	33966	33039	33015	33015
query24	8446	2365	2348	2348
query25	536	448	397	397
query26	1271	264	148	148
query27	2764	508	329	329
query28	4388	2407	2391	2391
query29	744	563	426	426
query30	288	220	193	193
query31	939	847	815	815
query32	78	66	64	64
query33	566	391	313	313
query34	831	843	507	507
query35	793	834	735	735
query36	956	1021	901	901
query37	117	97	77	77
query38	4362	4143	4085	4085
query39	1614	1370	1375	1370
query40	214	122	110	110
query41	58	52	52	52
query42	127	107	104	104
query43	486	496	468	468
query44	1288	821	795	795
query45	187	182	175	175
query46	827	1018	619	619
query47	1744	1780	1699	1699
query48	383	405	292	292
query49	773	525	432	432
query50	643	679	393	393
query51	4224	4204	4106	4106
query52	105	111	101	101
query53	232	249	179	179
query54	570	574	499	499
query55	94	85	92	85
query56	300	316	283	283
query57	1143	1142	1064	1064
query58	263	263	273	263
query59	2581	2737	2508	2508
query60	337	333	320	320
query61	157	149	150	149
query62	777	729	654	654
query63	226	186	192	186
query64	4416	1096	774	774
query65	4347	4272	4235	4235
query66	1151	456	328	328
query67	15896	15623	15105	15105
query68	7889	885	516	516
query69	470	302	285	285
query70	1221	1144	1141	1141
query71	461	346	300	300
query72	5719	4724	4880	4724
query73	707	623	351	351
query74	8968	9211	8645	8645
query75	3810	3209	2754	2754
query76	3692	1186	737	737
query77	791	385	301	301
query78	9992	10191	9178	9178
query79	2587	821	568	568
query80	653	502	459	459
query81	493	256	224	224
query82	464	130	102	102
query83	295	261	253	253
query84	302	106	85	85
query85	818	361	316	316
query86	385	285	277	277
query87	4407	4463	4363	4363
query88	3548	2205	2215	2205
query89	392	313	293	293
query90	1841	207	211	207
query91	143	139	115	115
query92	80	60	58	58
query93	1908	956	584	584
query94	670	385	302	302
query95	381	294	292	292
query96	478	554	274	274
query97	3208	3263	3122	3122
query98	226	207	200	200
query99	1457	1410	1237	1237
Total cold run time: 275182 ms
Total hot run time: 185492 ms

@doris-robot
Copy link

doris-robot commented Apr 1, 2025

ClickBench: Total hot run time: 31.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit db6808aba751117d15f73cf389578eaa9af4bbc6, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.10
query3	0.25	0.19	0.19
query4	1.59	0.20	0.19
query5	0.61	0.58	0.60
query6	1.21	0.71	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.51	0.53
query10	0.56	0.56	0.57
query11	0.16	0.12	0.11
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	2.79	2.72	2.82
query15	0.92	0.85	0.86
query16	0.40	0.38	0.38
query17	1.07	1.03	1.06
query18	0.23	0.21	0.20
query19	1.87	2.01	1.84
query20	0.02	0.01	0.01
query21	15.35	0.89	0.54
query22	0.73	1.12	0.64
query23	15.06	1.40	0.59
query24	6.98	1.29	1.32
query25	0.51	0.13	0.17
query26	0.54	0.15	0.14
query27	0.05	0.05	0.04
query28	10.45	0.90	0.42
query29	12.57	4.07	3.27
query30	0.25	0.08	0.06
query31	2.82	0.62	0.39
query32	3.22	0.57	0.46
query33	3.07	3.06	3.02
query34	15.69	5.18	4.48
query35	4.51	4.46	4.52
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.14	0.12
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.02
Total cold run time: 106.16 s
Total hot run time: 31.46 s

@seawinde
Copy link
Contributor Author

seawinde commented Apr 2, 2025

run buildall

@seawinde seawinde force-pushed the mtmv_rewrite_performance_pick_to_master branch from 789842a to cda01c2 Compare April 3, 2025 03:50
@seawinde
Copy link
Contributor Author

seawinde commented Apr 3, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 3, 2025

TPC-H: Total hot run time: 34411 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cda01c290ab91fa41aaf5e092f61dad38d14d978, data reload: false

------ Round 1 ----------------------------------
q1	26261	5099	5059	5059
q2	2079	275	189	189
q3	10500	1234	693	693
q4	10240	1002	544	544
q5	7705	2377	2374	2374
q6	189	164	133	133
q7	920	729	604	604
q8	9326	1308	1170	1170
q9	6831	5139	5106	5106
q10	6815	2324	1890	1890
q11	500	288	259	259
q12	347	355	217	217
q13	17764	3786	3159	3159
q14	238	219	215	215
q15	524	490	484	484
q16	659	639	582	582
q17	605	865	370	370
q18	8069	7241	7164	7164
q19	1735	983	561	561
q20	334	342	233	233
q21	4090	3382	2452	2452
q22	1056	998	953	953
Total cold run time: 116787 ms
Total hot run time: 34411 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5181	5061	5128	5061
q2	242	337	229	229
q3	2140	2662	2275	2275
q4	1423	1894	1474	1474
q5	4502	4390	4416	4390
q6	226	170	127	127
q7	2018	1987	1786	1786
q8	2661	2552	2459	2459
q9	7257	7134	7149	7134
q10	2970	3144	2778	2778
q11	587	517	490	490
q12	703	756	625	625
q13	3509	3979	3355	3355
q14	284	299	271	271
q15	536	488	478	478
q16	668	703	645	645
q17	1163	1537	1414	1414
q18	7866	7553	7365	7365
q19	817	813	865	813
q20	1951	2029	1868	1868
q21	5253	4941	4716	4716
q22	1062	1069	1035	1035
Total cold run time: 53019 ms
Total hot run time: 50788 ms

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@seawinde @zddr @koarz
[opt](mtmv) optimize mtmv rewrite performance (apache#49514)
d56dbbc 
### What problem does this PR solve?

- Obtaining available materialized view adjustments after partition
pruning RBO rewrite rules. This reduces partition version comparison
overhead for partitioned materializations, thereby improving
performance.

- Set maximum time threshold for transparent query rewriting with
default value of 1000ms. Rewrite attempts will be terminated if
exceeding this threshold.
set materialized_view_rewrite_duration_threshold_ms = 1000

- Optimize code structure in transparent rewriting framework:
    1. Replace Lambda expressions with conventional for-loops
    2. Remove redundant member variables

- Cache available partitions of materialized views. Since calculating
available partitions for materialized views is time-consuming,
caching can significantly enhance performance.


Co-authored-by: zhangdong <zhangdong@selectdb.com>
@seawinde seawinde mentioned this pull request Jun 13, 2025
Merged
16 tasks
dataroaring pushed a commit that referenced this pull request Jun 18, 2025
@seawinde @zddr
[opt](mtmv) optimize mtmv rewrite performance (#49514) (#51312)
0922e52 
### What problem does this PR solve?

commitId: 4da1c8a
pr: #49514


Co-authored-by: zhangdong <zhangdong@selectdb.com>
This was referenced Jun 24, 2025
Merged
Merged
seawinde added a commit to seawinde/doris that referenced this pull request Jun 24, 2025
@seawinde @zddr
[opt](mtmv) optimize mtmv rewrite performance (apache#49514)
42354be 
- Obtaining available materialized view adjustments after partition
pruning RBO rewrite rules. This reduces partition version comparison
overhead for partitioned materializations, thereby improving
performance.

- Set maximum time threshold for transparent query rewriting with
default value of 1000ms. Rewrite attempts will be terminated if
exceeding this threshold.
set materialized_view_rewrite_duration_threshold_ms = 1000

- Optimize code structure in transparent rewriting framework:
    1. Replace Lambda expressions with conventional for-loops
    2. Remove redundant member variables

- Cache available partitions of materialized views. Since calculating
available partitions for materialized views is time-consuming,
caching can significantly enhance performance.

Co-authored-by: zhangdong <zhangdong@selectdb.com>
@seawinde seawinde mentioned this pull request Jun 24, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jun 25, 2025
@seawinde @zddr
branch-3.1: [opt](mtmv) optimize mtmv rewrite performance #49514 (#52241
3fcde57 
)

Cherry-pick from #49514

---------

Co-authored-by: zhangdong <zhangdong@selectdb.com>
yiguolei pushed a commit that referenced this pull request Jun 25, 2025
@seawinde
[fix](mtmv) Fix compensate union all wrongly when partition mv use pa…
43b99dd 
…rtition external table (#52218)

Fix compensate union all wrongly when partition mv use partition
external table

in test mv/external_table/part_partition_invalid.groovy
order_qt_after_modify_data_and_refresh_catalog data should be as
following:
```
1	2	1	2023-10-17	2023-10-17
2	2	2	2023-10-18	2023-10-18
3	2	3	2023-10-19	2023-10-19
```


### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #49514 

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [x] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
dataroaring pushed a commit that referenced this pull request Jun 30, 2025
@seawinde
[fix](mtmv) Fix compensate union all wrongly when partition mv use pa…
e68e7cb 
…rtition external table (#52223)

Fix compensate union all wrongly when partition mv use partition
external table

in test mv/external_table/part_partition_invalid.groovy
order_qt_after_modify_data_and_refresh_catalog data should be as
following:
```
1	2	1	2023-10-17	2023-10-17
2	2	2	2023-10-18	2023-10-18
3	2	3	2023-10-19	2023-10-19
```

Related PR: #49514
koarz pushed a commit to koarz/doris that referenced this pull request Jul 3, 2025
@seawinde @koarz
[fix](mtmv) Fix compensate union all wrongly when partition mv use pa…
f9ecff3 
…rtition external table (apache#52223)

Fix compensate union all wrongly when partition mv use partition
external table

in test mv/external_table/part_partition_invalid.groovy
order_qt_after_modify_data_and_refresh_catalog data should be as
following:
```
1	2	1	2023-10-17	2023-10-17
2	2	2	2023-10-18	2023-10-18
3	2	3	2023-10-19	2023-10-19
```

Related PR: apache#49514
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
@seawinde seawinde mentioned this pull request Oct 31, 2025
Merged
16 tasks
morrySnow pushed a commit that referenced this pull request Nov 10, 2025
@seawinde
[fix](mtmv) Fix nested mv rewritten fail when bottom mv is partitioned (
54d3a63 
#57558)

Related PR: #49514 #56958 

Problem Summary:

In the scenario of nested materialized view rewriting, if the underlying
materialized view is a partitioned materialized view, the rewriting of
the upper-level materialized view would fail. This PR fixes the issue.
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Nov 18, 2025
@seawinde @wyxxxcat
[fix](mtmv) Fix nested mv rewritten fail when bottom mv is partitioned (
1b50dbf 
apache#57558)

Related PR: apache#49514 apache#56958 

Problem Summary:

In the scenario of nested materialized view rewriting, if the underlying
materialized view is a partitioned materialized view, the rewriting of
the upper-level materialized view would fail. This PR fixes the issue.
seawinde added a commit to seawinde/doris that referenced this pull request Nov 24, 2025
@seawinde
[fix](mtmv) Fix nested mv rewritten fail when bottom mv is partitioned (
5e43141 
apache#57558)

Related PR: apache#49514 apache#56958

Problem Summary:

In the scenario of nested materialized view rewriting, if the underlying
materialized view is a partitioned materialized view, the rewriting of
the upper-level materialized view would fail. This PR fixes the issue.
seawinde added a commit to seawinde/doris that referenced this pull request Nov 24, 2025
@seawinde
[fix](mtmv) Fix nested mv rewritten fail when bottom mv is partitioned (
7619a33 
apache#57558)

Related PR: apache#49514 apache#56958 

Problem Summary:

In the scenario of nested materialized view rewriting, if the underlying
materialized view is a partitioned materialized view, the rewriting of
the upper-level materialized view would fail. This PR fixes the issue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

+2 more reviewers

@zddr zddr zddr approved these changes

@zfr9527 zfr9527 zfr9527 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.7-merged dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@seawinde @hello-stephen @doris-robot @zddr @zfr9527 @morrySnow @yiguolei @dataroaring