Skip to content

branch-3.0: [fix](mtmv)before mtmv refresh,check column type if change #50768

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
May 19, 2025
Merged

branch-3.0: [fix](mtmv)before mtmv refresh,check column type if change #50768

dataroaring merged 1 commit into from
May 19, 2025

Conversation

@zddr
Copy link
Contributor

@zddr zddr commented May 9, 2025

pick: #49041

@zddr
[fix](mtmv)before mtmv refresh,check column type if change (apache#49041
7eb0ca1 
)

- before mtmv refresh,check column type if change
- fix tableMTMVs of fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelationManager.java is not correct after replace MTMV/table
- when MTMV is SCHEMA_CHANGE, will not refresh by commit
- add more case
@zddr zddr requested a review from dataroaring as a code owner May 9, 2025 12:01
@hello-stephen
Copy link
Contributor

hello-stephen commented May 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented May 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 9, 2025

TPC-H: Total hot run time: 40137 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7eb0ca142e26fdf03a5432a916c0788898272717, data reload: false

------ Round 1 ----------------------------------
q1	17571	6791	6615	6615
q2	2044	167	166	166
q3	10751	1070	1167	1070
q4	10543	728	648	648
q5	7754	2831	2834	2831
q6	224	137	135	135
q7	977	641	634	634
q8	9376	1955	2014	1955
q9	6624	6394	6453	6394
q10	7043	2248	2267	2248
q11	460	267	265	265
q12	404	222	217	217
q13	17804	2943	2983	2943
q14	238	207	209	207
q15	497	472	463	463
q16	668	585	585	585
q17	982	607	603	603
q18	7147	6695	6732	6695
q19	1452	1116	1033	1033
q20	473	203	201	201
q21	4366	3324	3232	3232
q22	1051	1015	997	997
Total cold run time: 108449 ms
Total hot run time: 40137 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6603	6742	6634	6634
q2	330	239	236	236
q3	2950	2831	3025	2831
q4	2061	1826	1775	1775
q5	5834	5815	5850	5815
q6	213	130	137	130
q7	2317	1896	1879	1879
q8	3417	3608	3581	3581
q9	8853	8860	8826	8826
q10	3548	3511	3479	3479
q11	591	500	509	500
q12	846	605	633	605
q13	8974	3185	3142	3142
q14	303	270	294	270
q15	512	462	470	462
q16	692	657	642	642
q17	1832	1628	1612	1612
q18	8179	7834	7588	7588
q19	1659	1492	1535	1492
q20	2080	1847	1947	1847
q21	5452	5260	5282	5260
q22	1135	1038	1039	1038
Total cold run time: 68381 ms
Total hot run time: 59644 ms

@doris-robot
Copy link

doris-robot commented May 9, 2025

TPC-DS: Total hot run time: 196365 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7eb0ca142e26fdf03a5432a916c0788898272717, data reload: false

query1	1292	934	877	877
query2	6291	1999	1922	1922
query3	10875	4294	4317	4294
query4	60741	29347	23453	23453
query5	5165	459	449	449
query6	390	175	183	175
query7	5551	332	311	311
query8	309	222	219	219
query9	8629	2577	2594	2577
query10	473	271	257	257
query11	17691	15151	15677	15151
query12	162	110	103	103
query13	1501	446	453	446
query14	10478	6764	6711	6711
query15	206	182	187	182
query16	7128	472	455	455
query17	1132	578	582	578
query18	1963	340	319	319
query19	214	171	163	163
query20	121	115	109	109
query21	208	104	99	99
query22	4750	4429	4772	4429
query23	34332	33821	33918	33821
query24	6141	2953	2927	2927
query25	532	413	411	411
query26	662	167	169	167
query27	1961	363	352	352
query28	4274	2458	2429	2429
query29	696	472	432	432
query30	246	165	157	157
query31	1000	824	819	819
query32	70	53	60	53
query33	483	288	272	272
query34	908	528	519	519
query35	857	736	747	736
query36	1062	935	944	935
query37	119	66	79	66
query38	4076	3963	3977	3963
query39	1512	1469	1469	1469
query40	196	97	98	97
query41	48	47	48	47
query42	112	100	96	96
query43	515	479	492	479
query44	1207	799	825	799
query45	184	170	173	170
query46	1139	723	721	721
query47	2012	1927	1916	1916
query48	471	387	395	387
query49	724	399	400	399
query50	857	438	442	438
query51	7251	7215	7438	7215
query52	109	95	86	86
query53	269	185	184	184
query54	570	461	460	460
query55	80	82	83	82
query56	278	240	256	240
query57	1261	1180	1133	1133
query58	222	211	225	211
query59	3280	3059	2950	2950
query60	301	272	270	270
query61	119	140	111	111
query62	791	694	686	686
query63	224	199	185	185
query64	1374	670	642	642
query65	3279	3191	3199	3191
query66	708	289	300	289
query67	15980	15507	15475	15475
query68	4431	585	562	562
query69	438	265	271	265
query70	1142	1112	1090	1090
query71	342	253	259	253
query72	6286	4151	4020	4020
query73	756	345	352	345
query74	10150	9198	9356	9198
query75	3374	2837	2677	2677
query76	2114	1003	1119	1003
query77	483	275	282	275
query78	10577	9649	9594	9594
query79	1661	590	603	590
query80	1181	427	430	427
query81	541	237	239	237
query82	416	90	89	89
query83	158	141	138	138
query84	279	79	76	76
query85	951	294	285	285
query86	347	297	301	297
query87	4455	4268	4211	4211
query88	4079	2395	2349	2349
query89	407	303	286	286
query90	1999	188	194	188
query91	181	147	147	147
query92	66	50	50	50
query93	1876	549	558	549
query94	784	301	284	284
query95	356	267	260	260
query96	610	283	278	278
query97	3340	3126	3182	3126
query98	210	203	197	197
query99	1574	1270	1253	1253
Total cold run time: 313326 ms
Total hot run time: 196365 ms

@doris-robot
Copy link

doris-robot commented May 9, 2025

ClickBench: Total hot run time: 31.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7eb0ca142e26fdf03a5432a916c0788898272717, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.10	0.10
query5	0.54	0.50	0.51
query6	1.15	0.73	0.74
query7	0.02	0.01	0.03
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.55	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.59
query14	2.90	2.76	2.78
query15	0.89	0.83	0.83
query16	0.38	0.39	0.39
query17	0.99	1.07	0.98
query18	0.23	0.21	0.21
query19	1.88	1.79	2.08
query20	0.01	0.02	0.01
query21	15.35	0.60	0.57
query22	2.55	2.35	1.86
query23	16.88	0.94	0.82
query24	2.87	1.64	0.48
query25	0.28	0.06	0.07
query26	0.41	0.14	0.14
query27	0.04	0.05	0.04
query28	10.59	0.51	0.45
query29	12.57	3.23	3.21
query30	0.25	0.06	0.06
query31	2.85	0.40	0.38
query32	3.26	0.47	0.46
query33	2.99	2.97	3.05
query34	17.14	4.44	4.46
query35	4.49	4.53	4.48
query36	0.69	0.49	0.48
query37	0.09	0.06	0.07
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.67 s
Total hot run time: 31.51 s

@zddr
Copy link
Contributor Author

zddr commented May 9, 2025

run external

1 similar comment
@zddr
Copy link
Contributor Author

zddr commented May 9, 2025

run external

@zddr
Copy link
Contributor Author

zddr commented May 11, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 11, 2025

TPC-H: Total hot run time: 39950 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7eb0ca142e26fdf03a5432a916c0788898272717, data reload: false

------ Round 1 ----------------------------------
q1	17613	6774	6581	6581
q2	2056	165	165	165
q3	10635	1087	1198	1087
q4	10562	770	786	770
q5	7731	2814	2738	2738
q6	223	136	134	134
q7	989	640	632	632
q8	9348	1937	2038	1937
q9	6606	6400	6434	6400
q10	6993	2228	2316	2228
q11	451	254	260	254
q12	396	210	213	210
q13	17762	2957	2971	2957
q14	240	206	206	206
q15	504	454	456	454
q16	651	580	588	580
q17	952	630	505	505
q18	7152	6651	6660	6651
q19	1403	1107	1039	1039
q20	479	206	200	200
q21	4018	3210	3430	3210
q22	1122	1051	1012	1012
Total cold run time: 107886 ms
Total hot run time: 39950 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6752	6658	6751	6658
q2	332	250	240	240
q3	3010	2832	2968	2832
q4	2038	1786	1805	1786
q5	5795	5754	5774	5754
q6	214	128	129	128
q7	2244	1842	1842	1842
q8	3415	3616	3551	3551
q9	8897	8884	8866	8866
q10	3650	3577	3542	3542
q11	612	501	489	489
q12	823	663	635	635
q13	10522	3342	3306	3306
q14	303	293	296	293
q15	525	470	470	470
q16	730	685	669	669
q17	1869	1685	1619	1619
q18	8255	7821	7808	7808
q19	2190	1630	1629	1629
q20	2154	1875	1882	1875
q21	5403	5193	5404	5193
q22	1089	1029	1027	1027
Total cold run time: 70822 ms
Total hot run time: 60212 ms

@doris-robot
Copy link

doris-robot commented May 11, 2025

TPC-DS: Total hot run time: 197342 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7eb0ca142e26fdf03a5432a916c0788898272717, data reload: false

query1	1290	899	907	899
query2	6371	1994	1951	1951
query3	10832	4350	4398	4350
query4	61054	29028	23670	23670
query5	5186	459	443	443
query6	398	170	173	170
query7	5497	306	306	306
query8	308	234	223	223
query9	8629	2598	2588	2588
query10	472	268	259	259
query11	17913	15288	15716	15288
query12	165	109	114	109
query13	1459	458	420	420
query14	9894	7565	7335	7335
query15	197	184	182	182
query16	7122	453	493	453
query17	1232	577	597	577
query18	1929	328	308	308
query19	225	160	165	160
query20	112	111	107	107
query21	218	110	110	110
query22	4560	4548	4534	4534
query23	34555	33914	34012	33914
query24	6159	2947	2963	2947
query25	558	457	418	418
query26	655	170	175	170
query27	1926	384	363	363
query28	4539	2438	2432	2432
query29	718	490	417	417
query30	240	175	160	160
query31	982	816	843	816
query32	69	57	56	56
query33	458	293	299	293
query34	902	513	544	513
query35	851	741	735	735
query36	1080	924	936	924
query37	114	68	68	68
query38	4107	4029	3961	3961
query39	1528	1506	1489	1489
query40	209	95	99	95
query41	48	49	47	47
query42	114	101	101	101
query43	514	496	479	479
query44	1207	823	809	809
query45	186	167	168	167
query46	1137	726	734	726
query47	2018	1875	1882	1875
query48	483	386	387	386
query49	741	394	389	389
query50	844	434	443	434
query51	7351	7162	7277	7162
query52	102	93	88	88
query53	266	189	198	189
query54	571	464	470	464
query55	77	74	82	74
query56	264	256	252	252
query57	1300	1187	1171	1171
query58	231	217	241	217
query59	3223	2992	2983	2983
query60	275	247	255	247
query61	151	110	133	110
query62	744	669	687	669
query63	215	184	182	182
query64	1369	660	646	646
query65	3307	3197	3221	3197
query66	723	299	307	299
query67	15899	15328	15430	15328
query68	4302	563	550	550
query69	435	266	271	266
query70	1196	1089	1087	1087
query71	341	263	259	259
query72	6372	4020	4030	4020
query73	743	343	347	343
query74	10190	9021	9258	9021
query75	3342	2615	2636	2615
query76	1974	1072	1172	1072
query77	521	265	280	265
query78	10627	9566	9627	9566
query79	1325	603	604	603
query80	890	437	434	434
query81	502	256	245	245
query82	1239	90	88	88
query83	240	153	141	141
query84	286	81	76	76
query85	870	299	285	285
query86	320	294	286	286
query87	4391	4222	4280	4222
query88	3564	2382	2355	2355
query89	411	294	294	294
query90	1979	192	183	183
query91	184	149	146	146
query92	67	49	51	49
query93	1877	544	544	544
query94	755	307	291	291
query95	359	272	260	260
query96	611	280	280	280
query97	3273	3100	3147	3100
query98	214	201	199	199
query99	1598	1293	1300	1293
Total cold run time: 313071 ms
Total hot run time: 197342 ms

@doris-robot
Copy link

doris-robot commented May 11, 2025

ClickBench: Total hot run time: 31.97 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7eb0ca142e26fdf03a5432a916c0788898272717, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.63	0.11	0.10
query5	0.54	0.50	0.52
query6	1.14	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.02
query9	0.55	0.50	0.50
query10	0.55	0.56	0.54
query11	0.15	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.59	0.58
query14	2.73	2.87	2.86
query15	0.90	0.83	0.82
query16	0.38	0.38	0.38
query17	1.05	1.04	1.02
query18	0.24	0.22	0.21
query19	1.83	1.83	2.02
query20	0.02	0.01	0.01
query21	15.36	0.59	0.58
query22	2.45	2.96	1.87
query23	17.02	1.10	0.73
query24	2.91	0.79	1.16
query25	0.36	0.14	0.14
query26	0.21	0.14	0.13
query27	0.04	0.04	0.05
query28	10.73	0.50	0.45
query29	12.56	3.22	3.19
query30	0.25	0.06	0.06
query31	2.86	0.39	0.39
query32	3.24	0.47	0.45
query33	3.01	3.02	3.02
query34	17.20	4.48	4.45
query35	4.50	4.50	4.48
query36	0.67	0.49	0.47
query37	0.09	0.05	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.7 s
Total hot run time: 31.97 s

@zddr
Copy link
Contributor Author

zddr commented May 12, 2025

run p0

dataroaring
dataroaring approved these changes May 19, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 69368b3 into apache:branch-3.0 May 19, 2025
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zddr @hello-stephen @doris-robot @dataroaring