Skip to content

[fix](mtmv) Fix compensate union all wrongly when partition mv use partition external table #52223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 2 commits into from
Jun 30, 2025
Merged

[fix](mtmv) Fix compensate union all wrongly when partition mv use partition external table #52223

dataroaring merged 2 commits into from
Jun 30, 2025

Conversation

@seawinde
Copy link
Contributor

@seawinde seawinde commented Jun 24, 2025

Fix compensate union all wrongly when partition mv use partition external table

in test mv/external_table/part_partition_invalid.groovy
order_qt_after_modify_data_and_refresh_catalog data should be as following:

1	2	1	2023-10-17	2023-10-17
2	2	2	2023-10-18	2023-10-18
3	2	3	2023-10-19	2023-10-19

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #49514

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@seawinde seawinde requested a review from dataroaring as a code owner June 24, 2025 09:35
@Thearas
Copy link
Contributor

Thearas commented Jun 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@seawinde
Copy link
Contributor Author

seawinde commented Jun 24, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 24, 2025

TPC-H: Total hot run time: 39802 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f96dfe0ffb533ae343f23ede2f6970d8131f67d6, data reload: false

------ Round 1 ----------------------------------
q1	17823	7051	6735	6735
q2	2130	162	162	162
q3	12722	1129	1169	1129
q4	10229	725	701	701
q5	7745	2817	2837	2817
q6	213	135	131	131
q7	966	637	603	603
q8	9363	1931	1992	1931
q9	6668	6388	6446	6388
q10	6951	2309	2314	2309
q11	456	270	259	259
q12	391	207	214	207
q13	17796	2962	2981	2962
q14	242	213	210	210
q15	499	464	465	464
q16	449	372	367	367
q17	957	578	574	574
q18	7339	6658	6664	6658
q19	1396	1018	1001	1001
q20	483	203	199	199
q21	4024	3026	3022	3022
q22	1100	973	979	973
Total cold run time: 109942 ms
Total hot run time: 39802 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6563	6547	6527	6527
q2	325	241	227	227
q3	2913	2714	3013	2714
q4	2071	1795	1793	1793
q5	5607	5806	5644	5644
q6	212	124	129	124
q7	2176	1763	1756	1756
q8	3352	3462	3564	3462
q9	8826	8848	8876	8848
q10	3556	3541	3538	3538
q11	600	503	512	503
q12	790	572	576	572
q13	7756	3144	3132	3132
q14	311	265	275	265
q15	501	467	455	455
q16	475	415	438	415
q17	1819	1609	1612	1609
q18	8133	7870	7691	7691
q19	1679	1547	1518	1518
q20	2095	1888	1846	1846
q21	5126	5052	4895	4895
q22	1101	1051	962	962
Total cold run time: 65987 ms
Total hot run time: 58496 ms

@doris-robot
Copy link

doris-robot commented Jun 24, 2025

TPC-DS: Total hot run time: 188699 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f96dfe0ffb533ae343f23ede2f6970d8131f67d6, data reload: false

query1	958	370	362	362
query2	6520	1876	1806	1806
query3	6693	225	222	222
query4	34078	23331	23861	23331
query5	4334	443	429	429
query6	283	171	181	171
query7	4625	314	310	310
query8	276	225	232	225
query9	9570	2555	2544	2544
query10	476	268	260	260
query11	18022	15151	15152	15151
query12	152	102	105	102
query13	1643	419	404	404
query14	8865	7272	6538	6538
query15	256	174	183	174
query16	8110	468	481	468
query17	1630	586	566	566
query18	2123	307	310	307
query19	258	170	155	155
query20	122	110	112	110
query21	205	105	102	102
query22	4437	4107	4171	4107
query23	34369	33423	33133	33133
query24	11799	2840	2806	2806
query25	694	424	417	417
query26	1733	168	167	167
query27	3027	347	357	347
query28	7925	2098	2116	2098
query29	1045	448	441	441
query30	318	162	162	162
query31	1020	804	813	804
query32	95	61	60	60
query33	792	311	339	311
query34	907	504	511	504
query35	882	720	715	715
query36	1111	919	927	919
query37	128	68	69	68
query38	3941	3844	3795	3795
query39	1495	1450	1435	1435
query40	299	103	101	101
query41	54	52	52	52
query42	115	109	102	102
query43	497	471	450	450
query44	1275	788	800	788
query45	185	169	176	169
query46	1127	701	712	701
query47	1956	1833	1830	1830
query48	480	389	380	380
query49	1253	403	421	403
query50	811	418	413	413
query51	7225	7031	7125	7031
query52	103	95	93	93
query53	260	188	190	188
query54	1274	475	469	469
query55	82	80	78	78
query56	284	252	261	252
query57	1300	1174	1154	1154
query58	250	218	224	218
query59	3115	2795	2878	2795
query60	298	256	248	248
query61	112	110	147	110
query62	883	686	656	656
query63	216	185	191	185
query64	5339	648	648	648
query65	3286	3237	3162	3162
query66	1432	326	293	293
query67	15808	15511	15462	15462
query68	4667	585	557	557
query69	440	267	269	267
query70	1119	1087	1123	1087
query71	400	260	260	260
query72	6398	3968	3899	3899
query73	756	340	342	340
query74	10400	8915	9217	8915
query75	3369	2639	2640	2639
query76	2819	1029	1088	1029
query77	405	280	272	272
query78	10495	9541	9613	9541
query79	1649	585	587	585
query80	1051	431	416	416
query81	530	224	222	222
query82	973	84	85	84
query83	217	147	143	143
query84	236	78	78	78
query85	1323	312	288	288
query86	421	293	294	293
query87	4380	4291	4292	4291
query88	3269	2391	2364	2364
query89	400	296	307	296
query90	1978	187	193	187
query91	187	151	170	151
query92	66	50	52	50
query93	1430	565	556	556
query94	942	304	281	281
query95	353	267	259	259
query96	604	280	282	280
query97	3264	3116	3124	3116
query98	210	200	200	200
query99	1526	1341	1296	1296
Total cold run time: 300513 ms
Total hot run time: 188699 ms

@doris-robot
Copy link

doris-robot commented Jun 24, 2025

ClickBench: Total hot run time: 29.83 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f96dfe0ffb533ae343f23ede2f6970d8131f67d6, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.61	0.10	0.11
query5	0.51	0.53	0.51
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.57	0.49	0.49
query10	0.55	0.56	0.57
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	0.76	0.80	0.80
query15	0.84	0.83	0.81
query16	0.38	0.37	0.38
query17	1.03	1.05	1.02
query18	0.24	0.23	0.22
query19	1.96	1.87	1.80
query20	0.02	0.01	0.01
query21	15.40	0.61	0.58
query22	2.33	2.26	1.66
query23	17.20	0.87	0.85
query24	2.73	1.18	0.82
query25	0.29	0.10	0.10
query26	0.39	0.14	0.14
query27	0.04	0.04	0.05
query28	10.81	0.47	0.46
query29	12.65	3.24	3.22
query30	0.24	0.07	0.06
query31	2.85	0.38	0.37
query32	3.27	0.47	0.46
query33	2.95	3.01	2.99
query34	17.03	4.45	4.45
query35	4.49	4.49	4.53
query36	0.65	0.47	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.03
query40	0.15	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.7 s
Total hot run time: 29.83 s

@seawinde
Copy link
Contributor Author

seawinde commented Jun 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-H: Total hot run time: 39566 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 79e3b4a4453707b06bad5b032c08b618af5dc301, data reload: false

------ Round 1 ----------------------------------
q1	17589	6725	6581	6581
q2	2081	166	166	166
q3	10611	1041	1175	1041
q4	10561	766	749	749
q5	7714	2890	2825	2825
q6	219	133	131	131
q7	993	605	611	605
q8	9348	1928	2040	1928
q9	6572	6420	6426	6420
q10	6991	2316	2268	2268
q11	457	268	259	259
q12	398	217	208	208
q13	17811	3005	3018	3005
q14	229	205	211	205
q15	506	461	487	461
q16	512	377	390	377
q17	976	581	521	521
q18	7162	6561	6753	6561
q19	1396	1068	930	930
q20	459	201	205	201
q21	3863	3214	3117	3117
q22	1058	1007	1033	1007
Total cold run time: 107506 ms
Total hot run time: 39566 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6610	6519	6545	6519
q2	330	233	240	233
q3	2914	2785	2818	2785
q4	2078	1818	1808	1808
q5	5734	5750	5789	5750
q6	208	127	125	125
q7	2253	1821	1747	1747
q8	3390	3576	3510	3510
q9	8950	8758	8884	8758
q10	3549	3530	3512	3512
q11	611	510	490	490
q12	822	605	610	605
q13	8811	3111	3135	3111
q14	317	286	284	284
q15	512	473	451	451
q16	493	433	447	433
q17	1843	1610	1610	1610
q18	8181	7907	7678	7678
q19	1721	1628	1483	1483
q20	2122	1864	1815	1815
q21	5165	5084	5102	5084
q22	1093	1038	1053	1038
Total cold run time: 67707 ms
Total hot run time: 58829 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-DS: Total hot run time: 196597 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 79e3b4a4453707b06bad5b032c08b618af5dc301, data reload: false

query1	1282	906	900	900
query2	6743	1998	1955	1955
query3	10903	4346	4369	4346
query4	61283	28682	23415	23415
query5	5216	450	443	443
query6	417	178	188	178
query7	5486	303	302	302
query8	316	223	243	223
query9	8545	2562	2557	2557
query10	458	277	258	258
query11	17573	15002	15707	15002
query12	164	104	109	104
query13	1472	436	428	428
query14	11004	7281	7844	7281
query15	199	179	182	179
query16	7154	507	499	499
query17	1249	602	603	602
query18	1958	320	315	315
query19	212	157	178	157
query20	122	110	108	108
query21	211	102	108	102
query22	4548	4291	4660	4291
query23	34713	34249	34213	34213
query24	6106	2871	2917	2871
query25	545	422	423	422
query26	672	167	169	167
query27	1991	360	362	360
query28	3938	2170	2146	2146
query29	725	472	444	444
query30	236	164	164	164
query31	1014	830	849	830
query32	71	58	64	58
query33	427	318	314	314
query34	916	502	507	502
query35	838	746	746	746
query36	1093	971	952	952
query37	112	65	70	65
query38	4133	3973	3954	3954
query39	1500	1474	1468	1468
query40	211	100	101	100
query41	52	49	50	49
query42	118	106	102	102
query43	561	502	501	501
query44	1192	804	800	800
query45	185	170	167	167
query46	1157	739	719	719
query47	2035	1934	1918	1918
query48	482	391	369	369
query49	730	391	392	391
query50	856	418	423	418
query51	7507	7241	7346	7241
query52	104	104	87	87
query53	257	184	185	184
query54	582	486	465	465
query55	79	75	77	75
query56	278	256	254	254
query57	1305	1159	1180	1159
query58	228	239	222	222
query59	3326	3095	3170	3095
query60	290	266	251	251
query61	108	108	108	108
query62	755	662	680	662
query63	213	182	182	182
query64	1406	675	654	654
query65	3270	3199	3188	3188
query66	702	293	298	293
query67	15819	15433	15503	15433
query68	4357	571	563	563
query69	444	268	266	266
query70	1167	1147	1101	1101
query71	336	278	251	251
query72	6422	4214	3832	3832
query73	757	350	350	350
query74	10264	9001	9268	9001
query75	3398	2620	2677	2620
query76	1965	1093	1108	1093
query77	491	287	260	260
query78	10685	9563	9702	9563
query79	2044	586	595	586
query80	1336	434	441	434
query81	519	224	222	222
query82	536	85	87	85
query83	165	139	138	138
query84	280	81	72	72
query85	973	319	301	301
query86	376	296	301	296
query87	4406	4270	4233	4233
query88	3921	2381	2335	2335
query89	427	300	293	293
query90	1981	184	183	183
query91	179	144	149	144
query92	58	51	50	50
query93	2463	549	544	544
query94	770	295	261	261
query95	367	259	256	256
query96	616	271	276	271
query97	3368	3142	3089	3089
query98	220	210	196	196
query99	1618	1300	1310	1300
Total cold run time: 316262 ms
Total hot run time: 196597 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

ClickBench: Total hot run time: 30.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 79e3b4a4453707b06bad5b032c08b618af5dc301, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.10	0.10
query5	0.51	0.50	0.51
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.50
query10	0.55	0.60	0.56
query11	0.14	0.09	0.10
query12	0.14	0.11	0.12
query13	0.59	0.60	0.60
query14	0.76	0.78	0.81
query15	0.84	0.83	0.81
query16	0.37	0.38	0.38
query17	1.04	1.07	1.05
query18	0.23	0.22	0.22
query19	1.95	1.78	1.76
query20	0.01	0.01	0.01
query21	15.42	0.62	0.57
query22	2.57	1.58	1.84
query23	16.95	0.97	0.78
query24	3.55	1.29	1.88
query25	0.35	0.19	0.15
query26	0.40	0.14	0.14
query27	0.06	0.03	0.06
query28	9.07	0.49	0.50
query29	12.58	3.20	3.18
query30	0.25	0.06	0.06
query31	2.85	0.38	0.37
query32	3.26	0.48	0.45
query33	2.94	3.01	3.03
query34	17.11	4.48	4.54
query35	4.56	4.53	4.59
query36	0.69	0.47	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.15	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.95 s
Total hot run time: 30.26 s

@morrySnow
Copy link
Contributor

morrySnow commented Jun 30, 2025

add test case

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 30, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 30, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 30, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jun 30, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit e68e7cb into apache:branch-3.0 Jun 30, 2025
21 of 22 checks passed
koarz pushed a commit to koarz/doris that referenced this pull request Jul 3, 2025
@seawinde @koarz
[fix](mtmv) Fix compensate union all wrongly when partition mv use pa…
f9ecff3 
…rtition external table (apache#52223)

Fix compensate union all wrongly when partition mv use partition
external table

in test mv/external_table/part_partition_invalid.groovy
order_qt_after_modify_data_and_refresh_catalog data should be as
following:
```
1	2	1	2023-10-17	2023-10-17
2	2	2	2023-10-18	2023-10-18
3	2	3	2023-10-19	2023-10-19
```

Related PR: apache#49514
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.7-merged not-merge/3.1 p0_w reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@seawinde @Thearas @doris-robot @morrySnow @dataroaring