Skip to content

[fix](mtmv) Fix hudi materialized view union all rewritten plan execute fail because of invalid slot #58643

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 2 commits into from
Dec 4, 2025
Merged

[fix](mtmv) Fix hudi materialized view union all rewritten plan execute fail because of invalid slot #58643

morrySnow merged 2 commits into from
Dec 4, 2025

Conversation

@seawinde
Copy link
Contributor

@seawinde seawinde commented Dec 2, 2025 •
edited
Loading

What problem does this PR solve?

This fix addresses the following three issues:

  1. When invoking the method org.apache.doris.nereids.trees.plans.logical.LogicalHudiScan#withRelationId, the output needs to be recalculated to meet expectations.
  2. After compensating with a union all due to partial partition invalidation of a materialized view, during the next round of transparent rewriting, the rewriting for the child of the union allshould use the query partitioncorresponding to the specific relation id to prevent infinite loops.
  3. Currently, in the test_hudi_rewrite_mtmv test, if the plan rewritten by the materialized view transparent rewriting is not selected by the CBO, it is difficult to troubleshoot because explain memo planis not used. Therefore, the corresponding test method is modified.

Issue Number: close #xxx

Related PR:
#57558
#58413

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 2, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@seawinde
Copy link
Contributor Author

seawinde commented Dec 2, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 2, 2025

TPC-H: Total hot run time: 34710 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 548a5a3ca3a8b4967a10ddfd539f5cc9afea66bf, data reload: false

------ Round 1 ----------------------------------
q1	17612	5136	4968	4968
q2	2044	326	209	209
q3	10229	1321	756	756
q4	10207	853	333	333
q5	7724	2501	2179	2179
q6	193	175	149	149
q7	991	854	681	681
q8	9366	1489	1222	1222
q9	6994	5327	5402	5327
q10	6892	2237	1789	1789
q11	537	304	305	304
q12	341	387	240	240
q13	17810	3679	3056	3056
q14	250	248	214	214
q15	595	506	514	506
q16	885	869	815	815
q17	668	877	529	529
q18	7929	6986	7000	6986
q19	1116	950	638	638
q20	384	358	230	230
q21	4051	3869	2627	2627
q22	1048	1007	952	952
Total cold run time: 107866 ms
Total hot run time: 34710 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5065	5012	5002	5002
q2	330	400	326	326
q3	2145	2712	2320	2320
q4	1330	1752	1307	1307
q5	4328	4501	4516	4501
q6	226	173	136	136
q7	2066	1976	1776	1776
q8	2789	2575	2553	2553
q9	7682	7376	7553	7376
q10	3116	3255	2906	2906
q11	611	521	500	500
q12	693	763	666	666
q13	3574	4052	3525	3525
q14	321	295	282	282
q15	553	525	510	510
q16	875	939	877	877
q17	1192	1471	1447	1447
q18	8000	7637	7424	7424
q19	944	867	912	867
q20	2113	2054	1798	1798
q21	4719	4267	4157	4157
q22	1070	1033	991	991
Total cold run time: 53742 ms
Total hot run time: 51247 ms

@doris-robot
Copy link

doris-robot commented Dec 2, 2025

TPC-DS: Total hot run time: 182427 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 548a5a3ca3a8b4967a10ddfd539f5cc9afea66bf, data reload: false

query1	1094	413	402	402
query2	6606	1224	1180	1180
query3	6743	229	226	226
query4	25707	23253	22938	22938
query5	4812	681	516	516
query6	341	262	227	227
query7	4661	535	311	311
query8	324	259	244	244
query9	8723	2643	2652	2643
query10	535	352	347	347
query11	15484	15173	14525	14525
query12	188	120	112	112
query13	1676	579	445	445
query14	9283	6181	6316	6181
query15	225	202	185	185
query16	7274	702	492	492
query17	1217	771	629	629
query18	2008	436	345	345
query19	220	216	194	194
query20	134	130	130	130
query21	227	173	117	117
query22	3877	3859	3887	3859
query23	32904	31965	31753	31753
query24	8620	2457	2484	2457
query25	635	513	459	459
query26	1246	284	172	172
query27	2740	522	339	339
query28	4386	2166	2131	2131
query29	856	643	496	496
query30	322	243	208	208
query31	859	754	627	627
query32	87	77	74	74
query33	605	390	344	344
query34	865	913	554	554
query35	813	858	767	767
query36	912	938	853	853
query37	128	123	93	93
query38	3947	3865	3863	3863
query39	1543	1447	1439	1439
query40	234	148	130	130
query41	66	65	69	65
query42	136	119	120	119
query43	439	456	430	430
query44	1377	775	752	752
query45	203	193	185	185
query46	960	1057	666	666
query47	1684	1696	1624	1624
query48	412	437	324	324
query49	779	496	422	422
query50	722	776	416	416
query51	3948	3850	3845	3845
query52	114	114	106	106
query53	249	273	197	197
query54	319	309	291	291
query55	99	99	92	92
query56	357	350	348	348
query57	1115	1159	1115	1115
query58	302	283	275	275
query59	2275	2360	2377	2360
query60	373	378	373	373
query61	192	189	185	185
query62	797	715	697	697
query63	248	207	201	201
query64	4792	1343	1012	1012
query65	4083	3968	3972	3968
query66	1210	471	355	355
query67	15400	14941	14770	14770
query68	6286	1023	634	634
query69	543	367	324	324
query70	1108	1051	1017	1017
query71	445	364	329	329
query72	6069	4948	4809	4809
query73	674	602	349	349
query74	8749	8850	8833	8833
query75	3072	3053	2569	2569
query76	3291	1217	786	786
query77	509	428	320	320
query78	9501	9929	8846	8846
query79	1413	876	571	571
query80	1693	587	496	496
query81	568	277	244	244
query82	424	148	117	117
query83	364	271	263	263
query84	255	122	97	97
query85	916	499	450	450
query86	389	297	294	294
query87	4119	4195	4040	4040
query88	3097	2339	2311	2311
query89	405	335	295	295
query90	1765	230	227	227
query91	183	178	146	146
query92	79	72	83	72
query93	1257	1028	663	663
query94	720	477	369	369
query95	518	435	421	421
query96	539	558	286	286
query97	2634	2674	2600	2600
query98	242	224	209	209
query99	1285	1392	1240	1240
Total cold run time: 268824 ms
Total hot run time: 182427 ms

@doris-robot
Copy link

doris-robot commented Dec 2, 2025

ClickBench: Total hot run time: 27.77 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 548a5a3ca3a8b4967a10ddfd539f5cc9afea66bf, data reload: false

query1	0.07	0.05	0.04
query2	0.10	0.05	0.05
query3	0.25	0.10	0.08
query4	1.61	0.11	0.10
query5	0.28	0.27	0.28
query6	1.20	0.64	0.63
query7	0.03	0.03	0.03
query8	0.06	0.05	0.04
query9	0.58	0.52	0.50
query10	0.56	0.56	0.56
query11	0.16	0.10	0.11
query12	0.15	0.12	0.12
query13	0.64	0.61	0.61
query14	0.99	0.98	0.97
query15	0.82	0.80	0.80
query16	0.40	0.38	0.41
query17	1.02	1.04	1.02
query18	0.24	0.26	0.21
query19	1.88	1.87	1.86
query20	0.02	0.01	0.01
query21	15.44	0.29	0.14
query22	4.99	0.05	0.05
query23	16.11	0.28	0.11
query24	1.27	0.70	0.88
query25	0.09	0.08	0.05
query26	0.15	0.13	0.13
query27	0.06	0.04	0.05
query28	4.82	1.23	1.03
query29	12.62	3.88	3.13
query30	0.30	0.13	0.12
query31	2.82	0.64	0.41
query32	3.23	0.56	0.47
query33	3.10	3.24	3.06
query34	16.99	5.17	4.56
query35	4.57	4.53	4.60
query36	0.67	0.50	0.49
query37	0.12	0.07	0.07
query38	0.07	0.04	0.04
query39	0.05	0.03	0.04
query40	0.18	0.14	0.14
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 98.89 s
Total hot run time: 27.77 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 2, 2025

FE UT Coverage Report

Increment line coverage 84.62% (11/13) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 2, 2025

FE Regression Coverage Report

Increment line coverage 76.92% (10/13) 🎉
Increment coverage report
Complete coverage report

@seawinde
Copy link
Contributor Author

seawinde commented Dec 3, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 3, 2025

TPC-H: Total hot run time: 34254 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 99f0df716682d530913e1e036dc906c3309b56e1, data reload: false

------ Round 1 ----------------------------------
q1	17724	5066	4929	4929
q2	2203	359	201	201
q3	10208	1297	747	747
q4	10270	871	318	318
q5	7513	2349	2176	2176
q6	189	177	145	145
q7	946	781	658	658
q8	9355	1397	1125	1125
q9	7077	5290	5320	5290
q10	7000	2182	1799	1799
q11	543	324	286	286
q12	389	374	231	231
q13	17783	3619	3015	3015
q14	226	241	215	215
q15	570	518	506	506
q16	857	848	813	813
q17	731	782	542	542
q18	7488	7109	7004	7004
q19	1094	955	608	608
q20	362	360	229	229
q21	4030	3457	2488	2488
q22	1023	987	929	929
Total cold run time: 107581 ms
Total hot run time: 34254 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5004	4966	4951	4951
q2	324	424	314	314
q3	2148	2652	2290	2290
q4	1292	1745	1296	1296
q5	4221	4408	4448	4408
q6	211	171	131	131
q7	2040	2024	1918	1918
q8	2733	2640	2533	2533
q9	7457	7562	7581	7562
q10	3077	3207	2902	2902
q11	574	512	507	507
q12	695	722	569	569
q13	3400	3871	3336	3336
q14	283	300	301	300
q15	547	498	496	496
q16	914	942	896	896
q17	1221	1527	1459	1459
q18	7871	7567	7590	7567
q19	919	898	914	898
q20	1976	1963	1804	1804
q21	4597	4236	4055	4055
q22	1093	1045	969	969
Total cold run time: 52597 ms
Total hot run time: 51161 ms

@doris-robot
Copy link

doris-robot commented Dec 3, 2025

TPC-DS: Total hot run time: 181466 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 99f0df716682d530913e1e036dc906c3309b56e1, data reload: false

query1	1116	403	400	400
query2	6650	1172	1172	1172
query3	6826	226	230	226
query4	25195	23400	22997	22997
query5	5249	649	497	497
query6	341	241	232	232
query7	4687	520	309	309
query8	302	255	234	234
query9	8757	2620	2618	2618
query10	574	373	322	322
query11	15390	14770	14892	14770
query12	190	123	114	114
query13	1697	582	452	452
query14	9925	5975	5979	5975
query15	214	201	183	183
query16	7680	699	536	536
query17	1251	791	648	648
query18	2077	442	345	345
query19	218	218	186	186
query20	130	124	119	119
query21	407	139	122	122
query22	3838	3885	3942	3885
query23	32833	31816	31838	31816
query24	8398	2410	2419	2410
query25	650	549	487	487
query26	1298	279	176	176
query27	2668	501	351	351
query28	4329	2168	2153	2153
query29	834	654	519	519
query30	354	243	213	213
query31	826	709	618	618
query32	90	80	76	76
query33	635	389	347	347
query34	840	880	538	538
query35	811	876	754	754
query36	890	916	844	844
query37	166	111	89	89
query38	3816	3864	3773	3773
query39	1459	1416	1431	1416
query40	227	135	118	118
query41	67	62	61	61
query42	121	118	113	113
query43	426	458	418	418
query44	1361	742	737	737
query45	200	192	187	187
query46	908	1021	642	642
query47	1673	1744	1636	1636
query48	425	432	329	329
query49	760	486	403	403
query50	698	705	411	411
query51	3940	3924	3848	3848
query52	113	113	103	103
query53	249	263	196	196
query54	317	323	280	280
query55	98	94	92	92
query56	345	333	339	333
query57	1108	1161	1110	1110
query58	297	277	272	272
query59	2333	2369	2297	2297
query60	349	350	325	325
query61	188	163	164	163
query62	791	727	642	642
query63	233	198	189	189
query64	4459	1230	894	894
query65	4031	3951	3968	3951
query66	1085	439	328	328
query67	15071	15114	14791	14791
query68	8304	998	619	619
query69	532	337	310	310
query70	1100	1005	1019	1005
query71	498	337	345	337
query72	5895	4857	4807	4807
query73	684	589	345	345
query74	8878	8897	8631	8631
query75	3666	3000	2535	2535
query76	3818	1134	753	753
query77	818	393	325	325
query78	9603	9493	8830	8830
query79	2318	845	590	590
query80	645	577	489	489
query81	509	267	240	240
query82	516	141	108	108
query83	276	269	253	253
query84	267	120	105	105
query85	906	501	450	450
query86	390	310	301	301
query87	4128	4099	3930	3930
query88	4143	2316	2286	2286
query89	397	330	299	299
query90	1993	227	231	227
query91	174	173	137	137
query92	89	67	66	66
query93	1964	1047	664	664
query94	720	441	339	339
query95	509	418	406	406
query96	543	576	287	287
query97	2622	2635	2525	2525
query98	250	210	204	204
query99	1360	1379	1272	1272
Total cold run time: 273999 ms
Total hot run time: 181466 ms

@doris-robot
Copy link

doris-robot commented Dec 3, 2025

ClickBench: Total hot run time: 27.3 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 99f0df716682d530913e1e036dc906c3309b56e1, data reload: false

query1	0.05	0.04	0.04
query2	0.10	0.05	0.04
query3	0.24	0.09	0.08
query4	1.61	0.11	0.11
query5	0.28	0.25	0.27
query6	1.14	0.63	0.64
query7	0.03	0.03	0.03
query8	0.06	0.04	0.04
query9	0.55	0.50	0.50
query10	0.55	0.56	0.56
query11	0.14	0.10	0.10
query12	0.15	0.11	0.11
query13	0.64	0.61	0.59
query14	0.99	1.00	0.98
query15	0.81	0.81	0.79
query16	0.42	0.42	0.40
query17	1.05	1.08	1.08
query18	0.23	0.22	0.21
query19	1.94	1.84	1.85
query20	0.02	0.01	0.01
query21	15.44	0.29	0.14
query22	4.79	0.05	0.05
query23	16.16	0.30	0.10
query24	1.05	0.24	0.18
query25	0.06	0.07	0.08
query26	0.14	0.14	0.13
query27	0.08	0.08	0.05
query28	3.32	1.23	1.02
query29	12.61	3.97	3.23
query30	0.28	0.13	0.12
query31	2.82	0.62	0.38
query32	3.30	0.55	0.45
query33	2.99	3.05	3.06
query34	16.98	5.21	4.54
query35	4.58	4.53	4.56
query36	0.66	0.50	0.49
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.16	0.14	0.14
query41	0.09	0.04	0.04
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.81 s
Total hot run time: 27.3 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 3, 2025

FE UT Coverage Report

Increment line coverage 80.95% (17/21) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 3, 2025

FE Regression Coverage Report

Increment line coverage 71.43% (15/21) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 3, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 3, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 3, 2025

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit a128c14 into apache:master Dec 4, 2025
28 of 30 checks passed
seawinde added a commit to seawinde/doris that referenced this pull request Dec 4, 2025
@seawinde
[fix](mtmv) Fix hudi materialized view union all rewritten plan execu…
eb2494e 
…te fail because of invalid slot (apache#58643)

Related PR: apache#57558 apache#58413

Problem Summary:

This fix addresses the following three issues:
1. When invoking the method
org.apache.doris.nereids.trees.plans.logical.LogicalHudiScan#withRelationId,
the output needs to be recalculated to meet expectations.
2. After compensating with a union all due to partial partition
invalidation of a materialized view, during the next round of
transparent rewriting, the rewriting for the child of the union
allshould use the query partitioncorresponding to the specific relation
id to prevent infinite loops.
3. Currently, in the `test_hudi_rewrite_mtmv` test, if the plan
rewritten by the materialized view transparent rewriting is not selected
by the CBO, it is difficult to troubleshoot because explain memo planis
not used. Therefore, the corresponding test method is modified.
@seawinde seawinde mentioned this pull request Dec 4, 2025
Merged
seawinde added a commit to seawinde/doris that referenced this pull request Dec 4, 2025
@seawinde
[fix](mtmv) Fix hudi materialized view union all rewritten plan execu…
c5428ea 
…te fail because of invalid slot (apache#58643)

Related PR: apache#57558 apache#58413

Problem Summary:

This fix addresses the following three issues:
1. When invoking the method
org.apache.doris.nereids.trees.plans.logical.LogicalHudiScan#withRelationId,
the output needs to be recalculated to meet expectations.
2. After compensating with a union all due to partial partition
invalidation of a materialized view, during the next round of
transparent rewriting, the rewriting for the child of the union
allshould use the query partitioncorresponding to the specific relation
id to prevent infinite loops.
3. Currently, in the `test_hudi_rewrite_mtmv` test, if the plan
rewritten by the materialized view transparent rewriting is not selected
by the CBO, it is difficult to troubleshoot because explain memo planis
not used. Therefore, the corresponding test method is modified.
@seawinde seawinde mentioned this pull request Dec 4, 2025
Merged
yiguolei pushed a commit that referenced this pull request Dec 5, 2025
@seawinde
branch-4.0:[fix](mtmv) Fix hudi materialized view union all rewritten…
2255d7a 
… plan execute fail because of invalid slot (#58643) (#58725)

pr: #58643
commitId: a128c14
morrySnow pushed a commit that referenced this pull request Dec 5, 2025
@seawinde
branch-3.1: [fix](mtmv) Fix hudi materialized view union all rewritte…
a9361ea 
…n plan execute fail because of invalid slot #58643 (#58724)

picked from #58643
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@seawinde @nagisa-kunhah
[fix](mtmv) Fix hudi materialized view union all rewritten plan execu…
b743cfb 
…te fail because of invalid slot (apache#58643)

### What problem does this PR solve?

Related PR: apache#57558 apache#58413

Problem Summary:

This fix addresses the following three issues:
1. When invoking the method
org.apache.doris.nereids.trees.plans.logical.LogicalHudiScan#withRelationId,
the output needs to be recalculated to meet expectations.
2. After compensating with a union all due to partial partition
invalidation of a materialized view, during the next round of
transparent rewriting, the rewriting for the child of the union
allshould use the query partitioncorresponding to the specific relation
id to prevent infinite loops.
3. Currently, in the `test_hudi_rewrite_mtmv` test, if the plan
rewritten by the materialized view transparent rewriting is not selected
by the CBO, it is difficult to troubleshoot because explain memo planis
not used. Therefore, the corresponding test method is modified.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

+1 more reviewer

@zddr zddr zddr approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@seawinde @Thearas @doris-robot @hello-stephen @zddr @morrySnow @yiguolei