[fix](mtmv) Fix nested mv rewritten fail when bottom mv is partitioned #57558

seawinde · 2025-10-31T04:05:49Z

What problem does this PR solve?

In the scenario of nested materialized view rewriting, if the underlying materialized view is a partitioned materialized view, the rewriting of the upper-level materialized view would fail. This PR fixes the issue.

Issue Number: close #xxx

Related PR: #49514 #56958

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-10-31T04:05:54Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

seawinde · 2025-10-31T04:05:57Z

run buildall

doris-robot · 2025-10-31T04:46:49Z

ClickBench: Total hot run time: 29.03 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 10c594832444bf9648bdb42ddb434050fa24cbc1, data reload: false

query1	0.06	0.05	0.06
query2	0.11	0.06	0.05
query3	0.26	0.10	0.10
query4	1.61	0.12	0.12
query5	0.28	0.27	0.27
query6	1.16	0.68	0.69
query7	0.03	0.03	0.04
query8	0.06	0.04	0.05
query9	0.65	0.58	0.56
query10	0.63	0.63	0.63
query11	0.19	0.13	0.13
query12	0.19	0.13	0.14
query13	0.64	0.62	0.63
query14	1.04	1.03	1.04
query15	0.92	0.87	0.93
query16	0.44	0.41	0.43
query17	1.11	1.16	1.28
query18	0.24	0.21	0.22
query19	2.06	1.96	1.88
query20	0.02	0.02	0.01
query21	15.35	0.23	0.15
query22	4.83	0.08	0.06
query23	15.69	0.29	0.12
query24	2.43	0.56	0.34
query25	0.09	0.06	0.08
query26	0.17	0.16	0.15
query27	0.07	0.06	0.06
query28	3.91	1.20	0.96
query29	12.59	4.70	3.86
query30	0.29	0.17	0.12
query31	2.84	0.67	0.43
query32	3.25	0.58	0.48
query33	3.13	3.10	3.23
query34	15.84	5.29	4.65
query35	4.64	4.60	4.69
query36	0.73	0.57	0.52
query37	0.12	0.07	0.07
query38	0.07	0.04	0.05
query39	0.04	0.04	0.04
query40	0.18	0.15	0.14
query41	0.10	0.03	0.04
query42	0.05	0.03	0.03
query43	0.06	0.04	0.04
Total cold run time: 98.17 s
Total hot run time: 29.03 s

zddr · 2025-10-31T07:57:44Z

...e-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/PartitionCompensator.java

        Map<List<String>, Set<String>> queryUsedRelatedTablePartitionsMap = new HashMap<>();
        tableLoop:
-        for (Map.Entry<List<String>, TableIf> queryUsedTableEntry : statementContext.getTables().entrySet()) {
+        for (List<String> queryUsedTable : tableUsedPartitionNameMap.keySet()) {


why change to use this map

beacuse some table would not in tables in statement context, such as bottom mv which is not query directly in query, so not need to use tables in statement context,

zddr · 2025-10-31T08:00:45Z

...e-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/PartitionCompensator.java

-                        }
-                        usedPartitionSet.addAll(partitionPair.value());
+                    tableUsedPartitionNameMap.get(queryUsedTable);
+            if (ALL_PARTITIONS_LIST.equals(tableUsedPartitions)) {


Add some comments to explain what ALL-PARTITION-LIST is and what ALL-PARTITION-LIST. equals (tableUsedPieces) means

have added comment

zddr · 2025-10-31T08:00:59Z

...e-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/PartitionCompensator.java

+            }
+            for (Pair<RelationId, Set<String>> tableUsedPartitionPair : tableUsedPartitions) {
+                if (!customRelationIdSet.isEmpty()) {
+                    if (ALL_PARTITIONS.equals(tableUsedPartitionPair)) {


have added comment

zddr · 2025-10-31T08:04:16Z

regression-test/suites/nereids_rules_p0/mv/nested_mtmv/nested_mtmv.groovy

    mv_rewrite_any_success(sql_5, [mv_3, mv_4, mv_5])
    compare_res(sql_5 + " order by 1,2,3,4,5,6,7,8,9,10,11,12,13")

+    sql """


Add some case validation to supplement union situations, such as inserting a row of data into a partition of the base table

union situations not support query rewrite by materialized view currently, so not need to add

hello-stephen · 2025-10-31T09:20:36Z

FE Regression Coverage Report

Increment line coverage 72.73% (16/22) 🎉
Increment coverage report
Complete coverage report

seawinde · 2025-11-03T02:56:55Z

run buildall

morrySnow · 2025-11-03T04:05:41Z

fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRewriteUtil.java

+    /**
+     *
+     * */


what's this?

Complete the comments

doris-robot · 2025-11-03T04:35:33Z

TPC-DS: Total hot run time: 190359 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 15e327b15b6d46e2ce485bcd63bfb5f680cc8c14, data reload: false

query1	1066	422	405	405
query2	6553	1733	1699	1699
query3	6756	228	227	227
query4	26742	23388	23417	23388
query5	5514	645	466	466
query6	363	254	235	235
query7	4660	494	299	299
query8	310	265	265	265
query9	8736	2605	2626	2605
query10	576	350	297	297
query11	15844	15138	14858	14858
query12	185	121	119	119
query13	1697	584	454	454
query14	11756	9240	9329	9240
query15	246	198	175	175
query16	7735	680	528	528
query17	1639	788	625	625
query18	2098	513	409	409
query19	255	225	185	185
query20	142	138	126	126
query21	225	137	113	113
query22	4571	4616	4547	4547
query23	34820	33898	34099	33898
query24	8740	2470	2529	2470
query25	598	553	502	502
query26	748	275	168	168
query27	2757	551	373	373
query28	4512	2249	2248	2248
query29	728	803	511	511
query30	304	236	204	204
query31	914	851	788	788
query32	88	73	70	70
query33	581	428	346	346
query34	820	901	543	543
query35	871	852	786	786
query36	1004	1040	892	892
query37	124	110	82	82
query38	3564	3496	3483	3483
query39	1484	1424	1423	1423
query40	219	129	120	120
query41	74	58	60	58
query42	125	114	121	114
query43	487	504	467	467
query44	1212	738	738	738
query45	189	180	174	174
query46	890	996	627	627
query47	1769	1829	1751	1751
query48	398	425	318	318
query49	726	498	398	398
query50	645	686	409	409
query51	3941	4131	3834	3834
query52	108	106	104	104
query53	235	264	199	199
query54	311	284	268	268
query55	87	85	82	82
query56	316	309	311	309
query57	1157	1195	1133	1133
query58	291	307	272	272
query59	2556	2755	2585	2585
query60	338	336	311	311
query61	159	155	152	152
query62	798	729	704	704
query63	220	198	198	198
query64	3315	1193	888	888
query65	4062	3949	3941	3941
query66	890	424	361	361
query67	15575	15246	15115	15115
query68	8147	907	602	602
query69	506	321	288	288
query70	1351	1341	1289	1289
query71	431	336	308	308
query72	5868	4957	4940	4940
query73	640	568	358	358
query74	8929	9104	9020	9020
query75	3500	3364	2855	2855
query76	3365	1185	759	759
query77	560	416	307	307
query78	9591	9541	8960	8960
query79	2752	800	610	610
query80	697	569	505	505
query81	502	262	232	232
query82	480	161	132	132
query83	270	259	245	245
query84	258	112	92	92
query85	887	493	431	431
query86	365	306	298	298
query87	3761	3729	3533	3533
query88	3841	2234	2207	2207
query89	398	319	298	298
query90	2047	219	217	217
query91	163	161	145	145
query92	89	71	66	66
query93	2552	988	635	635
query94	750	437	356	356
query95	398	321	310	310
query96	485	573	278	278
query97	2936	2990	2896	2896
query98	243	222	211	211
query99	1357	1441	1294	1294
Total cold run time: 279538 ms
Total hot run time: 190359 ms

doris-robot · 2025-11-03T04:40:43Z

ClickBench: Total hot run time: 27.83 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 15e327b15b6d46e2ce485bcd63bfb5f680cc8c14, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.08	0.09
query4	1.61	0.11	0.12
query5	0.27	0.26	0.26
query6	1.17	0.63	0.65
query7	0.04	0.02	0.03
query8	0.05	0.05	0.04
query9	0.60	0.53	0.53
query10	0.58	0.58	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.60	0.60
query14	1.01	1.01	1.00
query15	0.86	0.84	0.85
query16	0.40	0.38	0.40
query17	1.02	1.06	1.06
query18	0.22	0.22	0.21
query19	2.00	1.80	1.78
query20	0.02	0.01	0.01
query21	15.48	0.18	0.13
query22	5.13	0.06	0.05
query23	15.68	0.27	0.10
query24	2.97	0.64	1.17
query25	0.08	0.06	0.07
query26	0.14	0.13	0.13
query27	0.07	0.05	0.06
query28	5.59	1.15	0.93
query29	12.55	3.87	3.18
query30	0.28	0.15	0.14
query31	2.82	0.58	0.38
query32	3.23	0.57	0.47
query33	3.01	3.15	3.05
query34	15.87	5.18	4.62
query35	4.59	4.59	4.61
query36	0.66	0.50	0.50
query37	0.10	0.06	0.06
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.04	0.03
query43	0.04	0.04	0.03
Total cold run time: 99.89 s
Total hot run time: 27.83 s

seawinde · 2025-11-03T08:35:21Z

run buildall

doris-robot · 2025-11-03T09:19:59Z

ClickBench: Total hot run time: 29.34 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 828aea8eecaba36467837bb39e5c0e7c09d2be41, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.06	0.06
query3	0.26	0.09	0.08
query4	1.61	0.12	0.12
query5	0.30	0.28	0.28
query6	1.21	0.70	0.68
query7	0.03	0.03	0.03
query8	0.07	0.05	0.04
query9	0.66	0.58	0.57
query10	0.63	0.63	0.61
query11	0.18	0.12	0.13
query12	0.18	0.14	0.13
query13	0.64	0.62	0.61
query14	1.03	1.04	1.04
query15	0.89	0.90	0.92
query16	0.45	0.44	0.41
query17	1.14	1.15	1.29
query18	0.24	0.22	0.23
query19	2.03	1.88	1.96
query20	0.02	0.01	0.01
query21	15.37	0.20	0.15
query22	5.08	0.08	0.07
query23	15.67	0.33	0.12
query24	2.75	0.75	0.91
query25	0.09	0.07	0.08
query26	0.17	0.17	0.16
query27	0.08	0.06	0.06
query28	4.38	1.20	0.94
query29	12.57	4.50	3.77
query30	0.30	0.16	0.12
query31	2.84	0.66	0.41
query32	3.28	0.59	0.48
query33	3.10	3.12	3.20
query34	15.91	5.30	4.59
query35	4.67	4.64	4.60
query36	0.71	0.54	0.52
query37	0.12	0.08	0.08
query38	0.07	0.05	0.05
query39	0.04	0.03	0.04
query40	0.18	0.17	0.14
query41	0.10	0.04	0.05
query42	0.05	0.04	0.04
query43	0.05	0.04	0.05
Total cold run time: 99.31 s
Total hot run time: 29.34 s

hello-stephen · 2025-11-04T12:05:21Z

FE Regression Coverage Report

Increment line coverage 81.25% (13/16) 🎉
Increment coverage report
Complete coverage report

github-actions · 2025-11-07T09:55:31Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-11-07T09:55:34Z

PR approved by anyone and no changes requested.

apache#57558) Related PR: apache#49514 apache#56958 Problem Summary: In the scenario of nested materialized view rewriting, if the underlying materialized view is a partitioned materialized view, the rewriting of the upper-level materialized view would fail. This PR fixes the issue.

… partitioned (#57558) (#58277) pr: #57558 commitId: 5e43141 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

… partitioned #57558 (#58276) picked from #57558

…te fail because of invalid slot (#58643) ### What problem does this PR solve? Related PR: #57558 #58413 Problem Summary: This fix addresses the following three issues: 1. When invoking the method org.apache.doris.nereids.trees.plans.logical.LogicalHudiScan#withRelationId, the output needs to be recalculated to meet expectations. 2. After compensating with a union all due to partial partition invalidation of a materialized view, during the next round of transparent rewriting, the rewriting for the child of the union allshould use the query partitioncorresponding to the specific relation id to prevent infinite loops. 3. Currently, in the `test_hudi_rewrite_mtmv` test, if the plan rewritten by the materialized view transparent rewriting is not selected by the CBO, it is difficult to troubleshoot because explain memo planis not used. Therefore, the corresponding test method is modified.

…te fail because of invalid slot (apache#58643) Related PR: apache#57558 apache#58413 Problem Summary: This fix addresses the following three issues: 1. When invoking the method org.apache.doris.nereids.trees.plans.logical.LogicalHudiScan#withRelationId, the output needs to be recalculated to meet expectations. 2. After compensating with a union all due to partial partition invalidation of a materialized view, during the next round of transparent rewriting, the rewriting for the child of the union allshould use the query partitioncorresponding to the specific relation id to prevent infinite loops. 3. Currently, in the `test_hudi_rewrite_mtmv` test, if the plan rewritten by the materialized view transparent rewriting is not selected by the CBO, it is difficult to troubleshoot because explain memo planis not used. Therefore, the corresponding test method is modified.

…te fail because of invalid slot (apache#58643) ### What problem does this PR solve? Related PR: apache#57558 apache#58413 Problem Summary: This fix addresses the following three issues: 1. When invoking the method org.apache.doris.nereids.trees.plans.logical.LogicalHudiScan#withRelationId, the output needs to be recalculated to meet expectations. 2. After compensating with a union all due to partial partition invalidation of a materialized view, during the next round of transparent rewriting, the rewriting for the child of the union allshould use the query partitioncorresponding to the specific relation id to prevent infinite loops. 3. Currently, in the `test_hudi_rewrite_mtmv` test, if the plan rewritten by the materialized view transparent rewriting is not selected by the CBO, it is difficult to troubleshoot because explain memo planis not used. Therefore, the corresponding test method is modified.

[fix](mtmv) Fix nested mv rewritten fail when bottom mv is partitioned

10c5948

zddr reviewed Oct 31, 2025

View reviewed changes

fix code review comment

15e327b

morrySnow reviewed Nov 3, 2025

View reviewed changes

fix regression test

828aea8

morrySnow approved these changes Nov 7, 2025

View reviewed changes

morrySnow added dev/3.1.x dev/4.0.x labels Nov 7, 2025

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 7, 2025

github-actions bot added the reviewed label Nov 7, 2025

zddr approved these changes Nov 7, 2025

View reviewed changes

morrySnow merged commit 54d3a63 into apache:master Nov 10, 2025
32 of 36 checks passed

github-actions bot added dev/3.1.x-conflict dev/4.0.x-conflict labels Nov 10, 2025

seawinde mentioned this pull request Nov 24, 2025

branch-3.1:[fix](mtmv) Fix nested mv rewritten fail when bottom mv is partitioned #57558 #58276

Merged

seawinde mentioned this pull request Nov 24, 2025

branch-4.0:[fix](mtmv) Fix nested mv rewritten fail when bottom mv is partitioned (#57558) #58277

Merged

16 tasks

yiguolei added dev/4.0.2-merged and removed dev/4.0.x dev/4.0.x-conflict labels Nov 25, 2025

morrySnow pushed a commit that referenced this pull request Nov 26, 2025

branch-3.1:[fix](mtmv) Fix nested mv rewritten fail when bottom mv is…

8866161

… partitioned #57558 (#58276) picked from #57558

morrySnow added dev/3.1.4-merged and removed dev/3.1.x dev/3.1.x-conflict labels Nov 26, 2025

seawinde mentioned this pull request Dec 2, 2025

[fix](mtmv) Fix hudi materialized view union all rewritten plan execute fail because of invalid slot #58643

Merged

16 tasks

[fix](mtmv) Fix nested mv rewritten fail when bottom mv is partitioned #57558

[fix](mtmv) Fix nested mv rewritten fail when bottom mv is partitioned #57558

Uh oh!

Conversation

seawinde commented Oct 31, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Oct 31, 2025

Uh oh!

seawinde commented Oct 31, 2025

Uh oh!

doris-robot commented Oct 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Oct 31, 2025

FE Regression Coverage Report

Uh oh!

seawinde commented Nov 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Nov 3, 2025

Uh oh!

doris-robot commented Nov 3, 2025

Uh oh!

seawinde commented Nov 3, 2025

Uh oh!

doris-robot commented Nov 3, 2025

Uh oh!

hello-stephen commented Nov 4, 2025

FE Regression Coverage Report

Uh oh!

github-actions bot commented Nov 7, 2025

Uh oh!

github-actions bot commented Nov 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants