[fix](mtmv) Fix compensate union wrongly when direct query is empty relation #51700
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
morrySnow
reviewed
Jun 13, 2025
| return context.getRewritePlan(); | ||
| }, queryPlanWithUnionFilter, queryPlan), true); | ||
| }, queryPlanWithUnionFilter, queryPlan); | ||
| // need to collect table partition again |
Contributor
There was a problem hiding this comment.
write why need re-collect here in comment
Contributor
Author
There was a problem hiding this comment.
have added comment
| // this is needed before init hook | ||
| collectTableUsedPartitions(showPlanProcess); | ||
| // this is needed before init hook, because | ||
| MaterializedViewUtils.collectTableUsedPartitions(cascadesContext.getRewritePlan(), cascadesContext); |
| }, queryPlanWithUnionFilter, queryPlan), true); | ||
| }, queryPlanWithUnionFilter, queryPlan); | ||
| // need to collect table partition again | ||
| MaterializedViewUtils.collectTableUsedPartitions(filterAddedPlan, parentCascadesContext); |
Contributor
There was a problem hiding this comment.
if u need collect partition info multi times, u should sum the time used by it in summary profile
| }, queryPlanWithUnionFilter, queryPlan), true); | ||
| }, queryPlanWithUnionFilter, queryPlan); | ||
| // need to collect table partition again | ||
| MaterializedViewUtils.collectTableUsedPartitions(filterAddedPlan, parentCascadesContext); |
Contributor
There was a problem hiding this comment.
only collect for sub plan is enough?
Contributor
Author
There was a problem hiding this comment.
should collect table partition for all rewritten plan by mv, have fixed it
TPC-H: Total hot run time: 35905 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
|
run external 10 |
TPC-H: Total hot run time: 36238 ms |
Contributor
|
run external 5 |
4a2a8c6 to
b95c035
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34052 ms |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33763 ms |
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run external 10 |
TPC-H: Total hot run time: 34190 ms |
TPC-DS: Total hot run time: 184911 ms |
ClickBench: Total hot run time: 28.89 s |
Contributor
Author
|
run buildall |
c9108e7 to
fe19d56
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33682 ms |
morningman
approved these changes
Jun 18, 2025
seawinde
added a commit
to seawinde/doris
that referenced
this pull request
Jun 18, 2025
…elation (apache#51700) if query is as following which contains external table, if some partition in external table is empty, when mv partiton is invalid, this would cause compensate many union all empty relation ```sql SELECT * FROM ${catalogName}.`test_paimon_spark`.test_tb_mix_format a left join ${tableName} b on a.id=b.user_id; ```
seawinde
added a commit
to seawinde/doris
that referenced
this pull request
Jun 18, 2025
…elation (apache#51700) if query is as following which contains external table, if some partition in external table is empty, when mv partiton is invalid, this would cause compensate many union all empty relation ```sql SELECT * FROM ${catalogName}.`test_paimon_spark`.test_tb_mix_format a left join ${tableName} b on a.id=b.user_id; ```
yiguolei
pushed a commit
that referenced
this pull request
Jun 20, 2025
seawinde
added a commit
to seawinde/doris
that referenced
this pull request
Jun 26, 2025
…elation (apache#51700) if query is as following which contains external table, if some partition in external table is empty, when mv partiton is invalid, this would cause compensate many union all empty relation ```sql SELECT * FROM ${catalogName}.`test_paimon_spark`.test_tb_mix_format a left join ${tableName} b on a.id=b.user_id; ```
15 tasks
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jul 3, 2025
…y is empty relation apache#51700 (apache#51898) Cherry-picked apache#51700
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
if query is as following which contains external table, if some partition in external table is empty, when mv partiton is invalid, this would cause compensate many union all empty relation
Issue Number: close #xxx
Related PR: #49514
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)