-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[Opt](multi-catalog)Disable dict filter in parquet/orc reader if have non-single conjuncts. #44777
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Opt](multi-catalog)Disable dict filter in parquet/orc reader if have non-single conjuncts. #44777
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
6b0d1b8 to
1f8ccb3
Compare
|
run buildall |
1f8ccb3 to
3c6b546
Compare
|
run buildall |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 40268 ms |
TPC-DS: Total hot run time: 197585 ms |
ClickBench: Total hot run time: 32.7 s |
|
run cloud_p0 |
3c6b546 to
8640b08
Compare
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression P0 && UT Coverage ReportIncrement line coverage Increment coverage report
|
… non-single conjuncts.
8640b08 to
1c37449
Compare
|
run buildall |
TPC-H: Total hot run time: 33738 ms |
TPC-DS: Total hot run time: 193042 ms |
ClickBench: Total hot run time: 28.79 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
morningman
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
morningman
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
… non-single conjuncts. (apache#44777) Related PR: apache#26386 Problem Summary: Because of apache#26386, we split the conjunctions into `single_slot_filter_conjuncts` and `non_single_slot_filter_conjuncts`, where you can use the final dictionary filtering and delayed materialization of the `single_slot_filter_conjuncts` part to perform `non_single_slot_filter_conjuncts`. However, this results in fewer conditions for late materialization, so the effect will be poor.
… non-single conjuncts. (apache#44777) ### What problem does this PR solve? Related PR: apache#26386 Problem Summary: Because of apache#26386, we split the conjunctions into `single_slot_filter_conjuncts` and `non_single_slot_filter_conjuncts`, where you can use the final dictionary filtering and delayed materialization of the `single_slot_filter_conjuncts` part to perform `non_single_slot_filter_conjuncts`. However, this results in fewer conditions for late materialization, so the effect will be poor.
… non-single conjuncts. (apache#44777) Related PR: apache#26386 Problem Summary: Because of apache#26386, we split the conjunctions into `single_slot_filter_conjuncts` and `non_single_slot_filter_conjuncts`, where you can use the final dictionary filtering and delayed materialization of the `single_slot_filter_conjuncts` part to perform `non_single_slot_filter_conjuncts`. However, this results in fewer conditions for late materialization, so the effect will be poor.
… non-single conjuncts. (apache#44777) Related PR: apache#26386 Problem Summary: Because of apache#26386, we split the conjunctions into `single_slot_filter_conjuncts` and `non_single_slot_filter_conjuncts`, where you can use the final dictionary filtering and delayed materialization of the `single_slot_filter_conjuncts` part to perform `non_single_slot_filter_conjuncts`. However, this results in fewer conditions for late materialization, so the effect will be poor.
…ader if have non-single conjuncts apache#44777 (apache#52559) Cherry-picked from apache#44777
… non-single conjuncts. (apache#44777) Related PR: apache#26386 Problem Summary: Because of apache#26386, we split the conjunctions into `single_slot_filter_conjuncts` and `non_single_slot_filter_conjuncts`, where you can use the final dictionary filtering and delayed materialization of the `single_slot_filter_conjuncts` part to perform `non_single_slot_filter_conjuncts`. However, this results in fewer conditions for late materialization, so the effect will be poor.
What problem does this PR solve?
Related PR: #26386
Problem Summary:
Because of #26386, we split the conjunctions into
single_slot_filter_conjunctsandnon_single_slot_filter_conjuncts, where you can use the final dictionary filtering and delayed materialization of thesingle_slot_filter_conjunctspart to performnon_single_slot_filter_conjuncts. However, this results in fewer conditions for late materialization, so the effect will be poor.Release note
At present, it is a matter of how many conditions there are for late materialization and whether to perform dictionary filtering on multiple columns. Because late materialization is more important, we use when there are
non_single_slot_filter_conjunctsin the filter, directly without dictionary filtering, and use the conjuncts to do late materialization.None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)