Skip to content

[opt](mtmv) optimize mtmv rewrite performance (#49514) #51312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 4 commits into from
Jun 18, 2025
Merged

[opt](mtmv) optimize mtmv rewrite performance (#49514) #51312

dataroaring merged 4 commits into from
Jun 18, 2025

Conversation

@seawinde
Copy link
Contributor

@seawinde seawinde commented May 28, 2025

What problem does this PR solve?

commitId: 4da1c8a
pr: #49514

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@seawinde @zddr
[opt](mtmv) optimize mtmv rewrite performance (apache#49514)
1dbdb9a 
- Obtaining available materialized view adjustments after partition
pruning RBO rewrite rules. This reduces partition version comparison
overhead for partitioned materializations, thereby improving
performance.

- Set maximum time threshold for transparent query rewriting with
default value of 1000ms. Rewrite attempts will be terminated if
exceeding this threshold.
set materialized_view_rewrite_duration_threshold_ms = 1000

- Optimize code structure in transparent rewriting framework:
    1. Replace Lambda expressions with conventional for-loops
    2. Remove redundant member variables

- Cache available partitions of materialized views. Since calculating
available partitions for materialized views is time-consuming,
caching can significantly enhance performance.

Co-authored-by: zhangdong <zhangdong@selectdb.com>
@seawinde seawinde requested a review from dataroaring as a code owner May 28, 2025 03:59
@Thearas
Copy link
Contributor

Thearas commented May 28, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@seawinde
Copy link
Contributor Author

seawinde commented May 28, 2025

run buildall

@seawinde
Copy link
Contributor Author

seawinde commented May 28, 2025

run buildall

@seawinde
Copy link
Contributor Author

seawinde commented May 29, 2025

run buildall

@seawinde
Copy link
Contributor Author

seawinde commented May 30, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 30, 2025

TPC-H: Total hot run time: 39643 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 975695376b11181de6c0c632d5e1acf88c20b51b, data reload: false

------ Round 1 ----------------------------------
q1	17593	6705	6555	6555
q2	2053	166	197	166
q3	10546	1058	1150	1058
q4	10579	726	745	726
q5	7744	2839	2813	2813
q6	212	133	130	130
q7	959	628	616	616
q8	9360	1964	2025	1964
q9	6582	6384	6383	6383
q10	7024	2240	2292	2240
q11	468	266	255	255
q12	394	213	203	203
q13	17788	2933	2949	2933
q14	234	204	206	204
q15	515	456	455	455
q16	482	381	374	374
q17	980	637	590	590
q18	7284	6726	6650	6650
q19	1389	1076	1042	1042
q20	455	198	201	198
q21	3989	3102	3217	3102
q22	1120	1011	986	986
Total cold run time: 107750 ms
Total hot run time: 39643 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6544	6517	6521	6517
q2	326	234	227	227
q3	2921	2690	2770	2690
q4	2016	1805	1771	1771
q5	5750	5771	5712	5712
q6	207	123	129	123
q7	2229	1792	1809	1792
q8	3366	3517	3542	3517
q9	8953	8771	8868	8771
q10	3595	3524	3538	3524
q11	590	483	476	476
q12	770	596	652	596
q13	9401	3169	3114	3114
q14	309	294	291	291
q15	510	462	457	457
q16	481	435	432	432
q17	1822	1627	1604	1604
q18	8141	7981	7737	7737
q19	1702	1596	1563	1563
q20	2138	1830	1820	1820
q21	5495	5285	5322	5285
q22	1127	1069	1025	1025
Total cold run time: 68393 ms
Total hot run time: 59044 ms

@doris-robot
Copy link

doris-robot commented May 30, 2025

TPC-DS: Total hot run time: 196191 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 975695376b11181de6c0c632d5e1acf88c20b51b, data reload: false

query1	1267	907	925	907
query2	6346	1898	1878	1878
query3	10883	4373	4428	4373
query4	61459	28669	23389	23389
query5	5202	463	463	463
query6	412	187	185	185
query7	5513	314	309	309
query8	319	227	218	218
query9	8718	2585	2568	2568
query10	456	279	263	263
query11	17471	15137	15664	15137
query12	160	106	102	102
query13	1394	432	419	419
query14	10251	6601	6699	6601
query15	200	189	172	172
query16	7160	445	489	445
query17	1213	591	599	591
query18	1868	334	310	310
query19	235	168	156	156
query20	117	113	111	111
query21	204	102	105	102
query22	4712	4488	4659	4488
query23	34756	34209	34236	34209
query24	6256	3007	2939	2939
query25	536	426	435	426
query26	673	167	164	164
query27	2363	367	348	348
query28	4335	2164	2133	2133
query29	711	475	463	463
query30	247	159	157	157
query31	956	829	830	829
query32	68	60	59	59
query33	481	284	293	284
query34	895	508	519	508
query35	857	738	721	721
query36	1097	954	973	954
query37	111	66	71	66
query38	4086	3958	3973	3958
query39	1507	1480	1450	1450
query40	208	97	101	97
query41	50	51	49	49
query42	124	100	98	98
query43	526	473	476	473
query44	1158	804	813	804
query45	179	171	173	171
query46	1158	766	741	741
query47	2061	1934	1987	1934
query48	477	386	387	386
query49	734	397	404	397
query50	832	438	417	417
query51	7395	7268	7231	7231
query52	100	88	87	87
query53	260	186	186	186
query54	575	474	458	458
query55	77	81	80	80
query56	265	261	255	255
query57	1305	1168	1147	1147
query58	223	221	231	221
query59	3290	2982	3029	2982
query60	283	260	257	257
query61	109	121	109	109
query62	775	684	662	662
query63	220	185	183	183
query64	1370	668	636	636
query65	3323	3179	3185	3179
query66	675	289	295	289
query67	15992	15567	15374	15374
query68	4228	589	562	562
query69	425	269	266	266
query70	1164	1111	1065	1065
query71	322	267	247	247
query72	6399	4027	4004	4004
query73	741	356	353	353
query74	10190	9129	9145	9129
query75	3341	2678	2642	2642
query76	1908	1048	1074	1048
query77	511	267	265	265
query78	10514	9533	9591	9533
query79	1987	578	604	578
query80	1364	417	433	417
query81	535	227	215	215
query82	1274	91	89	89
query83	273	142	144	142
query84	288	79	80	79
query85	1011	303	292	292
query86	382	286	307	286
query87	4372	4215	4234	4215
query88	3989	2403	2384	2384
query89	405	283	287	283
query90	1895	184	188	184
query91	178	150	152	150
query92	65	48	49	48
query93	2763	547	544	544
query94	755	301	279	279
query95	355	257	260	257
query96	620	279	286	279
query97	3311	3141	3130	3130
query98	215	192	195	192
query99	1609	1302	1276	1276
Total cold run time: 316628 ms
Total hot run time: 196191 ms

@doris-robot
Copy link

doris-robot commented May 30, 2025

ClickBench: Total hot run time: 29.43 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 975695376b11181de6c0c632d5e1acf88c20b51b, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.06	0.06
query4	1.63	0.10	0.10
query5	0.50	0.53	0.50
query6	1.14	0.73	0.73
query7	0.02	0.01	0.02
query8	0.04	0.05	0.03
query9	0.56	0.51	0.48
query10	0.55	0.54	0.56
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.60	0.60	0.59
query14	0.78	0.77	0.81
query15	0.83	0.82	0.82
query16	0.41	0.38	0.37
query17	0.96	1.02	1.04
query18	0.23	0.21	0.21
query19	1.86	1.83	1.91
query20	0.02	0.01	0.01
query21	15.39	0.62	0.56
query22	2.94	3.17	1.91
query23	17.10	1.00	0.75
query24	3.86	1.36	0.19
query25	0.28	0.17	0.10
query26	0.32	0.14	0.14
query27	0.04	0.05	0.04
query28	10.35	0.53	0.45
query29	12.60	3.25	3.25
query30	0.24	0.07	0.06
query31	2.88	0.39	0.38
query32	3.25	0.47	0.46
query33	2.92	2.97	3.01
query34	16.82	4.52	4.53
query35	4.54	4.57	4.54
query36	0.65	0.48	0.47
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.15	0.12	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 105.41 s
Total hot run time: 29.43 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 4, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 4, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 4, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jun 18, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 0922e52 into apache:branch-3.0 Jun 18, 2025
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@seawinde @Thearas @doris-robot @dataroaring @morrySnow