Skip to content

Conversation

@seawinde
Copy link
Contributor

Proposed changes

pr: #41472
commitId: 2745e04

pr: #40106
commitId: 0fdb1ee

pr: #40173
commitId: 0d07e3d

pr: #42206
commitId: 2bcaa5b

…e is useless in some scene (apache#41472)

This is brought by apache#34050

if set `enable_materialized_view_nest_rewrite = false`, as expected, top
level materialized view should rewritten fail, but now successfully.

Such as first level materialized view def is

        CREATE MATERIALIZED VIEW level1
        BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
        DISTRIBUTED BY RANDOM BUCKETS 2
        PROPERTIES ('replication_num' = '1') 
        AS
    SELECT l_orderkey, l_linenumber, l_partkey, o_orderkey, o_custkey
    FROM lineitem_2 INNER JOIN orders_2
    ON l_orderkey = o_orderkey;

second level materialized view def is

        CREATE MATERIALIZED VIEW level2
        BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
        DISTRIBUTED BY RANDOM BUCKETS 2
        PROPERTIES ('replication_num' = '1') 
        AS
    SELECT
    l_orderkey,
    l_linenumber,
    o_orderkey,
    sum(l_partkey) AS total_revenue,
    max(o_custkey) AS max_discount
    FROM join_mv1
    GROUP BY l_orderkey, l_linenumber, o_orderkey;

if set `enable_materialized_view_nest_rewrite = false`, only `level1`
can rewriten succesfully and chosen by cbo
if set `enable_materialized_view_nest_rewrite = true`, both `level1` and
`level2` can rewriten succesfully and `level2` should be chosen by cbo.

This pr fixed this
…ize view (apache#40106)

In BI or some other SQL client,it will set `sql_select_limit` or
`default_order_by_limit` session variables automatically.
This will make mistake when query rewrite by materialized view, Because
after query rewrite by materialized view, the rewritten plan would be
optimized by rules again which should add limit on SQL.
This pr fix this.
…e mv, data maybe wrong (apache#40173)

When refresh partition mv, meanwhile, the query that can be written
successfully by the same mv, if we run the query, the result maybe
wrong.
this pr fix this.
…pache#42206)

This is brought by apache#32743

set enable_unicode_name_support = true;
If run create view sql should fail beausel_shipdate column name contains
invalid char '(' and ')', but now success
this pr fix this and throw exception
`ERROR 1105 (HY000): errCode = 2, detailMessage = Incorrect column name
'(日期)'. Column regex is
'^[_a-zA-Z@0-9\s/][.a-zA-Z0-9_+-/?@#$%^&*"\s,:]{0,255}$'`


CREATE VIEW view1
AS
SELECT "零售公司", l_shipdate as '(日期)', l_receiptdate as k2
FROM lineitem;

and if run create view sql as following, should success:

CREATE VIEW view2
AS
SELECT "零售公司", l_shipdate as '日期', l_receiptdate as k2
FROM lineitem;

and the schema of view2 should be

mysql> desc view2;
+-------------+-------------+------+-------+---------+-------+
| Field       | Type        | Null | Key   | Default | Extra |
+-------------+-------------+------+-------+---------+-------+
| __literal_0 | varchar(16) | No   | false | NULL    |       |
| 日期        | date        | No   | false | NULL    |       |
| k2          | date        | No   | false | NULL    |       |
+-------------+-------------+------+-------+---------+-------+
3 rows in set (0.01 sec)
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@seawinde
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40353 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 53813f3a0f4de39291acf27d8ae6eea464f1f9c8, data reload: false

------ Round 1 ----------------------------------
q1	17596	7349	7265	7265
q2	2016	273	270	270
q3	12178	1060	1157	1060
q4	10560	768	743	743
q5	7749	2821	2719	2719
q6	239	151	153	151
q7	991	614	618	614
q8	9568	1891	1922	1891
q9	7730	6449	6368	6368
q10	6993	2248	2345	2248
q11	435	253	242	242
q12	409	223	225	223
q13	17786	2942	2994	2942
q14	240	209	205	205
q15	550	532	515	515
q16	673	598	588	588
q17	956	563	504	504
q18	7143	6529	6518	6518
q19	1397	1036	1031	1031
q20	551	296	263	263
q21	3917	3020	3217	3020
q22	1091	977	973	973
Total cold run time: 110768 ms
Total hot run time: 40353 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7286	7284	7237	7237
q2	318	221	222	221
q3	2998	2831	2848	2831
q4	1995	1767	1736	1736
q5	5695	5683	5715	5683
q6	253	153	149	149
q7	2215	1792	1790	1790
q8	3298	3498	3468	3468
q9	8781	8852	8821	8821
q10	3542	3533	3483	3483
q11	581	478	493	478
q12	778	589	600	589
q13	16469	3143	3178	3143
q14	320	265	279	265
q15	556	529	537	529
q16	720	660	672	660
q17	1820	1641	1601	1601
q18	8292	7712	7479	7479
q19	7990	1535	1633	1535
q20	2154	1864	1904	1864
q21	5337	5336	5272	5272
q22	1116	1011	1011	1011
Total cold run time: 82514 ms
Total hot run time: 59845 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189907 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 53813f3a0f4de39291acf27d8ae6eea464f1f9c8, data reload: false

query1	938	386	385	385
query2	6257	2129	2007	2007
query3	8702	201	201	201
query4	33614	23397	23436	23397
query5	3713	493	456	456
query6	275	176	171	171
query7	4196	315	303	303
query8	262	213	215	213
query9	9563	2713	2681	2681
query10	464	292	307	292
query11	17801	15158	15125	15125
query12	141	97	100	97
query13	1523	407	409	407
query14	9768	6967	7492	6967
query15	224	176	171	171
query16	7702	435	466	435
query17	1547	594	592	592
query18	1954	328	309	309
query19	223	156	154	154
query20	121	117	110	110
query21	216	107	118	107
query22	4628	4604	4262	4262
query23	35050	33962	33666	33666
query24	11537	2900	2977	2900
query25	614	396	409	396
query26	724	162	161	161
query27	2105	305	298	298
query28	6532	2493	2472	2472
query29	809	434	433	433
query30	288	156	161	156
query31	1040	774	844	774
query32	100	54	54	54
query33	796	309	325	309
query34	933	499	527	499
query35	904	733	741	733
query36	1114	937	950	937
query37	143	94	83	83
query38	4047	3906	3855	3855
query39	1474	1418	1384	1384
query40	205	101	99	99
query41	51	47	51	47
query42	126	94	105	94
query43	540	492	495	492
query44	1212	774	785	774
query45	194	163	165	163
query46	1137	724	741	724
query47	1880	1784	1807	1784
query48	459	371	361	361
query49	921	391	384	384
query50	823	404	426	404
query51	7024	6978	6947	6947
query52	105	92	90	90
query53	270	192	189	189
query54	1298	466	473	466
query55	78	75	78	75
query56	287	264	289	264
query57	1174	1094	1089	1089
query58	228	231	231	231
query59	3176	2930	2876	2876
query60	293	279	262	262
query61	101	103	101	101
query62	851	646	661	646
query63	225	190	187	187
query64	3689	628	593	593
query65	3244	3296	3179	3179
query66	874	303	304	303
query67	15860	15307	15295	15295
query68	4553	544	546	544
query69	663	285	312	285
query70	1187	1154	1114	1114
query71	448	277	278	277
query72	7834	3853	3927	3853
query73	800	347	337	337
query74	10359	8899	8861	8861
query75	4631	2639	2667	2639
query76	3513	913	887	887
query77	735	299	292	292
query78	10044	9220	9150	9150
query79	3869	596	585	585
query80	1761	444	459	444
query81	594	241	245	241
query82	938	149	143	143
query83	370	136	152	136
query84	294	81	81	81
query85	1983	289	286	286
query86	469	312	297	297
query87	4485	4239	4282	4239
query88	4396	2401	2417	2401
query89	409	286	291	286
query90	2077	187	185	185
query91	172	139	142	139
query92	65	49	48	48
query93	5154	530	530	530
query94	1071	280	290	280
query95	349	256	247	247
query96	666	286	292	286
query97	3335	3229	3281	3229
query98	223	200	199	199
query99	1586	1351	1315	1315
Total cold run time: 307400 ms
Total hot run time: 189907 ms

@morrySnow morrySnow changed the title Pick some fix from master to 30(#41472)(#40106)(#40173)(#42206) [pick](mtmv) pick 4 PR from master #41472 #40106 #40173 #42206 Oct 24, 2024
@morrySnow morrySnow merged commit a1690f0 into apache:branch-3.0 Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants