Skip to content

branch-3.0: [Opt](mow) Forbid time_series compaction policy on unique table #49905 #50128

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Apr 27, 2025
Merged

branch-3.0: [Opt](mow) Forbid time_series compaction policy on unique table #49905 #50128

dataroaring merged 1 commit into from
Apr 27, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Apr 17, 2025

Cherry-picked from #49905

@github-actions github-actions bot requested a review from dataroaring as a code owner April 17, 2025 06:24
@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Apr 17, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Apr 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 17, 2025

TPC-H: Total hot run time: 39899 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3606bbfd795c6c268ad7e412f7089f941a9b3031, data reload: false

------ Round 1 ----------------------------------
q1	17563	6758	6616	6616
q2	2078	171	162	162
q3	10645	1082	1172	1082
q4	10561	769	816	769
q5	7752	2811	2797	2797
q6	217	132	131	131
q7	981	607	598	598
q8	9347	1914	2041	1914
q9	6519	6339	6364	6339
q10	7034	2262	2319	2262
q11	474	259	261	259
q12	404	204	203	203
q13	17778	2946	2996	2946
q14	234	216	207	207
q15	507	457	464	457
q16	668	582	571	571
q17	969	514	594	514
q18	7279	6669	6696	6669
q19	1398	1095	1004	1004
q20	466	209	206	206
q21	3884	3251	3210	3210
q22	1047	994	983	983
Total cold run time: 107805 ms
Total hot run time: 39899 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6637	6583	6566	6566
q2	339	240	233	233
q3	2951	2875	2927	2875
q4	2101	1906	1830	1830
q5	5722	5772	5683	5683
q6	204	129	126	126
q7	2189	1791	1867	1791
q8	3364	3536	3504	3504
q9	8828	8942	8847	8847
q10	3550	3546	3486	3486
q11	605	488	490	488
q12	813	613	580	580
q13	9171	3186	3150	3150
q14	311	289	283	283
q15	504	469	455	455
q16	677	640	637	637
q17	1826	1633	1610	1610
q18	8312	7845	7781	7781
q19	1662	1574	1350	1350
q20	2102	1855	1883	1855
q21	5494	5360	5340	5340
q22	1129	1031	1009	1009
Total cold run time: 68491 ms
Total hot run time: 59479 ms

@doris-robot
Copy link

doris-robot commented Apr 17, 2025

TPC-DS: Total hot run time: 198019 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3606bbfd795c6c268ad7e412f7089f941a9b3031, data reload: false

query1	1318	952	944	944
query2	6232	2127	2090	2090
query3	10876	4435	4480	4435
query4	65979	28373	24111	24111
query5	4989	477	473	473
query6	421	185	178	178
query7	5688	323	308	308
query8	296	221	212	212
query9	9151	2587	2573	2573
query10	458	265	251	251
query11	17647	15235	15731	15235
query12	156	104	105	104
query13	1549	450	464	450
query14	10579	7012	7215	7012
query15	210	187	190	187
query16	7135	522	511	511
query17	1063	610	621	610
query18	1867	329	338	329
query19	266	161	157	157
query20	124	110	110	110
query21	208	102	110	102
query22	4635	4432	4404	4404
query23	34091	33968	33979	33968
query24	6145	3004	2959	2959
query25	522	401	397	397
query26	655	172	173	172
query27	1836	360	355	355
query28	4201	2453	2440	2440
query29	692	441	438	438
query30	260	169	170	169
query31	980	851	827	827
query32	70	54	57	54
query33	426	297	286	286
query34	908	516	537	516
query35	858	752	721	721
query36	1062	965	944	944
query37	115	67	68	67
query38	4050	3950	4005	3950
query39	1534	1477	1461	1461
query40	201	97	105	97
query41	49	54	51	51
query42	113	102	100	100
query43	536	493	475	475
query44	1172	848	833	833
query45	187	168	166	166
query46	1154	742	738	738
query47	2016	1910	1901	1901
query48	487	405	403	403
query49	782	418	389	389
query50	859	435	429	429
query51	7334	7108	7067	7067
query52	100	95	89	89
query53	269	180	181	180
query54	561	463	460	460
query55	76	76	79	76
query56	263	245	240	240
query57	1220	1099	1092	1092
query58	209	211	205	205
query59	3100	2890	2922	2890
query60	288	271	266	266
query61	116	115	109	109
query62	794	679	672	672
query63	228	198	190	190
query64	1382	684	646	646
query65	3285	3238	3201	3201
query66	703	296	295	295
query67	15917	15485	15772	15485
query68	3321	627	601	601
query69	428	261	267	261
query70	1171	1070	1097	1070
query71	352	262	252	252
query72	6246	4170	4092	4092
query73	762	351	351	351
query74	10135	9084	9321	9084
query75	3399	2645	2725	2645
query76	1922	1003	1131	1003
query77	463	288	293	288
query78	10483	9566	9601	9566
query79	1256	621	601	601
query80	936	470	457	457
query81	518	246	238	238
query82	1262	95	92	92
query83	251	155	148	148
query84	292	86	81	81
query85	962	373	357	357
query86	355	305	286	286
query87	4421	4270	4254	4254
query88	3491	2413	2367	2367
query89	423	297	293	293
query90	1985	184	187	184
query91	185	155	150	150
query92	72	50	49	49
query93	1594	566	565	565
query94	822	304	311	304
query95	361	263	265	263
query96	620	283	279	279
query97	3328	3123	3161	3123
query98	220	192	193	192
query99	1540	1296	1268	1268
Total cold run time: 316229 ms
Total hot run time: 198019 ms

@doris-robot
Copy link

doris-robot commented Apr 17, 2025

ClickBench: Total hot run time: 31.69 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3606bbfd795c6c268ad7e412f7089f941a9b3031, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.04
query3	0.23	0.06	0.06
query4	1.62	0.10	0.11
query5	0.51	0.49	0.49
query6	1.13	0.72	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.49
query10	0.55	0.54	0.54
query11	0.13	0.10	0.11
query12	0.13	0.11	0.11
query13	0.60	0.59	0.59
query14	2.72	2.87	2.86
query15	0.88	0.82	0.83
query16	0.38	0.39	0.41
query17	1.01	1.08	1.09
query18	0.23	0.22	0.22
query19	1.89	1.81	2.01
query20	0.02	0.01	0.00
query21	15.37	0.59	0.56
query22	2.50	2.49	1.61
query23	16.79	0.86	0.84
query24	3.09	1.71	0.67
query25	0.18	0.21	0.12
query26	0.48	0.14	0.13
query27	0.05	0.04	0.06
query28	10.29	0.53	0.46
query29	12.57	3.20	3.22
query30	0.24	0.06	0.07
query31	2.85	0.39	0.39
query32	3.23	0.45	0.44
query33	3.01	2.96	3.02
query34	17.11	4.50	4.49
query35	4.55	4.50	4.47
query36	0.66	0.48	0.50
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.21 s
Total hot run time: 31.69 s

@bobhan1
Copy link
Contributor

bobhan1 commented Apr 23, 2025

run cloud_p0

3 similar comments
@bobhan1
Copy link
Contributor

bobhan1 commented Apr 23, 2025

run cloud_p0

@bobhan1
Copy link
Contributor

bobhan1 commented Apr 23, 2025

run cloud_p0

@bobhan1
Copy link
Contributor

bobhan1 commented Apr 24, 2025

run cloud_p0

dataroaring
dataroaring approved these changes Apr 25, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bobhan1 @dataroaring
[Opt](mow) Forbid time_series compaction policy on unique table (#49905)
18f43b0 
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [x] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [x] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@dataroaring dataroaring force-pushed the auto-pick-49905-branch-3.0 branch from 3606bbf to 18f43b0 Compare April 25, 2025 11:24
@dataroaring
Copy link
Contributor

dataroaring commented Apr 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 25, 2025

TPC-H: Total hot run time: 39928 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 18f43b0bdcfecfc3c5eb91a07d1ca20359bad150, data reload: false

------ Round 1 ----------------------------------
q1	17574	6797	6591	6591
q2	2056	176	181	176
q3	10575	1069	1140	1069
q4	10571	746	701	701
q5	7762	2857	2813	2813
q6	216	129	129	129
q7	945	613	606	606
q8	9364	1952	2019	1952
q9	6620	6405	6404	6404
q10	6991	2254	2255	2254
q11	477	269	263	263
q12	405	213	208	208
q13	17784	2962	2998	2962
q14	238	207	210	207
q15	496	446	471	446
q16	676	591	591	591
q17	987	602	529	529
q18	7267	6688	6588	6588
q19	1416	1070	1127	1070
q20	474	209	201	201
q21	3963	3277	3198	3198
q22	1058	986	970	970
Total cold run time: 107915 ms
Total hot run time: 39928 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6534	6536	6504	6504
q2	330	231	234	231
q3	2906	2726	2918	2726
q4	2000	1754	1852	1754
q5	5771	5747	5741	5741
q6	206	127	132	127
q7	2184	1844	1861	1844
q8	3354	3555	3543	3543
q9	8970	8780	8864	8780
q10	3535	3484	3481	3481
q11	584	481	484	481
q12	801	597	618	597
q13	9560	3112	3139	3112
q14	292	260	269	260
q15	506	456	461	456
q16	700	637	635	635
q17	1843	1639	1605	1605
q18	8311	7685	7796	7685
q19	1735	1525	1543	1525
q20	2037	1812	1806	1806
q21	5505	5475	5338	5338
q22	1097	1056	1030	1030
Total cold run time: 68761 ms
Total hot run time: 59261 ms

@doris-robot
Copy link

doris-robot commented Apr 25, 2025

TPC-DS: Total hot run time: 196716 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 18f43b0bdcfecfc3c5eb91a07d1ca20359bad150, data reload: false

query1	1272	906	975	906
query2	6265	2013	2026	2013
query3	10834	4308	4198	4198
query4	61622	29279	23547	23547
query5	5299	469	435	435
query6	396	174	184	174
query7	5454	297	309	297
query8	298	218	214	214
query9	8519	2623	2623	2623
query10	458	270	246	246
query11	17432	15185	15662	15185
query12	158	102	107	102
query13	1418	452	433	433
query14	10362	7109	7234	7109
query15	199	185	167	167
query16	7160	482	471	471
query17	1252	572	589	572
query18	1792	335	323	323
query19	205	164	159	159
query20	117	115	108	108
query21	214	100	105	100
query22	4533	4543	4599	4543
query23	34711	34075	34139	34075
query24	6108	2901	2859	2859
query25	537	429	432	429
query26	657	164	164	164
query27	2023	366	363	363
query28	4837	2467	2456	2456
query29	698	481	431	431
query30	256	168	161	161
query31	966	831	815	815
query32	67	58	57	57
query33	450	284	282	282
query34	911	506	534	506
query35	833	748	730	730
query36	1054	944	969	944
query37	117	70	72	70
query38	4002	3935	3901	3901
query39	1531	1487	1450	1450
query40	216	105	100	100
query41	48	47	46	46
query42	121	103	99	99
query43	533	490	483	483
query44	1185	826	830	826
query45	186	165	169	165
query46	1166	710	724	710
query47	2050	1931	1945	1931
query48	478	381	384	381
query49	745	380	397	380
query50	839	442	434	434
query51	7408	7264	7274	7264
query52	102	90	88	88
query53	253	189	183	183
query54	564	462	464	462
query55	80	79	80	79
query56	256	250	230	230
query57	1223	1178	1154	1154
query58	209	206	217	206
query59	3202	3114	3078	3078
query60	276	258	261	258
query61	115	112	106	106
query62	777	709	707	707
query63	231	187	191	187
query64	1455	704	654	654
query65	3270	3195	3175	3175
query66	728	295	298	295
query67	15791	15476	15607	15476
query68	4172	596	574	574
query69	426	260	266	260
query70	1106	1119	1068	1068
query71	361	251	268	251
query72	6329	4034	4091	4034
query73	766	345	342	342
query74	10139	9041	8690	8690
query75	3293	2608	2606	2606
query76	1873	1012	992	992
query77	508	269	261	261
query78	10577	9581	9558	9558
query79	1715	602	593	593
query80	864	431	427	427
query81	515	238	238	238
query82	1298	84	82	82
query83	160	146	137	137
query84	288	73	74	73
query85	922	319	287	287
query86	331	285	283	283
query87	4404	4205	4230	4205
query88	3859	2395	2356	2356
query89	419	291	298	291
query90	1980	181	184	181
query91	179	149	151	149
query92	62	48	49	48
query93	1999	543	550	543
query94	742	302	292	292
query95	361	253	252	252
query96	618	271	273	271
query97	3275	3144	3130	3130
query98	219	203	196	196
query99	1769	1300	1271	1271
Total cold run time: 314053 ms
Total hot run time: 196716 ms

@doris-robot
Copy link

doris-robot commented Apr 25, 2025

ClickBench: Total hot run time: 32.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 18f43b0bdcfecfc3c5eb91a07d1ca20359bad150, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.04	0.03
query3	0.24	0.06	0.06
query4	1.63	0.11	0.11
query5	0.49	0.49	0.50
query6	1.15	0.73	0.74
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.49	0.50
query10	0.56	0.55	0.53
query11	0.16	0.11	0.11
query12	0.15	0.11	0.10
query13	0.60	0.59	0.59
query14	2.76	2.71	2.73
query15	0.90	0.82	0.82
query16	0.39	0.38	0.37
query17	1.08	1.07	1.05
query18	0.26	0.21	0.22
query19	1.92	1.81	2.00
query20	0.01	0.01	0.02
query21	15.36	0.60	0.60
query22	2.76	2.64	1.98
query23	17.03	0.88	0.89
query24	2.94	1.70	1.29
query25	0.24	0.12	0.08
query26	0.52	0.13	0.14
query27	0.05	0.04	0.03
query28	9.57	0.49	0.52
query29	12.58	3.22	3.23
query30	0.24	0.06	0.06
query31	2.85	0.39	0.38
query32	3.26	0.46	0.48
query33	2.93	3.03	3.03
query34	17.18	4.56	4.49
query35	4.50	4.50	4.56
query36	0.66	0.47	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.18 s
Total hot run time: 32.68 s

@dataroaring dataroaring merged commit a6a9f99 into branch-3.0 Apr 27, 2025
25 of 27 checks passed
@github-actions github-actions bot deleted the auto-pick-49905-branch-3.0 branch April 27, 2025 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hello-stephen @doris-robot @bobhan1 @dataroaring