[Opt](mow) Forbid time_series compaction policy on unique table #49905

bobhan1 · 2025-04-09T08:15:44Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-04-09T08:15:50Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

bobhan1 · 2025-04-09T08:35:22Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java

bobhan1 · 2025-04-09T08:58:32Z

run buildall

zhannngchen

LGTM

github-actions · 2025-04-09T11:35:25Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-04-09T11:35:28Z

PR approved by anyone and no changes requested.

bobhan1 · 2025-04-10T06:49:57Z

run buildall

doris-robot · 2025-04-10T07:19:52Z

TPC-H: Total hot run time: 34017 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a35583ee6359a5e15003e0405d16676d37085346, data reload: false

------ Round 1 ----------------------------------
q1	25779	5092	5023	5023
q2	2056	287	184	184
q3	10389	1287	692	692
q4	10237	1027	526	526
q5	7557	2335	2363	2335
q6	191	165	134	134
q7	926	734	600	600
q8	9304	1285	1094	1094
q9	6849	5047	5078	5047
q10	6830	2304	1886	1886
q11	490	296	265	265
q12	349	355	235	235
q13	17777	3694	3096	3096
q14	226	217	211	211
q15	547	481	472	472
q16	643	604	583	583
q17	614	857	379	379
q18	7421	7093	6972	6972
q19	1508	942	574	574
q20	337	350	230	230
q21	4309	3294	2504	2504
q22	1050	1049	975	975
Total cold run time: 115389 ms
Total hot run time: 34017 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5155	5197	5154	5154
q2	252	346	237	237
q3	2227	2705	2344	2344
q4	1440	1832	1437	1437
q5	4541	4476	4407	4407
q6	215	168	128	128
q7	1985	1922	1745	1745
q8	2584	2583	2790	2583
q9	7297	7160	7205	7160
q10	2977	3174	2768	2768
q11	572	515	504	504
q12	701	759	601	601
q13	3486	3855	3378	3378
q14	281	294	268	268
q15	522	473	478	473
q16	668	690	654	654
q17	1147	1516	1413	1413
q18	7786	7565	7343	7343
q19	842	828	867	828
q20	1968	1926	1827	1827
q21	5277	4857	4727	4727
q22	1119	1057	1038	1038
Total cold run time: 53042 ms
Total hot run time: 51017 ms

doris-robot · 2025-04-10T07:31:54Z

TPC-DS: Total hot run time: 193940 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a35583ee6359a5e15003e0405d16676d37085346, data reload: false

query1	1425	1064	1055	1055
query2	6175	1953	1956	1953
query3	11151	4626	4562	4562
query4	25391	23854	23341	23341
query5	5230	614	448	448
query6	293	196	193	193
query7	3985	492	283	283
query8	302	250	247	247
query9	8542	2577	2575	2575
query10	512	326	293	293
query11	15527	15010	14809	14809
query12	162	114	109	109
query13	1557	541	402	402
query14	8934	6208	6154	6154
query15	220	190	178	178
query16	7671	638	463	463
query17	1173	789	633	633
query18	2027	414	306	306
query19	195	188	165	165
query20	130	122	111	111
query21	207	125	109	109
query22	4562	4514	4297	4297
query23	34577	33817	33559	33559
query24	8626	2437	2430	2430
query25	535	469	400	400
query26	1229	279	157	157
query27	2867	509	335	335
query28	4942	2461	2426	2426
query29	713	559	465	465
query30	285	233	198	198
query31	892	881	869	869
query32	78	60	66	60
query33	567	371	324	324
query34	794	872	517	517
query35	815	869	749	749
query36	951	1002	910	910
query37	122	99	78	78
query38	4393	4225	4247	4225
query39	1548	1438	1488	1438
query40	221	123	120	120
query41	54	52	52	52
query42	115	105	111	105
query43	516	531	494	494
query44	1376	824	827	824
query45	176	177	168	168
query46	873	1025	647	647
query47	1842	1889	1836	1836
query48	387	431	305	305
query49	768	551	433	433
query50	685	709	411	411
query51	4317	4239	4232	4232
query52	107	107	99	99
query53	234	273	191	191
query54	606	595	511	511
query55	86	78	86	78
query56	314	307	304	304
query57	1185	1166	1071	1071
query58	269	268	283	268
query59	2811	2858	2774	2774
query60	326	325	334	325
query61	138	128	124	124
query62	764	738	687	687
query63	227	190	183	183
query64	3992	1243	839	839
query65	4428	4403	4383	4383
query66	966	479	307	307
query67	16113	15607	15309	15309
query68	9060	893	522	522
query69	510	303	275	275
query70	1213	1134	1126	1126
query71	473	316	299	299
query72	5565	4797	5033	4797
query73	754	667	351	351
query74	9241	9092	9068	9068
query75	4287	3214	2666	2666
query76	3831	1207	754	754
query77	814	379	285	285
query78	9977	10289	9301	9301
query79	4290	794	553	553
query80	686	527	441	441
query81	483	250	223	223
query82	369	123	95	95
query83	336	252	233	233
query84	294	102	88	88
query85	748	354	317	317
query86	345	321	276	276
query87	4441	4514	4406	4406
query88	2879	2228	2203	2203
query89	431	313	281	281
query90	2064	215	220	215
query91	145	142	110	110
query92	79	61	55	55
query93	2734	975	594	594
query94	693	420	321	321
query95	378	299	293	293
query96	496	555	270	270
query97	3110	3278	3130	3130
query98	225	227	196	196
query99	1421	1407	1290	1290
Total cold run time: 285573 ms
Total hot run time: 193940 ms

doris-robot · 2025-04-10T07:37:12Z

ClickBench: Total hot run time: 31.16 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a35583ee6359a5e15003e0405d16676d37085346, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.10
query3	0.28	0.19	0.19
query4	1.59	0.19	0.19
query5	0.58	0.56	0.59
query6	1.19	0.72	0.71
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.51	0.50
query10	0.59	0.59	0.57
query11	0.16	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.61
query14	2.81	2.69	2.80
query15	0.91	0.85	0.84
query16	0.38	0.38	0.38
query17	1.02	1.01	1.02
query18	0.21	0.20	0.19
query19	1.85	1.90	1.92
query20	0.01	0.01	0.02
query21	15.36	0.89	0.55
query22	0.76	1.17	0.70
query23	14.91	1.39	0.66
query24	7.65	0.77	0.73
query25	0.45	0.19	0.08
query26	0.71	0.16	0.13
query27	0.05	0.05	0.05
query28	9.26	0.90	0.46
query29	12.52	3.98	3.30
query30	0.26	0.08	0.06
query31	2.84	0.59	0.39
query32	3.24	0.56	0.48
query33	3.01	3.04	3.03
query34	15.85	5.14	4.48
query35	4.55	4.52	4.63
query36	0.66	0.50	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.02	0.02	0.02
query40	0.16	0.13	0.14
query41	0.09	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.66 s
Total hot run time: 31.16 s

bobhan1 · 2025-04-10T08:02:49Z

run buildall

bobhan1 · 2025-04-10T09:21:57Z

run buildall

doris-robot · 2025-04-11T10:14:49Z

TPC-DS: Total hot run time: 193437 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 04ccdf5c7932f06b30a8adf4c660e68b13fef148, data reload: false

query1	1399	1073	1014	1014
query2	6379	1966	1943	1943
query3	11139	4741	4548	4548
query4	25810	23882	23172	23172
query5	4809	622	450	450
query6	299	210	202	202
query7	3993	479	281	281
query8	290	232	234	232
query9	8518	2591	2609	2591
query10	489	323	279	279
query11	15350	15384	14763	14763
query12	169	115	114	114
query13	1560	548	402	402
query14	9172	6205	6195	6195
query15	201	187	162	162
query16	7325	640	490	490
query17	1148	789	593	593
query18	1991	419	311	311
query19	193	190	162	162
query20	127	127	121	121
query21	214	130	113	113
query22	4357	4633	4296	4296
query23	34584	33857	33784	33784
query24	8478	2487	2471	2471
query25	533	465	398	398
query26	1196	279	150	150
query27	3029	498	341	341
query28	4790	2456	2429	2429
query29	718	584	442	442
query30	283	228	201	201
query31	939	880	796	796
query32	82	65	63	63
query33	531	359	325	325
query34	808	889	524	524
query35	834	864	759	759
query36	1021	1028	920	920
query37	124	98	75	75
query38	4230	4239	4187	4187
query39	1498	1488	1429	1429
query40	217	120	108	108
query41	52	52	50	50
query42	123	107	108	107
query43	524	497	496	496
query44	1351	853	834	834
query45	190	185	170	170
query46	865	1044	658	658
query47	1829	1857	1840	1840
query48	396	418	305	305
query49	780	565	458	458
query50	666	705	409	409
query51	4279	4342	4255	4255
query52	115	108	101	101
query53	250	279	187	187
query54	604	585	530	530
query55	86	88	88	88
query56	327	309	300	300
query57	1181	1173	1145	1145
query58	273	272	269	269
query59	2764	2857	2704	2704
query60	330	334	309	309
query61	136	145	176	145
query62	762	749	667	667
query63	240	186	190	186
query64	4183	1058	693	693
query65	4422	4360	4340	4340
query66	1106	410	308	308
query67	15658	15398	15275	15275
query68	6454	886	512	512
query69	475	304	262	262
query70	1198	1103	1045	1045
query71	409	332	297	297
query72	5745	4643	4644	4643
query73	654	650	354	354
query74	9191	9021	8941	8941
query75	3221	3198	2744	2744
query76	3176	1186	762	762
query77	490	364	288	288
query78	9851	9990	9299	9299
query79	2349	818	568	568
query80	1457	526	464	464
query81	532	262	228	228
query82	428	127	90	90
query83	326	251	234	234
query84	243	107	82	82
query85	799	424	318	318
query86	449	306	272	272
query87	4417	4437	4399	4399
query88	3488	2269	2275	2269
query89	399	313	293	293
query90	1893	212	219	212
query91	144	141	112	112
query92	78	58	59	58
query93	2676	927	574	574
query94	726	406	308	308
query95	380	301	288	288
query96	506	562	281	281
query97	3107	3220	3114	3114
query98	243	202	205	202
query99	1331	1446	1259	1259
Total cold run time: 279154 ms
Total hot run time: 193437 ms

doris-robot · 2025-04-11T10:20:07Z

ClickBench: Total hot run time: 31.63 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 04ccdf5c7932f06b30a8adf4c660e68b13fef148, data reload: false

query1	0.03	0.04	0.03
query2	0.11	0.11	0.10
query3	0.25	0.19	0.19
query4	1.59	0.19	0.19
query5	0.59	0.57	0.60
query6	1.20	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.51	0.52
query10	0.57	0.57	0.58
query11	0.16	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.61	0.60
query14	2.65	2.70	2.71
query15	0.93	0.85	0.84
query16	0.38	0.38	0.39
query17	0.99	1.06	1.03
query18	0.20	0.19	0.20
query19	1.91	1.90	1.86
query20	0.01	0.01	0.02
query21	15.35	0.92	0.57
query22	0.74	1.28	0.65
query23	14.89	1.41	0.63
query24	7.22	1.33	1.22
query25	0.47	0.13	0.07
query26	0.65	0.16	0.14
query27	0.05	0.05	0.05
query28	9.52	0.89	0.44
query29	12.54	3.96	3.26
query30	0.25	0.08	0.06
query31	2.82	0.60	0.38
query32	3.23	0.56	0.47
query33	3.07	3.06	3.09
query34	15.86	5.12	4.56
query35	4.52	4.52	4.53
query36	0.67	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.03	0.04	0.03
Total cold run time: 105.29 s
Total hot run time: 31.63 s

### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [x] Yes.  - Does this need documentation? - [ ] No. - [x] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…he#49905) Issue Number: close #xxx Related PR: #xxx Problem Summary: None - Test  - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [x] Yes.  - Does this need documentation? - [ ] No. - [x] Yes.  - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…nique table (#49905) (#50132) pick #49905

…50194) some cases in p1/p2 are not modified in #49905

…pache#50194) some cases in p1/p2 are not modified in apache#49905

### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [x] Yes.  - Does this need documentation? - [ ] No. - [x] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

… table #49905 (#50128) Cherry-picked from #49905 Co-authored-by: bobhan1 <baohan@selectdb.com>

…he#49905) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [x] Yes.  - Does this need documentation? - [ ] No. - [x] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

…pache#50194) some cases in p1/p2 are not modified in apache#49905

… policy on unique table (apache#49905) (apache#3916) pick apache#49905

bobhan1 force-pushed the mow-forbid-time-series branch 2 times, most recently from c2387c4 to fb79316 Compare April 9, 2025 08:33

bobhan1 requested review from dataroaring, gavinchou and w41ter as code owners April 9, 2025 08:33

zhannngchen reviewed Apr 9, 2025

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java Outdated Show resolved Hide resolved

bobhan1 force-pushed the mow-forbid-time-series branch from 5c088fd to a35583e Compare April 9, 2025 08:57

bobhan1 changed the title ~~[Opt](mow) Forbid time_series compaction policy on mow table~~ [Opt](mow) Forbid time_series compaction policy on unique table Apr 9, 2025

zhannngchen previously approved these changes Apr 9, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 9, 2025

github-actions bot added the reviewed label Apr 9, 2025

bobhan1 dismissed zhannngchen’s stale review via 2d110b2 April 10, 2025 08:01

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 10, 2025

bobhan1 force-pushed the mow-forbid-time-series branch from 2d110b2 to 58e35be Compare April 10, 2025 09:21

bobhan1 added 4 commits April 11, 2025 10:00

update

a07f4d3

update

8f5517a

fix

7b5f645

fix

04ccdf5

bobhan1 force-pushed the mow-forbid-time-series branch from 58e35be to 04ccdf5 Compare April 11, 2025 02:00

zhannngchen merged commit 44f093d into apache:master Apr 17, 2025
25 of 26 checks passed

zhannngchen added dev/2.1.x dev/3.0.x labels Apr 17, 2025

github-actions bot mentioned this pull request Apr 17, 2025

branch-3.0: [Opt](mow) Forbid time_series compaction policy on unique table #49905 #50128

Merged

github-actions bot added the dev/2.1.x-conflict label Apr 17, 2025

bobhan1 mentioned this pull request Apr 17, 2025

branch-2.1-pick: [Opt](mow) Forbid time_series compaction policy on unique table (#49905) #50132

Merged

yiguolei pushed a commit that referenced this pull request Apr 18, 2025

branch-2.1-pick: [Opt](mow) Forbid time_series compaction policy on u…

4146cbb

…nique table (#49905) (#50132) pick #49905

yiguolei added dev/2.1.10-merged and removed dev/2.1.x dev/2.1.x-conflict labels Apr 18, 2025

bobhan1 mentioned this pull request Apr 19, 2025

[Fix](case) Fix case that use time series compaction on unique table #50194

Merged

16 tasks

dataroaring pushed a commit that referenced this pull request Apr 21, 2025

[Fix](case) Fix case that use time series compaction on unique table (#…

2cca481

…50194) some cases in p1/p2 are not modified in #49905

github-actions bot pushed a commit that referenced this pull request Apr 21, 2025

[Fix](case) Fix case that use time series compaction on unique table (#…

f398d90

…50194) some cases in p1/p2 are not modified in #49905

bobhan1 added a commit to bobhan1/doris that referenced this pull request Apr 21, 2025

[Fix](case) Fix case that use time series compaction on unique table (a…

b949be4

…pache#50194) some cases in p1/p2 are not modified in apache#49905

dataroaring added the p0_b label Apr 24, 2025

dataroaring pushed a commit that referenced this pull request Apr 27, 2025

branch-3.0: [Opt](mow) Forbid time_series compaction policy on unique…

a6a9f99

… table #49905 (#50128) Cherry-picked from #49905 Co-authored-by: bobhan1 <baohan@selectdb.com>

dataroaring added dev/3.0.6-merged and removed dev/3.0.x labels Apr 27, 2025

yiguolei mentioned this pull request May 13, 2025

2.1.10 Release Note #50853

Closed

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025

[Fix](case) Fix case that use time series compaction on unique table (a…

cb015f9

…pache#50194) some cases in p1/p2 are not modified in apache#49905

dataroaring added the kind/behavior-changed label Jun 5, 2025

gavinchou mentioned this pull request Jun 11, 2025

Release Note 3.0.6 #51585

Closed

suxiaogang223 pushed a commit to suxiaogang223/doris that referenced this pull request Jul 15, 2025

(selectdb-cloud-4.0.4): pick [Opt](mow) Forbid time_series compaction…

9d065fb

… policy on unique table (apache#49905) (apache#3916) pick apache#49905

Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Jul 21, 2025

(selectdb-cloud-4.0.4): pick [Opt](mow) Forbid time_series compaction…

e807db9

… policy on unique table (apache#49905) (apache#3916) pick apache#49905

[Opt](mow) Forbid time_series compaction policy on unique table #49905

[Opt](mow) Forbid time_series compaction policy on unique table #49905

Uh oh!

Conversation

bobhan1 commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Apr 9, 2025

Uh oh!

bobhan1 commented Apr 9, 2025

Uh oh!

Uh oh!

bobhan1 commented Apr 9, 2025

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

bobhan1 commented Apr 10, 2025

Uh oh!

doris-robot commented Apr 10, 2025

Uh oh!

doris-robot commented Apr 10, 2025

Uh oh!

doris-robot commented Apr 10, 2025

Uh oh!

bobhan1 commented Apr 10, 2025

Uh oh!

bobhan1 commented Apr 10, 2025

Uh oh!

doris-robot commented Apr 11, 2025

Uh oh!

doris-robot commented Apr 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bobhan1 commented Apr 9, 2025 •

edited

Loading