Skip to content

[enhancement](time_series) increase the version limit for the time series table #51371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 2 commits into from
Jun 1, 2025
Merged

[enhancement](time_series) increase the version limit for the time series table #51371

dataroaring merged 2 commits into from
Jun 1, 2025

Conversation

@luwei16
Copy link
Contributor

@luwei16 luwei16 commented May 30, 2025

No description provided.

@Thearas
Copy link
Contributor

Thearas commented May 30, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@luwei16
Copy link
Contributor Author

luwei16 commented May 30, 2025

run buildall

@dataroaring dataroaring added dev/3.0.x dev/2.1.x usercase Important user case type label labels May 30, 2025
dataroaring
dataroaring approved these changes May 30, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented May 30, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels May 30, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 30, 2025

PR approved by anyone and no changes requested.

csun5285
csun5285 reviewed May 30, 2025
View reviewed changes
be/src/olap/base_tablet.cpp

int32_t BaseTablet::max_version_config() {
int32_t max_version = tablet_meta()->compaction_policy() == CUMULATIVE_TIME_SERIES_POLICY
? config::time_series_max_tablet_version_num
Copy link
Contributor

@csun5285 csun5285 May 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tablet_meta()->time_series_compaction_file_count_threshold() * 10 might be a better config

@doris-robot
Copy link

doris-robot commented May 30, 2025

TPC-H: Total hot run time: 33919 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 53b8485fe6520b3ed386d3cd5daf2489497c47be, data reload: false

------ Round 1 ----------------------------------
q1	26298	5124	5344	5124
q2	1982	287	192	192
q3	10374	1268	692	692
q4	10229	1026	537	537
q5	7642	2356	2353	2353
q6	190	160	136	136
q7	911	747	607	607
q8	9312	1300	1001	1001
q9	6822	5126	5114	5114
q10	6854	2318	1918	1918
q11	496	278	267	267
q12	341	344	216	216
q13	17807	3713	3090	3090
q14	236	249	215	215
q15	575	494	483	483
q16	425	442	378	378
q17	584	859	354	354
q18	7725	7277	7109	7109
q19	1838	973	564	564
q20	328	342	216	216
q21	3591	3267	2380	2380
q22	1095	1030	973	973
Total cold run time: 115655 ms
Total hot run time: 33919 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5279	5139	5021	5021
q2	252	319	216	216
q3	2153	2628	2304	2304
q4	1390	1811	1398	1398
q5	4547	4410	4380	4380
q6	224	173	131	131
q7	1982	1914	1769	1769
q8	2590	2532	2523	2523
q9	7292	7077	7165	7077
q10	3028	3215	2738	2738
q11	600	497	513	497
q12	659	788	591	591
q13	3538	3864	3272	3272
q14	278	300	267	267
q15	521	480	466	466
q16	443	482	457	457
q17	1158	1543	1397	1397
q18	7871	7537	7434	7434
q19	835	863	948	863
q20	2004	2032	1837	1837
q21	4787	4339	4308	4308
q22	1103	1043	980	980
Total cold run time: 52534 ms
Total hot run time: 49926 ms

@doris-robot
Copy link

doris-robot commented May 30, 2025

TPC-DS: Total hot run time: 185737 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 53b8485fe6520b3ed386d3cd5daf2489497c47be, data reload: false

query1	1024	491	476	476
query2	6569	1808	1777	1777
query3	6743	230	214	214
query4	26401	23744	23527	23527
query5	4533	627	476	476
query6	320	238	205	205
query7	4637	500	293	293
query8	271	232	228	228
query9	8645	2687	2711	2687
query10	487	330	304	304
query11	15768	15235	14892	14892
query12	169	109	111	109
query13	1660	527	421	421
query14	9946	6216	6245	6216
query15	215	195	175	175
query16	7648	642	473	473
query17	1201	734	618	618
query18	2007	385	296	296
query19	191	182	152	152
query20	124	118	115	115
query21	212	125	110	110
query22	4096	4304	3982	3982
query23	34090	33146	33255	33146
query24	8430	2388	2385	2385
query25	509	447	376	376
query26	1227	278	150	150
query27	2698	527	350	350
query28	4386	2146	2132	2132
query29	734	558	434	434
query30	286	220	187	187
query31	926	857	778	778
query32	72	65	62	62
query33	555	346	335	335
query34	787	864	520	520
query35	791	801	730	730
query36	960	966	896	896
query37	110	94	82	82
query38	4126	4241	4043	4043
query39	1481	1421	1395	1395
query40	212	119	107	107
query41	61	57	57	57
query42	135	106	111	106
query43	502	475	467	467
query44	1304	837	830	830
query45	182	185	171	171
query46	834	1019	625	625
query47	1763	1793	1727	1727
query48	381	435	313	313
query49	750	482	431	431
query50	618	668	405	405
query51	4077	4147	4082	4082
query52	105	109	98	98
query53	222	252	191	191
query54	576	566	497	497
query55	97	82	81	81
query56	305	324	274	274
query57	1125	1162	1085	1085
query58	263	248	260	248
query59	2628	2613	2493	2493
query60	340	326	309	309
query61	130	125	130	125
query62	817	738	643	643
query63	213	191	181	181
query64	4341	1020	670	670
query65	4211	4113	4101	4101
query66	1073	421	316	316
query67	15960	15780	15287	15287
query68	8604	891	536	536
query69	473	316	271	271
query70	1191	1127	1058	1058
query71	470	316	306	306
query72	5624	4759	4573	4573
query73	720	606	349	349
query74	9225	9100	8939	8939
query75	3981	3247	2654	2654
query76	3598	1193	764	764
query77	789	387	294	294
query78	10121	10071	9258	9258
query79	2385	820	579	579
query80	631	524	443	443
query81	469	260	217	217
query82	434	122	97	97
query83	291	313	230	230
query84	287	113	85	85
query85	794	353	321	321
query86	371	309	286	286
query87	4358	4487	4254	4254
query88	3387	2265	2286	2265
query89	395	324	280	280
query90	1943	210	204	204
query91	137	141	112	112
query92	74	64	56	56
query93	1739	961	580	580
query94	662	406	309	309
query95	380	289	286	286
query96	489	575	300	300
query97	2741	2779	2690	2690
query98	304	217	211	211
query99	1424	1412	1263	1263
Total cold run time: 276751 ms
Total hot run time: 185737 ms

@doris-robot
Copy link

doris-robot commented May 30, 2025

ClickBench: Total hot run time: 29.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 53b8485fe6520b3ed386d3cd5daf2489497c47be, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.12
query3	0.25	0.19	0.20
query4	1.59	0.19	0.12
query5	0.45	0.43	0.40
query6	1.18	0.65	0.66
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.57	0.53	0.50
query10	0.57	0.57	0.58
query11	0.16	0.11	0.11
query12	0.16	0.11	0.12
query13	0.62	0.59	0.60
query14	0.80	0.81	0.81
query15	0.88	0.88	0.85
query16	0.38	0.39	0.38
query17	1.03	1.04	1.07
query18	0.24	0.21	0.21
query19	1.99	1.79	1.87
query20	0.02	0.01	0.01
query21	15.40	0.88	0.55
query22	0.76	1.19	0.63
query23	14.94	1.40	0.63
query24	6.88	1.12	0.88
query25	0.54	0.19	0.08
query26	0.63	0.15	0.15
query27	0.05	0.04	0.05
query28	9.84	0.96	0.44
query29	12.56	3.97	3.33
query30	0.25	0.10	0.06
query31	2.82	0.61	0.39
query32	3.23	0.55	0.47
query33	3.00	3.07	3.10
query34	15.70	5.13	4.60
query35	4.56	4.58	4.56
query36	0.68	0.49	0.48
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.14
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.05	0.03	0.03
Total cold run time: 103.48 s
Total hot run time: 29.2 s

@dataroaring dataroaring merged commit 96fc8df into apache:master Jun 1, 2025
28 of 31 checks passed
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@luwei16 @koarz
[enhancement](time_series) increase the version limit for the time se…
6cc416b 
…ries table (apache#51371)
luwei16 added a commit to luwei16/incubator-doris that referenced this pull request Jun 12, 2025
@luwei16
[enhancement](time_series) increase the version limit for the time se…
4acbeb1 
…ries table (apache#51371)
@luwei16 luwei16 mentioned this pull request Jun 12, 2025
Merged
dataroaring pushed a commit that referenced this pull request Jun 17, 2025
@luwei16
[enhancement](time_series) increase the version limit for the time se…
6acf137 
…ries table (#51371) (#51656)

pick #51371
luwei16 added a commit to luwei16/incubator-doris that referenced this pull request Jun 20, 2025
@luwei16
[enhancement](time_series) increase the version limit for the time se…
ae4b6ad 
…ries table (apache#51371)
@luwei16 luwei16 mentioned this pull request Jun 20, 2025
Merged
luwei16 added a commit to luwei16/incubator-doris that referenced this pull request Jun 20, 2025
@luwei16
[enhancement](time_series) increase the version limit for the time se…
8cb90fd 
…ries table (apache#51371)
@luwei16 luwei16 mentioned this pull request Jun 20, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jun 23, 2025
@luwei16
branch-3.1: [enhancement](time_series) increase the version limit for…
b2a9fce 
… the time series table #51371 (#52073)

pick master #51371
yiguolei pushed a commit that referenced this pull request Jun 24, 2025
@luwei16
[enhancement](time_series) increase the version limit for the time se…
4909636 
…ries table (#51371) (#52077)

pick master #51371
@yiguolei yiguolei mentioned this pull request Aug 11, 2025
Open
BiteTheDDDDt pushed a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Sep 1, 2025
@luwei16
[enhancement](time_series) increase the version limit for the time se…
a71bc88 
…ries table (apache#51371) (apache#4156) (apache#4269)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

+2 more reviewers

@csun5285 csun5285 csun5285 left review comments

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.7-merged dev/3.1.0-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@luwei16 @Thearas @doris-robot @csun5285 @Yukang-Lian @dataroaring @yiguolei @morrySnow