Skip to content

[feature](cloud) support event driven or periodic warm up #52370

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 25 commits into from
Aug 5, 2025
Merged

[feature](cloud) support event driven or periodic warm up #52370

dataroaring merged 25 commits into from
Aug 5, 2025

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Jun 26, 2025

What problem does this PR solve?

Extend WARM UP statement, support event driven or periodic warm up jobs.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

kaijchen added 10 commits June 26, 2025 16:49
@Thearas
Copy link
Contributor

Thearas commented Jun 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaijchen
Copy link
Member Author

kaijchen commented Jun 26, 2025

run buildall

@kaijchen
Copy link
Member Author

kaijchen commented Jun 26, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 26, 2025

BE UT Coverage Report

Increment line coverage 0.31% (1/326) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.03% (15367/26945)
Line Coverage 46.08% (139412/302544)
Region Coverage 45.41% (70637/155555)
Branch Coverage 40.17% (37310/92870)

@kaijchen
Copy link
Member Author

kaijchen commented Jun 27, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 27, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.52% (1171/1402)
Line Coverage 67.47% (20645/30601)
Region Coverage 67.03% (10253/15295)
Branch Coverage 56.32% (5366/9528)

@doris-robot
Copy link

doris-robot commented Jun 27, 2025

BE UT Coverage Report

Increment line coverage 0.30% (1/334) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.02% (15366/26950)
Line Coverage 46.07% (139429/302639)
Region Coverage 45.40% (70643/155600)
Branch Coverage 40.17% (37320/92902)

This was referenced Jun 30, 2025
Merged
Merged
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 30, 2025

BE Regression && UT Coverage Report

Increment line coverage 22.88% (135/590) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.20% (22185/27320)
Line Coverage 73.82% (229694/311148)
Region Coverage 61.38% (192080/312940)
Branch Coverage 65.39% (82799/126626)

@kaijchen
Copy link
Member Author

kaijchen commented Jul 30, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 30, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.69% (1341/1662)
Line Coverage 65.63% (23059/35136)
Region Coverage 66.79% (11417/17094)
Branch Coverage 56.43% (5986/10608)

@doris-robot
Copy link

doris-robot commented Jul 30, 2025

TPC-H: Total hot run time: 33712 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 63c5ca756391d7d248d75a981000e187ab343bba, data reload: false

------ Round 1 ----------------------------------
q1	16947	5881	5420	5420
q2	1917	271	207	207
q3	10066	1254	690	690
q4	10113	961	508	508
q5	7699	2361	2270	2270
q6	178	165	132	132
q7	892	733	626	626
q8	9307	1286	1036	1036
q9	6793	5078	5069	5069
q10	6884	2364	1945	1945
q11	463	293	261	261
q12	350	362	221	221
q13	17766	3476	2950	2950
q14	234	245	215	215
q15	532	484	465	465
q16	432	441	371	371
q17	525	819	346	346
q18	7485	6985	7036	6985
q19	1094	954	526	526
q20	316	328	214	214
q21	2789	2444	2251	2251
q22	1054	1035	1004	1004
Total cold run time: 103836 ms
Total hot run time: 33712 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5703	5685	5678	5678
q2	233	332	222	222
q3	2068	2530	2187	2187
q4	1299	1752	1332	1332
q5	4125	4263	4302	4263
q6	213	182	134	134
q7	2112	1869	1776	1776
q8	2508	2420	2373	2373
q9	7330	7351	7462	7351
q10	3113	3315	2969	2969
q11	528	505	514	505
q12	690	783	647	647
q13	3360	3643	3277	3277
q14	319	319	284	284
q15	522	530	440	440
q16	417	474	443	443
q17	1142	1434	1348	1348
q18	12836	7901	7700	7700
q19	5969	931	1142	931
q20	1986	2051	1764	1764
q21	14911	4219	4220	4219
q22	1100	1056	988	988
Total cold run time: 72484 ms
Total hot run time: 50831 ms

@doris-robot
Copy link

doris-robot commented Jul 30, 2025

TPC-DS: Total hot run time: 187191 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 63c5ca756391d7d248d75a981000e187ab343bba, data reload: false

query1	1005	386	418	386
query2	6521	1692	1651	1651
query3	6741	223	218	218
query4	26663	23917	23091	23091
query5	4326	621	510	510
query6	328	241	223	223
query7	4628	519	296	296
query8	283	233	222	222
query9	8616	3148	3152	3148
query10	460	341	299	299
query11	16145	14961	14733	14733
query12	182	136	131	131
query13	1669	558	423	423
query14	9553	7673	7624	7624
query15	214	199	172	172
query16	7125	673	469	469
query17	1023	747	644	644
query18	1990	430	325	325
query19	216	203	186	186
query20	155	150	142	142
query21	219	126	127	126
query22	4303	4123	3931	3931
query23	34100	34275	33910	33910
query24	5069	2401	2452	2401
query25	496	511	446	446
query26	729	290	159	159
query27	2262	506	352	352
query28	2983	2270	2264	2264
query29	629	603	489	489
query30	284	225	192	192
query31	865	789	739	739
query32	89	77	79	77
query33	488	407	389	389
query34	789	877	521	521
query35	792	821	761	761
query36	1045	1046	943	943
query37	133	114	89	89
query38	4087	4018	4009	4009
query39	1419	1384	1370	1370
query40	244	138	137	137
query41	63	58	57	57
query42	138	120	125	120
query43	506	495	462	462
query44	1411	883	866	866
query45	208	186	184	184
query46	941	1046	671	671
query47	1807	1854	1746	1746
query48	393	438	315	315
query49	713	517	428	428
query50	662	684	418	418
query51	4143	4231	4164	4164
query52	130	130	115	115
query53	262	283	215	215
query54	654	641	552	552
query55	89	86	90	86
query56	350	342	351	342
query57	1216	1212	1159	1159
query58	345	330	331	330
query59	2574	2654	2570	2570
query60	404	412	381	381
query61	157	127	125	125
query62	794	744	659	659
query63	252	213	213	213
query64	2357	1107	777	777
query65	4163	4122	4089	4089
query66	752	442	338	338
query67	15872	16203	15641	15641
query68	14782	587	593	587
query69	1009	317	290	290
query70	1303	1083	1091	1083
query71	742	333	321	321
query72	9334	2266	2345	2266
query73	3125	651	356	356
query74	9091	8942	8855	8855
query75	7675	3136	2673	2673
query76	8667	1197	789	789
query77	1150	405	425	405
query78	9408	10253	9179	9179
query79	14805	603	566	566
query80	788	575	475	475
query81	555	265	240	240
query82	558	148	117	117
query83	296	287	270	270
query84	297	93	90	90
query85	927	424	327	327
query86	368	300	296	296
query87	4269	4247	4213	4213
query88	5782	2239	2235	2235
query89	479	359	308	308
query90	2548	223	223	223
query91	142	139	113	113
query92	87	73	67	67
query93	5821	963	665	665
query94	1004	404	281	281
query95	418	320	319	319
query96	496	588	282	282
query97	2678	2728	2593	2593
query98	250	237	226	226
query99	1560	1415	1260	1260
Total cold run time: 310638 ms
Total hot run time: 187191 ms

@doris-robot
Copy link

doris-robot commented Jul 30, 2025

BE UT Coverage Report

Increment line coverage 0.30% (1/335) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.78% (16076/27821)
Line Coverage 46.48% (144604/311096)
Region Coverage 35.71% (108847/304812)
Branch Coverage 38.39% (48039/125141)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 30, 2025

BE Regression && UT Coverage Report

Increment line coverage 22.75% (134/589) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.21% (22182/27314)
Line Coverage 73.78% (229502/311072)
Region Coverage 61.33% (191937/312945)
Branch Coverage 65.35% (82716/126580)

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 30, 2025

BE Regression && UT Coverage Report

Increment line coverage 22.75% (134/589) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.21% (22182/27314)
Line Coverage 73.78% (229502/311072)
Region Coverage 61.33% (191937/312945)
Branch Coverage 65.35% (82716/126580)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 30, 2025

BE Regression && UT Coverage Report

Increment line coverage 22.75% (134/589) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.20% (22179/27315)
Line Coverage 73.75% (229449/311113)
Region Coverage 61.34% (191984/312987)
Branch Coverage 65.32% (82700/126607)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 30, 2025

BE Regression && UT Coverage Report

Increment line coverage 22.75% (134/589) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.21% (22183/27315)
Line Coverage 73.77% (229503/311113)
Region Coverage 61.33% (191957/312987)
Branch Coverage 65.33% (82715/126607)

@zhannngchen
Copy link
Contributor

zhannngchen commented Aug 4, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.16% (1361/1677)
Line Coverage 65.78% (23501/35725)
Region Coverage 66.99% (11674/17427)
Branch Coverage 56.63% (6117/10802)

@doris-robot
Copy link

doris-robot commented Aug 4, 2025

TPC-H: Total hot run time: 33581 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 63c5ca756391d7d248d75a981000e187ab343bba, data reload: false

------ Round 1 ----------------------------------
q1	17714	5131	5199	5131
q2	1945	303	180	180
q3	10682	1347	700	700
q4	10349	1058	518	518
q5	10163	2359	2282	2282
q6	218	166	131	131
q7	888	769	633	633
q8	9319	1250	1059	1059
q9	7064	5034	5062	5034
q10	6958	2374	2032	2032
q11	463	274	266	266
q12	345	360	223	223
q13	17789	3472	2950	2950
q14	225	236	222	222
q15	546	469	459	459
q16	416	442	375	375
q17	548	818	382	382
q18	7220	6988	6982	6982
q19	1477	978	530	530
q20	590	310	217	217
q21	3402	3015	2287	2287
q22	1062	1061	988	988
Total cold run time: 109383 ms
Total hot run time: 33581 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5359	5267	5319	5267
q2	356	307	212	212
q3	2084	2591	2235	2235
q4	1322	1713	1324	1324
q5	4323	4442	4526	4442
q6	221	172	136	136
q7	1949	1936	1708	1708
q8	2616	2604	2596	2596
q9	7386	7456	7598	7456
q10	3163	3364	2922	2922
q11	540	512	513	512
q12	955	778	865	778
q13	3319	3571	3304	3304
q14	299	316	286	286
q15	488	442	459	442
q16	483	481	464	464
q17	1390	1406	1358	1358
q18	7740	7650	7680	7650
q19	12027	913	1140	913
q20	3373	2016	1757	1757
q21	12751	4338	4275	4275
q22	1122	1029	982	982
Total cold run time: 73266 ms
Total hot run time: 51019 ms

@doris-robot
Copy link

doris-robot commented Aug 4, 2025

TPC-DS: Total hot run time: 185168 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 63c5ca756391d7d248d75a981000e187ab343bba, data reload: false

query1	994	378	426	378
query2	6520	1706	1722	1706
query3	6752	233	221	221
query4	26956	24085	22759	22759
query5	4330	655	515	515
query6	320	250	213	213
query7	4639	533	300	300
query8	269	231	218	218
query9	8579	2958	2925	2925
query10	469	357	289	289
query11	15763	15283	14711	14711
query12	187	142	134	134
query13	1664	572	437	437
query14	8582	5873	5849	5849
query15	213	190	168	168
query16	7130	678	513	513
query17	1032	745	619	619
query18	1993	435	313	313
query19	226	216	206	206
query20	151	146	139	139
query21	221	125	110	110
query22	4025	3991	3822	3822
query23	34740	34441	34380	34380
query24	8251	2420	2472	2420
query25	651	533	442	442
query26	1247	290	172	172
query27	2717	511	348	348
query28	4393	2313	2308	2308
query29	803	608	491	491
query30	299	234	204	204
query31	909	791	748	748
query32	94	76	77	76
query33	565	413	382	382
query34	816	858	519	519
query35	828	830	748	748
query36	1020	1055	948	948
query37	143	116	98	98
query38	4002	3973	3941	3941
query39	1439	1382	1404	1382
query40	250	142	148	142
query41	64	57	54	54
query42	149	128	131	128
query43	535	522	478	478
query44	1471	881	878	878
query45	208	185	181	181
query46	939	1124	693	693
query47	1794	1819	1772	1772
query48	409	444	328	328
query49	733	507	432	432
query50	680	711	442	442
query51	4196	4237	4208	4208
query52	141	140	123	123
query53	256	305	211	211
query54	658	660	565	565
query55	91	88	86	86
query56	358	361	350	350
query57	1201	1217	1140	1140
query58	333	328	336	328
query59	2585	2655	2515	2515
query60	424	398	394	394
query61	122	122	120	120
query62	843	720	668	668
query63	264	230	214	214
query64	4348	1109	817	817
query65	4290	4179	4133	4133
query66	1155	443	335	335
query67	15833	16109	15573	15573
query68	15906	607	614	607
query69	1010	337	274	274
query70	1324	1108	1121	1108
query71	724	341	321	321
query72	9224	2288	2324	2288
query73	3480	651	355	355
query74	9100	8981	8681	8681
query75	7918	3102	2628	2628
query76	8547	1207	782	782
query77	1171	411	361	361
query78	9522	10680	8979	8979
query79	12968	618	598	598
query80	786	556	512	512
query81	567	264	227	227
query82	388	157	128	128
query83	301	299	269	269
query84	332	100	82	82
query85	855	368	402	368
query86	362	326	329	326
query87	4164	4256	4090	4090
query88	3675	2284	2266	2266
query89	471	372	327	327
query90	2749	236	239	236
query91	143	138	111	111
query92	90	76	69	69
query93	3354	998	682	682
query94	938	404	287	287
query95	407	328	330	328
query96	521	622	290	290
query97	2702	2732	2618	2618
query98	252	235	231	231
query99	1471	1435	1279	1279
Total cold run time: 313684 ms
Total hot run time: 185168 ms

@doris-robot
Copy link

doris-robot commented Aug 4, 2025

ClickBench: Total hot run time: 35.19 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 63c5ca756391d7d248d75a981000e187ab343bba, data reload: false

query1	0.04	0.04	0.03
query2	0.11	0.06	0.05
query3	0.31	0.06	0.06
query4	1.60	0.09	0.09
query5	0.42	0.40	0.42
query6	1.16	0.67	0.65
query7	0.02	0.01	0.02
query8	0.07	0.05	0.05
query9	0.62	0.47	0.49
query10	0.54	0.53	0.53
query11	0.26	0.13	0.13
query12	0.26	0.14	0.14
query13	0.69	0.68	0.68
query14	0.96	1.27	1.00
query15	1.06	0.93	0.95
query16	0.40	0.39	0.39
query17	1.11	1.12	1.07
query18	0.24	0.23	0.24
query19	1.97	1.94	2.03
query20	0.02	0.01	0.01
query21	15.40	1.04	0.73
query22	1.00	1.31	1.03
query23	14.73	1.58	0.95
query24	5.58	0.59	0.34
query25	0.19	0.12	0.11
query26	0.57	0.24	0.20
query27	0.12	0.11	0.11
query28	11.06	1.18	0.66
query29	12.59	4.05	3.52
query30	3.08	3.04	3.02
query31	2.83	0.64	0.47
query32	3.24	0.68	0.57
query33	3.06	3.34	3.26
query34	16.75	5.47	4.94
query35	4.90	5.16	5.04
query36	0.68	0.54	0.52
query37	0.25	0.23	0.22
query38	0.23	0.22	0.23
query39	0.07	0.07	0.06
query40	0.20	0.18	0.16
query41	0.12	0.07	0.08
query42	0.09	0.08	0.07
query43	0.08	0.07	0.06
Total cold run time: 108.68 s
Total hot run time: 35.19 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

BE UT Coverage Report

Increment line coverage 0.30% (1/335) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.97% (16154/27864)
Line Coverage 46.73% (145438/311231)
Region Coverage 35.83% (109119/304552)
Branch Coverage 38.58% (48247/125061)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

BE Regression && UT Coverage Report

Increment line coverage 22.75% (134/589) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.76% (22068/27327)
Line Coverage 73.17% (226915/310136)
Region Coverage 60.93% (189814/311534)
Branch Coverage 64.75% (81449/125791)

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

BE Regression && UT Coverage Report

Increment line coverage 22.75% (134/589) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.76% (22068/27327)
Line Coverage 73.17% (226915/310136)
Region Coverage 60.93% (189814/311534)
Branch Coverage 64.75% (81449/125791)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

BE Regression && UT Coverage Report

Increment line coverage 22.75% (134/589) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.78% (22096/27352)
Line Coverage 73.14% (227639/311227)
Region Coverage 60.83% (190229/312716)
Branch Coverage 64.61% (81753/126527)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

BE Regression && UT Coverage Report

Increment line coverage 22.75% (134/589) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.79% (22099/27352)
Line Coverage 73.16% (227706/311227)
Region Coverage 60.86% (190330/312716)
Branch Coverage 64.66% (81807/126527)

dataroaring
dataroaring approved these changes Aug 5, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 5, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Aug 5, 2025

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 70720b1 into apache:master Aug 5, 2025
26 of 28 checks passed
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
@kaijchen kaijchen mentioned this pull request Aug 22, 2025
Merged
16 tasks
dataroaring pushed a commit that referenced this pull request Aug 22, 2025
@kaijchen
[fix](warmup) fix show warm up tables (#55192)
4797096 
### What problem does this PR solve?

Related PR: #51594  #52370 

Problem Summary:

Fix show warm up jobs after merging periodic and event driven warmup
feature.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

@w41ter w41ter Awaiting requested review from w41ter w41ter is a code owner

@liaoxin01 liaoxin01 Awaiting requested review from liaoxin01

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. cloud dev/3.0.7-merged dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

9 participants

@kaijchen @Thearas @hello-stephen @doris-robot @zhannngchen @gavinchou @liaoxin01 @dataroaring @morrySnow