Skip to content

[feat](warmup) display tables in SHOW WARM UP JOB results #51594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 3 commits into from
Jun 17, 2025
Merged

[feat](warmup) display tables in SHOW WARM UP JOB results #51594

dataroaring merged 3 commits into from
Jun 17, 2025

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Jun 9, 2025 •
edited
Loading

What problem does this PR solve?

Enhance the SHOW WARM UP JOB command to display tables for "warm up by tables" jobs.
Table names are shown in the format db.table.partition, separated by commas.

For example:

mysql> show warm up job;                                                                                             
+---------------+--------------+----------+-------+-------------------------+-------------+----------+-------------------------+--------+------------------+
| JobId         | ComputeGroup | Status   | Type  | CreateTime              | FinishBatch | AllBatch | FinishTime              | ErrMsg | Tables           |
+---------------+--------------+----------+-------+-------------------------+-------------+----------+-------------------------+--------+------------------+
| 1749488013342 | cluster1     | FINISHED | TABLE | 2025-06-10 01:01:43.670 | 1           | 1        | 2025-06-10 01:01:46.344 |        | test.y, test.t.t |
| 1749488013329 | cluster1     | FINISHED | TABLE | 2025-06-10 00:59:41.135 | 1           | 1        | 2025-06-10 00:59:43.336 |        | test.y, test.t   |
| 1749488013306 | cluster1     | FINISHED | TABLE | 2025-06-10 00:58:43.781 | 1           | 1        | 2025-06-10 00:58:46.332 |        | test.y           |
+---------------+--------------+----------+-------+-------------------------+-------------+----------+-------------------------+--------+------------------+

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jun 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaijchen
Copy link
Member Author

kaijchen commented Jun 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 9, 2025

TPC-H: Total hot run time: 33534 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 28da1221d08268bb825a13a11d31303bf4a3164e, data reload: false

------ Round 1 ----------------------------------
q1	26080	5046	4993	4993
q2	1941	273	181	181
q3	10305	1243	677	677
q4	10218	1001	521	521
q5	7541	2219	2381	2219
q6	181	163	131	131
q7	880	750	629	629
q8	9342	1284	1099	1099
q9	7530	5053	5122	5053
q10	6883	2284	1890	1890
q11	478	280	269	269
q12	349	345	214	214
q13	17771	3659	3083	3083
q14	237	230	211	211
q15	567	496	486	486
q16	419	424	375	375
q17	591	847	368	368
q18	7423	7030	7123	7030
q19	1550	956	558	558
q20	321	333	211	211
q21	3603	3049	2356	2356
q22	1030	980	982	980
Total cold run time: 115240 ms
Total hot run time: 33534 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5092	5027	4998	4998
q2	232	317	218	218
q3	2139	2627	2298	2298
q4	1363	1767	1308	1308
q5	4330	4383	4320	4320
q6	227	174	128	128
q7	1987	1897	1762	1762
q8	2590	2691	2568	2568
q9	7163	7176	7003	7003
q10	2968	3224	2752	2752
q11	566	520	494	494
q12	665	762	604	604
q13	3523	3874	3277	3277
q14	277	295	262	262
q15	541	496	494	494
q16	445	502	444	444
q17	1127	1503	1362	1362
q18	7697	7470	7416	7416
q19	794	819	895	819
q20	1981	2051	1920	1920
q21	4790	4426	4438	4426
q22	1048	1038	994	994
Total cold run time: 51545 ms
Total hot run time: 49867 ms

@doris-robot
Copy link

doris-robot commented Jun 9, 2025

TPC-DS: Total hot run time: 193247 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 28da1221d08268bb825a13a11d31303bf4a3164e, data reload: false

query1	1457	1053	1047	1047
query2	6205	1842	1898	1842
query3	11002	4490	4452	4452
query4	54582	24719	23392	23392
query5	5060	551	481	481
query6	334	217	204	204
query7	4894	511	308	308
query8	278	223	209	209
query9	5344	2683	2677	2677
query10	433	342	280	280
query11	15216	14994	14830	14830
query12	178	111	109	109
query13	1040	527	402	402
query14	10197	6446	6523	6446
query15	205	204	189	189
query16	7144	682	489	489
query17	1096	751	608	608
query18	1542	420	325	325
query19	207	192	173	173
query20	136	134	123	123
query21	220	133	120	120
query22	4313	4435	4501	4435
query23	34376	33566	33507	33507
query24	6736	2464	2478	2464
query25	485	501	426	426
query26	727	275	148	148
query27	2405	498	350	350
query28	3350	2186	2195	2186
query29	580	576	439	439
query30	277	221	196	196
query31	897	891	798	798
query32	72	64	63	63
query33	494	370	338	338
query34	815	855	544	544
query35	806	842	768	768
query36	952	1019	883	883
query37	138	100	80	80
query38	4177	4307	4241	4241
query39	1513	1481	1486	1481
query40	215	125	109	109
query41	63	57	57	57
query42	124	119	112	112
query43	499	496	481	481
query44	1323	843	829	829
query45	178	177	165	165
query46	863	1028	661	661
query47	1906	1880	1779	1779
query48	414	441	312	312
query49	681	499	398	398
query50	667	701	428	428
query51	4226	4247	4271	4247
query52	122	112	103	103
query53	232	260	190	190
query54	581	594	505	505
query55	87	86	87	86
query56	305	307	293	293
query57	1211	1251	1170	1170
query58	277	285	256	256
query59	2813	2826	2748	2748
query60	329	328	319	319
query61	132	128	132	128
query62	762	744	670	670
query63	238	186	214	186
query64	2129	1048	684	684
query65	4235	4116	4162	4116
query66	758	393	301	301
query67	15772	15639	15551	15551
query68	9099	871	517	517
query69	536	323	272	272
query70	1217	1146	1077	1077
query71	515	323	300	300
query72	5323	4731	4645	4645
query73	1429	622	352	352
query74	8955	8974	9062	8974
query75	4091	3191	2681	2681
query76	4284	1196	774	774
query77	653	451	291	291
query78	10110	10414	9341	9341
query79	1939	828	568	568
query80	603	523	445	445
query81	483	262	220	220
query82	421	124	97	97
query83	251	248	226	226
query84	302	110	93	93
query85	773	356	418	356
query86	357	298	259	259
query87	4462	4436	4323	4323
query88	3408	2268	2258	2258
query89	392	315	284	284
query90	1955	200	206	200
query91	148	139	115	115
query92	72	62	54	54
query93	1241	954	580	580
query94	693	413	305	305
query95	371	295	284	284
query96	506	554	274	274
query97	2742	2764	2668	2668
query98	219	208	202	202
query99	1674	1401	1275	1275
Total cold run time: 300249 ms
Total hot run time: 193247 ms

@doris-robot
Copy link

doris-robot commented Jun 9, 2025

ClickBench: Total hot run time: 28.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 28da1221d08268bb825a13a11d31303bf4a3164e, data reload: false

query1	0.04	0.04	0.04
query2	0.12	0.10	0.11
query3	0.25	0.20	0.19
query4	1.60	0.19	0.19
query5	0.45	0.44	0.45
query6	1.17	0.66	0.66
query7	0.03	0.02	0.02
query8	0.05	0.04	0.03
query9	0.60	0.52	0.51
query10	0.56	0.58	0.56
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.59	0.59
query14	0.79	0.80	0.81
query15	0.86	0.86	0.84
query16	0.39	0.38	0.37
query17	1.03	1.03	1.02
query18	0.23	0.20	0.21
query19	1.89	1.80	1.80
query20	0.01	0.01	0.02
query21	15.39	0.88	0.54
query22	0.76	1.19	0.61
query23	15.03	1.41	0.62
query24	7.04	1.54	0.56
query25	0.50	0.16	0.10
query26	0.56	0.16	0.14
query27	0.05	0.05	0.04
query28	9.76	0.92	0.44
query29	12.54	3.98	3.39
query30	0.25	0.10	0.06
query31	2.84	0.60	0.38
query32	3.23	0.56	0.46
query33	3.05	3.06	3.12
query34	15.69	5.06	4.48
query35	4.52	4.48	4.45
query36	0.66	0.49	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.14	0.12
query41	0.08	0.02	0.03
query42	0.04	0.03	0.02
query43	0.04	0.02	0.03
Total cold run time: 103.37 s
Total hot run time: 28.72 s

freemandealer
freemandealer reviewed Jun 11, 2025
View reviewed changes
fe/fe-core/src/test/java/org/apache/doris/cloud/cache/CacheHotspotManagerTest.java

import mockit.Mock;
import mockit.MockUp;
import org.apache.commons.lang3.tuple.Triple;
Copy link
Contributor

@freemandealer freemandealer Jun 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just ask: what's wrong with the original package? @kaijchen

Copy link
Member Author

@kaijchen kaijchen Jun 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for gson support

freemandealer
freemandealer approved these changes Jun 11, 2025
View reviewed changes
Copy link
Contributor

@freemandealer freemandealer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jun 11, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jun 17, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

dataroaring
dataroaring approved these changes Jun 17, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 3c6fd95 into apache:master Jun 17, 2025
31 of 32 checks passed
kaijchen added a commit to kaijchen/doris that referenced this pull request Jun 25, 2025
@kaijchen
[feat](warmup) display tables in SHOW WARM UP JOB results (apache#51594)
444bbfa 
Enhance the `SHOW WARM UP JOB` command to display tables for "warm up by
tables" jobs.
Table names are shown in the format `db.table.partition`, separated by
commas.

For example:

```
mysql> show warm up job;
+---------------+--------------+----------+-------+-------------------------+-------------+----------+-------------------------+--------+------------------+
| JobId         | ComputeGroup | Status   | Type  | CreateTime              | FinishBatch | AllBatch | FinishTime              | ErrMsg | Tables           |
+---------------+--------------+----------+-------+-------------------------+-------------+----------+-------------------------+--------+------------------+
| 1749488013342 | cluster1     | FINISHED | TABLE | 2025-06-10 01:01:43.670 | 1           | 1        | 2025-06-10 01:01:46.344 |        | test.y, test.t.t |
| 1749488013329 | cluster1     | FINISHED | TABLE | 2025-06-10 00:59:41.135 | 1           | 1        | 2025-06-10 00:59:43.336 |        | test.y, test.t   |
| 1749488013306 | cluster1     | FINISHED | TABLE | 2025-06-10 00:58:43.781 | 1           | 1        | 2025-06-10 00:58:46.332 |        | test.y           |
+---------------+--------------+----------+-------+-------------------------+-------------+----------+-------------------------+--------+------------------+
```

None
@kaijchen kaijchen mentioned this pull request Jun 25, 2025
Merged
dataroaring pushed a commit that referenced this pull request Jun 25, 2025
@kaijchen
[feat](warmup) display tables in SHOW WARM UP JOB results (#51594) (#…
294db35 
…52291)

backport #51594
kaijchen added a commit to kaijchen/doris that referenced this pull request Jun 27, 2025
@kaijchen
[feat](warmup) display tables in SHOW WARM UP JOB results (apache#51594)
e99aeb2 
backport apache#51594
@kaijchen kaijchen mentioned this pull request Jun 27, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jun 28, 2025
@kaijchen
branch-3.1: [feat](warmup) display tables in SHOW WARM UP JOB results #…
c56791d 
…51594 (#52419)

backport #51594
koarz pushed a commit to koarz/doris that referenced this pull request Jul 3, 2025
@kaijchen @koarz
[feat](warmup) display tables in SHOW WARM UP JOB results (apache#51594…
e193d33 
…) (apache#52291)

backport apache#51594
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
@kaijchen kaijchen mentioned this pull request Aug 22, 2025
Merged
16 tasks
dataroaring pushed a commit that referenced this pull request Aug 22, 2025
@kaijchen
[fix](warmup) fix show warm up tables (#55192)
4797096 
### What problem does this PR solve?

Related PR: #51594  #52370 

Problem Summary:

Fix show warm up jobs after merging periodic and event driven warmup
feature.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

@w41ter w41ter Awaiting requested review from w41ter w41ter is a code owner

+1 more reviewer

@freemandealer freemandealer freemandealer approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

dev/3.0.7-merged dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@kaijchen @Thearas @doris-robot @freemandealer @dataroaring @morrySnow