Skip to content

Conversation

@Yukang-Lian
Copy link
Collaborator

@Yukang-Lian Yukang-Lian commented Jul 10, 2024

Pick #34043

The usage is curl http://{ip}:{host}/api/compaction/run_status?tablet_id={tablet_id} e.g. curl http://127.0.0.1:8040/api/compaction/run_status?tablet_id=10084

If full compaction is running, the output will be

{
"status" : "Success",
"run_status" : true,
"msg" : "compaction task for this tablet is running",
"tablet_id" : 10084,
"compact_type" : "full"
}

else the ouput will be

{
"status" : "Success",
"run_status" : false,
"msg" : "compaction task for this tablet is not running",
"tablet_id" : 10084,
"compact_type" : "full"
}

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Yukang-Lian Yukang-Lian changed the title [Enhancement](full compaction) Add run status support for full compaction (#34043) [cherry-pick](branch-2.0) Pick "[Enhancement](full compaction) Add run status support for full compaction (#34043)" Jul 10, 2024
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@dataroaring
Copy link
Contributor

run buildall

…tion (apache#34043)

* The usage is `curl http://{ip}:{host}/api/compaction/run_status?tablet_id={tablet_id}`
e.g. `curl http://127.0.0.1:8040/api/compaction/run_status?tablet_id=10084`

If full compaction is running, the output will be
```
{
"status" : "Success",
"run_status" : true,
"msg" : "compaction task for this tablet is running",
"tablet_id" : 10084,
"compact_type" : "full"
}
```
else the ouput will be
```
{
"status" : "Success",
"run_status" : false,
"msg" : "compaction task for this tablet is not running",
"tablet_id" : 10084,
"compact_type" : "full"
}
```

* 2
@Yukang-Lian
Copy link
Collaborator Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

: Compaction(tablet, "FullCompaction:" + std::to_string(tablet->tablet_id())) {}

FullCompaction::~FullCompaction() {}
FullCompaction::~FullCompaction() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: use '= default' to define a trivial destructor [modernize-use-equals-default]

FullCompaction::~FullCompaction() {
                ^

@doris-robot
Copy link

TPC-H: Total hot run time: 49959 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d621860205db886b2f926cbd4b2df8ca92123e73, data reload: false

------ Round 1 ----------------------------------
q1	17581	4486	4382	4382
q2	2064	158	150	150
q3	10468	1949	1938	1938
q4	10296	1290	1330	1290
q5	8536	3896	3898	3896
q6	259	145	127	127
q7	2048	1598	1568	1568
q8	9285	2776	2733	2733
q9	10651	10282	10223	10223
q10	8613	3508	3518	3508
q11	432	249	252	249
q12	476	307	315	307
q13	18371	3959	4029	3959
q14	358	327	319	319
q15	513	459	467	459
q16	674	564	579	564
q17	1156	979	961	961
q18	7277	6822	6924	6822
q19	1824	1668	1631	1631
q20	537	312	293	293
q21	4511	4179	4145	4145
q22	540	435	449	435
Total cold run time: 116470 ms
Total hot run time: 49959 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4390	4420	4644	4420
q2	327	234	237	234
q3	4230	4189	4194	4189
q4	2775	2745	2767	2745
q5	7156	7130	7128	7128
q6	243	123	121	121
q7	3261	2874	2831	2831
q8	4373	4521	4553	4521
q9	16947	16836	16775	16775
q10	4272	4286	4332	4286
q11	784	669	687	669
q12	1032	858	834	834
q13	6781	3744	3725	3725
q14	452	421	432	421
q15	503	451	454	451
q16	741	681	677	677
q17	3960	3835	3782	3782
q18	8815	8831	8798	8798
q19	1753	1739	1689	1689
q20	2383	2132	2115	2115
q21	8558	8547	8624	8547
q22	1075	992	1024	992
Total cold run time: 84811 ms
Total hot run time: 79950 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.86% (8119/21446)
Line Coverage: 29.53% (66544/225324)
Region Coverage: 29.00% (34296/118247)
Branch Coverage: 24.88% (17621/70828)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d621860205db886b2f926cbd4b2df8ca92123e73_d621860205db886b2f926cbd4b2df8ca92123e73/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 202814 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d621860205db886b2f926cbd4b2df8ca92123e73, data reload: false

query1	935	414	383	383
query2	6524	2795	2693	2693
query3	6916	202	204	202
query4	20183	18183	17982	17982
query5	19738	6519	6505	6505
query6	281	216	235	216
query7	4146	300	299	299
query8	461	400	397	397
query9	3063	2644	2588	2588
query10	398	300	287	287
query11	11373	10724	10682	10682
query12	127	76	73	73
query13	5590	705	706	705
query14	17810	13253	13463	13253
query15	358	250	243	243
query16	6472	291	264	264
query17	1706	1460	874	874
query18	2328	423	422	422
query19	220	143	149	143
query20	83	78	80	78
query21	190	100	96	96
query22	5110	4953	4923	4923
query23	32534	32103	31945	31945
query24	7000	6495	6426	6426
query25	518	445	428	428
query26	528	167	166	166
query27	1887	296	304	296
query28	6116	2334	2273	2273
query29	3046	2854	2871	2854
query30	240	161	170	161
query31	913	736	736	736
query32	67	59	64	59
query33	407	271	248	248
query34	849	488	495	488
query35	1110	884	875	875
query36	1451	1177	1227	1177
query37	92	60	62	60
query38	3054	2928	2921	2921
query39	1394	1324	1323	1323
query40	207	95	95	95
query41	47	45	45	45
query42	85	83	92	83
query43	790	853	731	731
query44	1161	703	709	703
query45	251	237	238	237
query46	1234	980	986	980
query47	1761	1773	1655	1655
query48	1019	712	722	712
query49	641	370	363	363
query50	867	604	632	604
query51	4768	4649	4756	4649
query52	97	89	91	89
query53	463	330	318	318
query54	2631	2500	2465	2465
query55	99	78	86	78
query56	247	199	217	199
query57	1346	1050	1046	1046
query58	215	206	193	193
query59	4357	4159	3774	3774
query60	223	206	207	206
query61	98	92	96	92
query62	850	519	445	445
query63	498	341	348	341
query64	2605	1517	1508	1508
query65	3645	3577	3533	3533
query66	786	381	382	381
query67	15454	15111	16314	15111
query68	10113	664	663	663
query69	571	351	354	351
query70	1831	1296	1535	1296
query71	420	311	311	311
query72	6500	3472	3510	3472
query73	743	319	306	306
query74	6301	5818	5813	5813
query75	5466	3635	3762	3635
query76	6553	1174	1205	1174
query77	1105	260	249	249
query78	12463	11548	11543	11543
query79	7752	649	638	638
query80	1407	409	406	406
query81	491	241	231	231
query82	1707	99	98	98
query83	173	137	132	132
query84	257	70	71	70
query85	877	325	328	325
query86	351	304	300	300
query87	3269	2998	3006	2998
query88	4739	2289	2301	2289
query89	382	273	275	273
query90	1970	208	207	207
query91	183	154	152	152
query92	59	52	55	52
query93	6622	553	595	553
query94	710	207	208	207
query95	1122	1029	1052	1029
query96	634	321	319	319
query97	6481	6302	6387	6302
query98	184	175	172	172
query99	2993	868	933	868
Total cold run time: 316092 ms
Total hot run time: 202814 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.5 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d621860205db886b2f926cbd4b2df8ca92123e73, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.05	0.06
query4	1.78	0.10	0.10
query5	0.54	0.53	0.52
query6	1.23	0.61	0.62
query7	0.01	0.00	0.01
query8	0.03	0.02	0.02
query9	0.53	0.49	0.49
query10	0.54	0.54	0.54
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.61	0.61	0.62
query14	0.79	0.78	0.79
query15	0.76	0.76	0.76
query16	0.36	0.38	0.39
query17	0.98	1.02	1.02
query18	0.23	0.23	0.27
query19	1.87	1.77	1.82
query20	0.01	0.01	0.01
query21	15.50	0.54	0.54
query22	2.01	2.04	1.48
query23	17.34	0.97	1.12
query24	6.06	0.85	0.96
query25	0.38	0.06	0.05
query26	0.63	0.17	0.16
query27	0.05	0.05	0.04
query28	7.29	0.73	0.71
query29	12.65	2.26	2.27
query30	0.59	0.54	0.53
query31	2.80	0.38	0.37
query32	3.39	0.51	0.48
query33	3.06	3.10	3.09
query34	15.24	4.83	4.79
query35	4.87	4.83	4.82
query36	1.05	1.01	1.01
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.13 s
Total hot run time: 30.5 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d621860205db886b2f926cbd4b2df8ca92123e73 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@dataroaring dataroaring merged commit e912ff6 into apache:branch-2.0 Jul 15, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
…n status support for full compaction (apache#34043)" (apache#37607)

Pick apache#34043

The usage is `curl
http://{ip}:{host}/api/compaction/run_status?tablet_id={tablet_id}` e.g.
`curl http://127.0.0.1:8040/api/compaction/run_status?tablet_id=10084`

If full compaction is running, the output will be
```
{
"status" : "Success",
"run_status" : true,
"msg" : "compaction task for this tablet is running",
"tablet_id" : 10084,
"compact_type" : "full"
}
```
else the ouput will be
```
{
"status" : "Success",
"run_status" : false,
"msg" : "compaction task for this tablet is not running",
"tablet_id" : 10084,
"compact_type" : "full"
}
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants