Skip to content

[chore](memory) remove memory tracker hook code and simplify cpu controller code #53794

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 1 commit into from
Jul 25, 2025
Merged

[chore](memory) remove memory tracker hook code and simplify cpu controller code #53794

yiguolei merged 1 commit into from
Jul 25, 2025

Conversation

@yiguolei
Copy link
Contributor

@yiguolei yiguolei commented Jul 23, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 23, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yiguolei yiguolei force-pushed the simplify_memtracker branch from d41510a to 6e1ffe5 Compare July 23, 2025 10:29
@yiguolei
Copy link
Contributor Author

yiguolei commented Jul 23, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 23, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.42% (1302/1619)
Line Coverage 65.78% (21799/33140)
Region Coverage 67.03% (10945/16328)
Branch Coverage 56.63% (5763/10176)

@doris-robot
Copy link

doris-robot commented Jul 23, 2025

TPC-H: Total hot run time: 33790 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6e1ffe549472cece403ca57e8e7864feb17d2014, data reload: false

------ Round 1 ----------------------------------
q1	17584	5419	5201	5201
q2	1923	278	175	175
q3	10335	1315	710	710
q4	10216	1008	524	524
q5	7557	2374	2331	2331
q6	176	164	131	131
q7	890	757	596	596
q8	9301	1363	1063	1063
q9	6769	5152	5032	5032
q10	6885	2365	1951	1951
q11	462	280	267	267
q12	351	359	221	221
q13	17770	3472	3026	3026
q14	222	230	220	220
q15	538	458	481	458
q16	432	410	374	374
q17	576	845	354	354
q18	7439	7162	7108	7108
q19	3011	1048	538	538
q20	344	306	212	212
q21	3508	3077	2260	2260
q22	1020	1061	1038	1038
Total cold run time: 107309 ms
Total hot run time: 33790 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5450	5255	5213	5213
q2	230	311	220	220
q3	2081	2571	2180	2180
q4	1358	1733	1334	1334
q5	4137	4534	4261	4261
q6	217	178	132	132
q7	2194	2254	1809	1809
q8	2567	2617	2559	2559
q9	7319	7343	7455	7343
q10	3179	3350	2990	2990
q11	540	528	498	498
q12	685	831	641	641
q13	3618	3685	3240	3240
q14	358	339	318	318
q15	505	493	595	493
q16	428	488	446	446
q17	1155	1491	1412	1412
q18	7993	7726	7653	7653
q19	10705	885	849	849
q20	3205	1965	1868	1868
q21	12007	4262	4278	4262
q22	1071	1022	973	973
Total cold run time: 71002 ms
Total hot run time: 50694 ms

@doris-robot
Copy link

doris-robot commented Jul 23, 2025

TPC-DS: Total hot run time: 187494 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6e1ffe549472cece403ca57e8e7864feb17d2014, data reload: false

query1	994	403	418	403
query2	6535	2037	1701	1701
query3	6753	221	222	221
query4	26473	23354	23070	23070
query5	4411	622	558	558
query6	339	243	239	239
query7	4641	497	299	299
query8	282	239	229	229
query9	8596	2994	2964	2964
query10	510	352	292	292
query11	15821	15000	14826	14826
query12	179	136	134	134
query13	1644	518	395	395
query14	8669	5927	5876	5876
query15	213	195	172	172
query16	7131	659	423	423
query17	946	734	614	614
query18	1987	443	312	312
query19	219	210	187	187
query20	141	138	135	135
query21	218	133	106	106
query22	4328	4023	3784	3784
query23	34517	34442	34348	34348
query24	6093	2362	2435	2362
query25	500	520	434	434
query26	717	289	161	161
query27	2367	501	358	358
query28	3006	2278	2275	2275
query29	691	597	488	488
query30	287	237	193	193
query31	914	846	765	765
query32	86	76	83	76
query33	551	413	366	366
query34	795	855	508	508
query35	811	821	738	738
query36	1004	1051	966	966
query37	129	107	91	91
query38	3951	4034	4051	4034
query39	1441	1384	1421	1384
query40	244	144	134	134
query41	69	55	57	55
query42	134	131	125	125
query43	515	516	483	483
query44	1395	850	862	850
query45	197	193	185	185
query46	928	1059	659	659
query47	1787	1877	1726	1726
query48	412	437	319	319
query49	674	499	410	410
query50	673	698	428	428
query51	5524	5593	5424	5424
query52	131	131	119	119
query53	257	293	234	234
query54	645	649	569	569
query55	89	84	91	84
query56	359	365	352	352
query57	1203	1232	1105	1105
query58	338	327	329	327
query59	2678	2683	2538	2538
query60	418	411	386	386
query61	126	122	120	120
query62	754	756	666	666
query63	250	225	216	216
query64	2464	1230	785	785
query65	4195	4064	4086	4064
query66	1010	435	353	353
query67	16331	16677	15926	15926
query68	14251	575	566	566
query69	999	308	292	292
query70	1283	1102	1167	1102
query71	705	337	326	326
query72	9233	2267	2334	2267
query73	3097	620	350	350
query74	9005	9005	8797	8797
query75	7370	3123	2676	2676
query76	8576	1261	776	776
query77	1143	407	330	330
query78	9672	10608	9732	9732
query79	13572	615	573	573
query80	1527	559	492	492
query81	573	257	228	228
query82	436	153	118	118
query83	372	278	275	275
query84	307	95	83	83
query85	927	419	322	322
query86	382	338	305	305
query87	4363	4316	4197	4197
query88	4027	2243	2246	2243
query89	467	367	308	308
query90	2666	227	227	227
query91	144	138	112	112
query92	87	72	69	69
query93	4528	968	645	645
query94	1071	389	267	267
query95	406	324	314	314
query96	508	598	288	288
query97	2727	2702	2599	2599
query98	259	233	222	222
query99	1481	1442	1264	1264
Total cold run time: 308854 ms
Total hot run time: 187494 ms

@doris-robot
Copy link

doris-robot commented Jul 23, 2025

ClickBench: Total hot run time: 33.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6e1ffe549472cece403ca57e8e7864feb17d2014, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.05
query3	0.24	0.07	0.08
query4	1.66	0.11	0.11
query5	0.44	0.45	0.45
query6	1.17	0.70	0.68
query7	0.03	0.02	0.02
query8	0.05	0.03	0.03
query9	0.58	0.48	0.47
query10	0.52	0.52	0.52
query11	0.16	0.10	0.11
query12	0.15	0.11	0.11
query13	0.66	0.67	0.65
query14	0.97	1.17	0.97
query15	0.92	0.90	0.91
query16	0.39	0.38	0.40
query17	1.09	1.04	1.04
query18	0.22	0.21	0.21
query19	1.99	1.84	1.96
query20	0.02	0.02	0.02
query21	15.36	0.86	0.55
query22	0.79	1.12	0.73
query23	14.90	1.18	0.61
query24	6.81	1.60	0.90
query25	0.49	0.20	0.07
query26	0.55	0.18	0.15
query27	0.06	0.05	0.06
query28	10.09	0.85	0.46
query29	12.54	3.84	3.35
query30	3.01	3.06	2.92
query31	2.81	0.56	0.40
query32	3.23	0.58	0.48
query33	3.03	3.26	3.22
query34	15.92	5.30	4.91
query35	4.88	4.98	4.95
query36	0.67	0.50	0.50
query37	0.09	0.07	0.07
query38	0.06	0.05	0.04
query39	0.03	0.03	0.03
query40	0.16	0.14	0.13
query41	0.09	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 107.01 s
Total hot run time: 33.25 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 23, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/14) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.50% (15926/27699)
Line Coverage 46.29% (143068/309082)
Region Coverage 35.77% (107822/301464)
Branch Coverage 38.25% (47550/124311)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 23, 2025

BE Regression && UT Coverage Report

Increment line coverage 14.29% (2/14) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.10% (22048/27185)
Line Coverage 73.73% (227532/308611)
Region Coverage 61.23% (189308/309186)
Branch Coverage 65.05% (81769/125699)

@yiguolei yiguolei force-pushed the simplify_memtracker branch from 6e1ffe5 to 0d42595 Compare July 25, 2025 06:03
@yiguolei
Copy link
Contributor Author

yiguolei commented Jul 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 25, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.42% (1302/1619)
Line Coverage 65.79% (21805/33142)
Region Coverage 67.06% (10949/16328)
Branch Coverage 56.63% (5763/10176)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 25, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/14) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.56% (15936/27687)
Line Coverage 46.34% (143264/309127)
Region Coverage 35.76% (107909/301739)
Branch Coverage 38.30% (47637/124388)

@yiguolei yiguolei force-pushed the simplify_memtracker branch from 0d42595 to a4fcd6b Compare July 25, 2025 09:08
@yiguolei
Copy link
Contributor Author

yiguolei commented Jul 25, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 25, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.42% (1302/1619)
Line Coverage 65.79% (21803/33142)
Region Coverage 67.06% (10949/16328)
Branch Coverage 56.64% (5764/10176)

@doris-robot
Copy link

doris-robot commented Jul 25, 2025

TPC-H: Total hot run time: 33734 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a4fcd6bcf97d2e5442bff9848fecd1bf85b4a30f, data reload: false

------ Round 1 ----------------------------------
q1	17560	5395	5121	5121
q2	1939	268	186	186
q3	10318	1277	690	690
q4	10224	991	508	508
q5	7548	2237	2336	2237
q6	177	164	132	132
q7	877	786	607	607
q8	9308	1315	1122	1122
q9	6735	5120	5047	5047
q10	6897	2379	1968	1968
q11	501	294	285	285
q12	341	369	225	225
q13	17819	3512	3043	3043
q14	235	236	207	207
q15	530	465	455	455
q16	421	420	377	377
q17	598	846	373	373
q18	7544	7135	7113	7113
q19	2930	983	553	553
q20	328	327	227	227
q21	3482	3073	2265	2265
q22	1037	1011	993	993
Total cold run time: 107349 ms
Total hot run time: 33734 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5534	5323	5304	5304
q2	235	310	210	210
q3	2130	2534	2188	2188
q4	1321	1741	1329	1329
q5	4139	4478	4482	4478
q6	226	174	135	135
q7	1981	2012	2050	2012
q8	2543	2578	2497	2497
q9	7341	7336	7319	7319
q10	3125	3420	2922	2922
q11	580	511	473	473
q12	691	825	643	643
q13	3503	3891	3240	3240
q14	284	308	294	294
q15	510	463	461	461
q16	456	511	442	442
q17	1192	1481	1422	1422
q18	7926	7706	7776	7706
q19	12004	1074	1289	1074
q20	1990	2054	1775	1775
q21	14502	4242	4282	4242
q22	1031	1041	999	999
Total cold run time: 73244 ms
Total hot run time: 51165 ms

@doris-robot
Copy link

doris-robot commented Jul 25, 2025

TPC-DS: Total hot run time: 172338 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a4fcd6bcf97d2e5442bff9848fecd1bf85b4a30f, data reload: false

============================================
query1	1011	402	409	402
query2	6525	1774	1746	1746
query3	6739	227	228	227
query4	27651	24022	22939	22939
query5	4379	626	527	527
query6	331	250	223	223
query7	4644	513	290	290
query8	280	236	226	226
query9	8573	2980	2962	2962
query10	476	357	314	314
query11	15423	15058	15395	15058
query12	183	139	133	133
query13	1661	538	431	431
query14	8645	6055	6043	6043
query15	235	196	173	173
query16	7780	633	453	453
query17	1634	800	644	644
query18	2058	449	328	328
query19	322	212	189	189
query20	156	147	133	133
query21	222	124	106	106
query22	3981	3986	3794	3794
query23	34353	33980	34232	33980
query24	7537	2381	2382	2381
query25	569	512	441	441
query26	724	290	158	158
query27	2539	507	354	354
query28	3181	2277	2274	2274
query29	680	615	484	484
query30	296	232	206	206
query31	896	832	716	716
query32	100	84	79	79
query33	555	412	366	366
query34	820	853	514	514
query35	820	807	747	747
query36	1021	1059	952	952
query37	132	113	91	91
query38	4013	4067	4009	4009
query39	1439	1366	1350	1350
query40	243	145	131	131
query41	61	61	60	60
query42	144	126	129	126
query43	516	511	470	470
query44	1382	858	862	858
query45	200	188	182	182
query46	934	1057	661	661
query47	1816	1843	1772	1772
query48	407	422	318	318
query49	673	508	403	403
query50	653	682	430	430
query51	5515	5532	5540	5532
query52	144	137	121	121
query53	266	288	201	201
query54	647	649	591	591
query55	91	92	90	90
query56	381	363	356	356
query57	1176	1212	1128	1128
query58	349	338	348	338
query59	2537	2656	2503	2503
query60	416	414	396	396
query61	153	125	152	125
query62	840	731	636	636
query63	250	222	210	210
query64	2814	1066	814	814
query65	4240	4097	4082	4082
query66	897	449	358	358
query67	query68	19179	841	576	576
query69	1035	302	285	285
query70	1461	1166	1138	1138
query71	709	353	331	331
query72	9152	2305	2330	2305
query73	3681	628	365	365
query74	8976	8873	8733	8733
query75	7567	3113	2701	2701
query76	8914	1256	769	769
query77	1148	396	347	347
query78	10007	11041	10066	10066
query79	15138	601	590	590
query80	1961	551	500	500
query81	569	252	228	228
query82	520	153	118	118
query83	350	284	266	266
query84	287	102	85	85
query85	915	361	342	342
query86	368	329	310	310
query87	4367	4361	4292	4292
query88	5594	2244	2247	2244
query89	494	358	311	311
query90	2511	226	224	224
query91	151	139	128	128
query92	92	77	70	70
query93	6484	946	644	644
query94	1086	396	289	289
query95	430	339	329	329
query96	507	590	286	286
query97	2714	2765	2648	2648
query98	257	246	232	232
query99	1438	1399	1251	1251
Total cold run time: 308391 ms
Total hot run time: 172338 ms

@doris-robot
Copy link

doris-robot commented Jul 25, 2025

ClickBench: Total hot run time: 33.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a4fcd6bcf97d2e5442bff9848fecd1bf85b4a30f, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.05	0.04
query3	0.24	0.07	0.07
query4	1.62	0.11	0.11
query5	0.44	0.44	0.44
query6	1.18	0.70	0.70
query7	0.03	0.02	0.02
query8	0.06	0.04	0.04
query9	0.56	0.49	0.47
query10	0.53	0.52	0.51
query11	0.15	0.10	0.11
query12	0.15	0.11	0.11
query13	0.65	0.66	0.66
query14	1.00	1.06	1.17
query15	0.91	0.91	0.90
query16	0.38	0.38	0.39
query17	1.04	1.08	1.08
query18	0.22	0.21	0.21
query19	2.06	1.87	1.98
query20	0.02	0.01	0.01
query21	15.39	0.83	0.54
query22	0.79	1.17	0.80
query23	14.77	1.17	0.62
query24	6.47	0.75	0.70
query25	0.48	0.12	0.14
query26	0.63	0.16	0.15
query27	0.05	0.05	0.07
query28	9.58	0.89	0.45
query29	12.63	3.89	3.41
query30	3.11	2.97	2.93
query31	2.82	0.57	0.39
query32	3.24	0.58	0.49
query33	3.06	3.24	3.27
query34	15.82	5.33	4.87
query35	4.87	5.01	4.94
query36	0.72	0.53	0.49
query37	0.09	0.07	0.07
query38	0.05	0.05	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.05	0.03	0.04
Total cold run time: 106.31 s
Total hot run time: 33.37 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 25, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/14) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.57% (15942/27691)
Line Coverage 46.36% (143341/309216)
Region Coverage 35.76% (107917/301766)
Branch Coverage 38.31% (47659/124406)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 25, 2025

BE Regression && UT Coverage Report

Increment line coverage 14.29% (2/14) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 81.07% (22039/27185)
Line Coverage 73.70% (227591/308792)
Region Coverage 61.42% (190115/309518)
Branch Coverage 65.12% (81933/125814)

@yiguolei yiguolei changed the title Simplify memtracker [chore](memory) remove memory tracker hook code and simplify cpu controller code Jul 25, 2025
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 25, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jul 25, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 25, 2025

PR approved by anyone and no changes requested.

@shuke987
Copy link
Collaborator

shuke987 commented Jul 25, 2025

skip check_coverage

@shuke987
Copy link
Collaborator

shuke987 commented Jul 25, 2025

skip buildall

@yiguolei yiguolei merged commit 726c8d1 into apache:master Jul 25, 2025
31 of 33 checks passed
w41ter pushed a commit to w41ter/incubator-doris that referenced this pull request Jul 30, 2025
@yiguolei @w41ter
[chore](memory) remove memory tracker hook code and simplify cpu cont…
f48b558 
…roller code (apache#53794)
zhiqiang-hhhh pushed a commit to zhiqiang-hhhh/doris that referenced this pull request Aug 4, 2025
@yiguolei @zhiqiang-hhhh
[chore](memory) remove memory tracker hook code and simplify cpu cont…
801ffde 
…roller code (apache#53794)
@BiteTheDDDDt BiteTheDDDDt mentioned this pull request Aug 11, 2025
Merged
16 tasks
BiteTheDDDDt added a commit that referenced this pull request Aug 11, 2025
@BiteTheDDDDt
[Chore](memory) add jemalloc_hook to make jemalloc control malloc (#5…
852c306 
…4547)

In #53794 and #54028, we remove jemalloc_hook, but I found that without
a jemalloc hook would cause doris to allocate memory not to apply
jemalloc, but instead use glibc's malloc, so I added it back.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@zclllyybb zclllyybb zclllyybb approved these changes

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

@w41ter w41ter Awaiting requested review from w41ter w41ter is a code owner

+1 more reviewer

@zhiqiang-hhhh zhiqiang-hhhh zhiqiang-hhhh approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@yiguolei @hello-stephen @doris-robot @shuke987 @gavinchou @zhiqiang-hhhh @zclllyybb