[fix](load) fix memtable mem_tracker too large #25205

kaijchen · 2023-10-10T02:10:08Z

Proposed changes

Target block allocated bytes may be much larger than target block bytes in some conditions, causing memtable mem_tracker to be too large.
Use target block bytes for counting memtable mem_usage instead of target block allocated bytes.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

kaijchen · 2023-10-10T02:10:18Z

run buildall

github-actions · 2023-10-10T02:18:08Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-10-10T02:43:10Z

TeamCity be ut coverage result:
Function Coverage: 36.31% (8151/22448)
Line Coverage: 28.45% (65195/229173)
Region Coverage: 27.38% (33786/123397)
Branch Coverage: 24.06% (17217/71564)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6d9edd724bb331dcc8bc85e5390ad3c94b3ffc7e_6d9edd724bb331dcc8bc85e5390ad3c94b3ffc7e/report/index.html

dataroaring

LGTM

github-actions · 2023-10-11T08:52:44Z

PR approved by at least one committer and no changes requested.

freemandealer

LGTM

github-actions · 2023-10-11T08:52:48Z

PR approved by anyone and no changes requested.

kaijchen · 2023-10-18T09:03:50Z

run buildall

github-actions · 2023-10-18T09:12:17Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-10-18T09:22:40Z

TeamCity be ut coverage result:
Function Coverage: 37.00% (8277/22369)
Line Coverage: 29.07% (66383/228342)
Region Coverage: 27.73% (34452/124259)
Branch Coverage: 24.34% (17503/71914)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4ad55c1eda48ed4003570241419de2023b6ca708_4ad55c1eda48ed4003570241419de2023b6ca708/report/index.html

zhannngchen

LGTM

hello-stephen · 2023-10-25T06:55:34Z

run clickbench

dataroaring · 2023-11-14T10:19:55Z

run buildall

github-actions · 2023-11-14T10:23:33Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-11-14T10:40:05Z

TeamCity be ut coverage result:
Function Coverage: 36.74% (8418/22915)
Line Coverage: 29.26% (68387/233715)
Region Coverage: 27.86% (35339/126866)
Branch Coverage: 24.65% (18063/73280)
Coverage Report: http://coverage.selectdb-in.cc/coverage/76f78efadff157b8089b4a438c0434b8fc4284e0_76f78efadff157b8089b4a438c0434b8fc4284e0/report/index.html

doris-robot · 2023-11-14T12:51:02Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 76f78efadff157b8089b4a438c0434b8fc4284e0, data reload: true

run tpch-sf100 query with default conf and session variables
q1	5350	5105	5049	5049
q2	385	126	124	124
q3	2069	1994	1997	1994
q4	1900	1320	1346	1320
q5	4039	3954	3987	3954
q6	254	119	119	119
q7	1417	870	877	870
q8	2848	2797	2790	2790
q9	11334	9628	9679	9628
q10	9947	3542	3550	3542
q11	374	257	248	248
q12	453	279	279	279
q13	4593	4125	4107	4107
q14	309	279	270	270
q15	587	536	540	536
q16	678	583	585	583
q17	1138	1091	1077	1077
q18	8084	7689	7589	7589
q19	1720	1677	1673	1673
q20	552	298	287	287
q21	4724	4342	4380	4342
q22	487	394	409	394
Total cold run time: 63242 ms
Total hot run time: 50775 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5045	5021	5087	5021
q2	329	209	226	209
q3	4126	4066	3982	3982
q4	2821	2772	2777	2772
q5	9718	9691	9693	9691
q6	241	121	119	119
q7	2659	2284	2263	2263
q8	4889	4823	4789	4789
q9	13371	13173	13297	13173
q10	4130	4182	4430	4182
q11	790	658	680	658
q12	1019	810	816	810
q13	4296	3913	3827	3827
q14	387	344	332	332
q15	638	560	539	539
q16	769	676	695	676
q17	3876	3914	3886	3886
q18	9684	9338	9416	9338
q19	1905	1794	1789	1789
q20	2418	2058	2058	2058
q21	8938	8668	8836	8668
q22	982	871	882	871
Total cold run time: 83031 ms
Total hot run time: 79653 ms

doris-robot · 2023-11-17T10:11:59Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 76f78efadff157b8089b4a438c0434b8fc4284e0, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4990	4697	4707	4697
q2	365	147	143	143
q3	2040	1940	1940	1940
q4	1398	1241	1248	1241
q5	4010	3972	4031	3972
q6	255	130	132	130
q7	1414	894	882	882
q8	2755	2808	2795	2795
q9	9961	9648	9563	9563
q10	3495	3547	3520	3520
q11	368	236	257	236
q12	444	292	297	292
q13	4571	3843	3779	3779
q14	323	288	286	286
q15	585	540	528	528
q16	666	584	584	584
q17	1140	985	922	922
q18	7841	7400	7498	7400
q19	1673	1673	1693	1673
q20	560	314	299	299
q21	4409	3984	3981	3981
q22	481	364	373	364
Total cold run time: 53744 ms
Total hot run time: 49227 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4618	4607	4602	4602
q2	341	224	267	224
q3	4039	4004	3993	3993
q4	2714	2707	2702	2702
q5	9640	9717	9774	9717
q6	247	121	124	121
q7	2634	2294	2330	2294
q8	4455	4472	4438	4438
q9	13274	13103	13102	13102
q10	4120	4196	4175	4175
q11	764	734	719	719
q12	975	822	817	817
q13	4266	3561	3552	3552
q14	389	348	348	348
q15	577	525	524	524
q16	740	664	698	664
q17	3930	3826	3867	3826
q18	9560	9061	9160	9061
q19	1795	1757	1797	1757
q20	2426	2084	2063	2063
q21	8873	8611	8632	8611
q22	878	801	781	781
Total cold run time: 81255 ms
Total hot run time: 78091 ms

kaijchen · 2023-12-06T08:37:27Z

run buildall

github-actions · 2023-12-06T08:44:04Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-12-06T09:45:26Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 6ac81f1f08946100c3f0c915362b935638197cc5, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4672	4381	4364	4364
q2	362	157	154	154
q3	1466	1277	1290	1277
q4	1137	931	924	924
q5	3181	3207	3188	3188
q6	248	130	129	129
q7	980	501	487	487
q8	2203	2230	2176	2176
q9	6636	6652	6654	6652
q10	3221	3273	3283	3273
q11	322	201	201	201
q12	356	212	212	212
q13	4520	3781	3774	3774
q14	244	205	220	205
q15	567	518	526	518
q16	448	386	382	382
q17	1002	614	571	571
q18	7334	6891	8419	6891
q19	1537	1450	1446	1446
q20	597	336	332	332
q21	3126	2657	2684	2657
q22	360	293	296	293
Total cold run time: 44519 ms
Total hot run time: 40106 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4363	4341	4323	4323
q2	270	162	170	162
q3	3513	3518	3513	3513
q4	2372	2372	2370	2370
q5	5730	5744	5733	5733
q6	240	122	124	122
q7	2387	1841	1876	1841
q8	3544	3522	3518	3518
q9	9013	8984	9000	8984
q10	3906	3997	4000	3997
q11	507	390	384	384
q12	773	587	593	587
q13	4268	3562	3558	3558
q14	282	259	250	250
q15	574	526	523	523
q16	499	463	461	461
q17	1886	1843	1837	1837
q18	8612	8301	8306	8301
q19	1744	1763	1761	1761
q20	2249	1944	1939	1939
q21	6514	6163	6180	6163
q22	505	419	425	419
Total cold run time: 63751 ms
Total hot run time: 60746 ms

kaijchen · 2023-12-06T11:02:44Z

run clickbench

doris-robot · 2023-12-06T11:34:36Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.07 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17194381377 Bytes

dataroaring

LGTM

doris-robot · 2023-12-06T13:32:51Z

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 6ac81f1f08946100c3f0c915362b935638197cc5, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4690	4484	4422	4422
q2	362	152	153	152
q3	1457	1226	1160	1160
q4	1128	935	878	878
q5	3187	3186	3177	3177
q6	248	129	126	126
q7	1007	501	489	489
q8	2167	2254	2186	2186
q9	6704	6697	6646	6646
q10	3194	3253	3250	3250
q11	328	195	204	195
q12	352	206	215	206
q13	4543	3815	3782	3782
q14	243	212	212	212
q15	562	530	530	530
q16	445	400	386	386
q17	1015	588	548	548
q18	7437	7367	7889	7367
q19	1526	1394	1408	1394
q20	516	431	322	322
q21	3100	2703	2611	2611
q22	350	298	295	295
Total cold run time: 44561 ms
Total hot run time: 40334 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4433	4363	4356	4356
q2	268	167	172	167
q3	3553	3532	3523	3523
q4	2401	2392	2392	2392
q5	5729	5738	5731	5731
q6	239	120	121	120
q7	2380	1875	1911	1875
q8	3524	3515	3530	3515
q9	9107	9002	8979	8979
q10	3888	4016	3980	3980
q11	504	391	382	382
q12	761	608	585	585
q13	4271	3517	3561	3517
q14	284	247	255	247
q15	577	520	521	520
q16	499	471	480	471
q17	1864	1865	1889	1865
q18	8721	8353	8373	8353
q19	1732	1774	1755	1755
q20	2255	1942	1934	1934
q21	6535	6182	6119	6119
q22	492	412	400	400
Total cold run time: 64017 ms
Total hot run time: 60786 ms

xiaokang added the dev/2.0.3 label Oct 10, 2023

dataroaring approved these changes Oct 11, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 11, 2023

freemandealer approved these changes Oct 11, 2023

View reviewed changes

github-actions bot added the reviewed label Oct 11, 2023

zhannngchen approved these changes Oct 24, 2023

View reviewed changes

dataroaring added dev/2.0.4 and removed dev/2.0.3 labels Nov 11, 2023

[fix](load) fix memtable mem_tracker too large

76f78ef

dataroaring force-pushed the fix-memtable-tracker branch from 4ad55c1 to 76f78ef Compare November 14, 2023 10:16

dataroaring self-assigned this Nov 25, 2023

Merge branch 'master' into fix-memtable-tracker

6ac81f1

dataroaring approved these changes Dec 6, 2023

View reviewed changes

dataroaring removed the dev/2.0.4 label Dec 6, 2023

dataroaring merged commit d4f9e12 into apache:master Dec 6, 2023

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[fix](load) fix memtable mem_tracker too large (apache#25205)

fcc19cd

kaijchen mentioned this pull request Dec 18, 2023

[fix](load) fix orphan mem limiter too large #28600

Merged

kaijchen deleted the fix-memtable-tracker branch July 31, 2025 06:10

[fix](load) fix memtable mem_tracker too large #25205

[fix](load) fix memtable mem_tracker too large #25205

Uh oh!

Conversation

kaijchen commented Oct 10, 2023

Proposed changes

Further comments

Uh oh!

kaijchen commented Oct 10, 2023

Uh oh!

github-actions bot commented Oct 10, 2023

Uh oh!

doris-robot commented Oct 10, 2023

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 11, 2023

Uh oh!

freemandealer left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 11, 2023

Uh oh!

kaijchen commented Oct 18, 2023

Uh oh!

github-actions bot commented Oct 18, 2023

Uh oh!

doris-robot commented Oct 18, 2023

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Oct 25, 2023

Uh oh!

dataroaring commented Nov 14, 2023

Uh oh!

github-actions bot commented Nov 14, 2023

Uh oh!

doris-robot commented Nov 14, 2023

Uh oh!

doris-robot commented Nov 14, 2023

Uh oh!

doris-robot commented Nov 17, 2023

Uh oh!

kaijchen commented Dec 6, 2023

Uh oh!

github-actions bot commented Dec 6, 2023

Uh oh!

doris-robot commented Dec 6, 2023

Uh oh!

kaijchen commented Dec 6, 2023

Uh oh!

doris-robot commented Dec 6, 2023

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Dec 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants