branch-3.1: [fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable #52902 #52928

liaoxin01 · 2025-07-08T07:18:31Z

pick from：#52902

liaoxin01 · 2025-07-08T07:18:40Z

run buildall

hello-stephen · 2025-07-08T07:19:02Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

liaoxin01 · 2025-07-08T07:20:52Z

run buildall

doris-robot · 2025-07-08T08:26:00Z

TPC-H: Total hot run time: 39867 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 809b77dd4bb0c000ef542f5c63dab24c0c94ea10, data reload: false

------ Round 1 ----------------------------------
q1	17839	7027	6611	6611
q2	2130	215	186	186
q3	10574	1141	1169	1141
q4	10634	765	704	704
q5	7787	2890	2963	2890
q6	223	137	139	137
q7	1010	630	631	630
q8	9491	1996	2082	1996
q9	6819	6432	6426	6426
q10	7044	2329	2283	2283
q11	475	257	261	257
q12	399	213	221	213
q13	17789	2961	3007	2961
q14	245	205	206	205
q15	511	466	458	458
q16	472	372	381	372
q17	996	542	589	542
q18	7296	6737	6648	6648
q19	1324	1054	989	989
q20	457	203	201	201
q21	3863	3042	3162	3042
q22	1074	988	975	975
Total cold run time: 108452 ms
Total hot run time: 39867 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6657	6560	6608	6560
q2	333	238	240	238
q3	2957	2941	2938	2938
q4	2079	1815	1821	1815
q5	5761	5570	5659	5570
q6	203	127	124	124
q7	2195	1811	1798	1798
q8	3341	3494	3496	3494
q9	8772	8900	8859	8859
q10	3598	3536	3574	3536
q11	606	496	496	496
q12	811	617	583	583
q13	7837	3162	3148	3148
q14	302	265	284	265
q15	509	460	465	460
q16	480	443	468	443
q17	1851	1651	1600	1600
q18	8148	7400	7850	7400
q19	1721	1657	1476	1476
q20	2144	1852	1844	1844
q21	5081	5103	5132	5103
q22	1108	1060	1026	1026
Total cold run time: 66494 ms
Total hot run time: 58776 ms

doris-robot · 2025-07-08T08:38:22Z

TPC-DS: Total hot run time: 196394 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 809b77dd4bb0c000ef542f5c63dab24c0c94ea10, data reload: false

query1	1256	761	746	746
query2	6280	2010	1986	1986
query3	10804	4229	4286	4229
query4	33024	23402	23277	23277
query5	3520	466	430	430
query6	275	181	177	177
query7	4018	316	310	310
query8	286	248	220	220
query9	9567	2635	2635	2635
query10	467	269	273	269
query11	18314	15278	15135	15135
query12	163	110	107	107
query13	1554	436	414	414
query14	9777	7040	7619	7040
query15	281	192	197	192
query16	8003	484	499	484
query17	1718	584	577	577
query18	2168	321	317	317
query19	368	155	152	152
query20	121	113	118	113
query21	208	112	108	108
query22	4631	4512	4494	4494
query23	35137	33809	34140	33809
query24	11831	3020	2947	2947
query25	620	430	437	430
query26	1600	174	173	173
query27	2700	361	359	359
query28	7894	2215	2183	2183
query29	752	464	465	464
query30	275	161	164	161
query31	1039	821	865	821
query32	100	58	56	56
query33	794	313	327	313
query34	940	543	550	543
query35	882	740	746	740
query36	1127	934	957	934
query37	237	75	71	71
query38	4168	3986	3980	3980
query39	1524	1483	1448	1448
query40	267	106	102	102
query41	55	48	50	48
query42	118	103	105	103
query43	528	496	508	496
query44	1372	810	845	810
query45	191	177	172	172
query46	1210	761	801	761
query47	1987	1893	1917	1893
query48	448	355	348	348
query49	1101	397	409	397
query50	907	428	416	416
query51	7181	7027	7214	7027
query52	104	89	96	89
query53	261	192	187	187
query54	1294	479	491	479
query55	82	81	80	80
query56	278	259	254	254
query57	1304	1175	1143	1143
query58	238	209	211	209
query59	3191	2982	2993	2982
query60	276	257	248	248
query61	111	112	114	112
query62	858	695	661	661
query63	211	194	182	182
query64	5088	665	645	645
query65	3275	3210	3227	3210
query66	1350	292	306	292
query67	15759	15409	15418	15409
query68	4773	577	599	577
query69	454	270	269	269
query70	1174	1102	1097	1097
query71	334	270	268	268
query72	6001	4048	4232	4048
query73	786	366	375	366
query74	10430	9199	9270	9199
query75	3339	2650	2690	2650
query76	2765	1060	1111	1060
query77	396	304	275	275
query78	10511	9595	9586	9586
query79	1872	602	619	602
query80	1209	428	422	422
query81	539	223	224	223
query82	922	86	92	86
query83	236	151	143	143
query84	242	80	85	80
query85	1470	308	314	308
query86	467	268	304	268
query87	4430	4215	4257	4215
query88	4133	2440	2397	2397
query89	424	288	288	288
query90	2110	195	194	194
query91	149	109	112	109
query92	69	49	50	49
query93	2600	554	557	554
query94	1007	293	278	278
query95	363	268	263	263
query96	639	284	285	284
query97	3267	3143	3113	3113
query98	209	201	200	200
query99	1520	1324	1326	1324
Total cold run time: 306308 ms
Total hot run time: 196394 ms

doris-robot · 2025-07-08T08:43:45Z

ClickBench: Total hot run time: 29.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 809b77dd4bb0c000ef542f5c63dab24c0c94ea10, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.12	0.10
query5	0.53	0.51	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.56	0.50	0.52
query10	0.55	0.56	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.60
query14	0.77	0.80	0.82
query15	0.84	0.84	0.83
query16	0.38	0.40	0.38
query17	1.05	1.07	1.02
query18	0.23	0.21	0.23
query19	2.01	1.78	1.87
query20	0.02	0.01	0.01
query21	15.40	0.57	0.58
query22	2.33	2.47	1.43
query23	17.02	0.86	0.93
query24	3.21	1.75	0.38
query25	0.14	0.09	0.06
query26	0.52	0.14	0.14
query27	0.05	0.05	0.05
query28	10.45	0.54	0.48
query29	12.60	3.24	3.24
query30	0.24	0.06	0.06
query31	2.85	0.40	0.39
query32	3.23	0.48	0.46
query33	2.96	2.98	3.04
query34	17.15	4.45	4.42
query35	4.50	4.47	4.47
query36	0.67	0.48	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.77 s
Total hot run time: 29.15 s

doris-robot · 2025-07-08T08:44:50Z

BE UT Coverage Report

Increment line coverage 88.46% (23/26) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	45.23% (12466/27562)
Line Coverage	36.19% (110479/305304)
Region Coverage	35.33% (57300/162168)
Branch Coverage	32.42% (31073/95842)

liaoxin01 · 2025-07-08T13:05:39Z

run buildall

liaoxin01 · 2025-07-08T13:18:06Z

run buildall

… leaks in MemTable (apache#52902) Issue Number: close #xxx Related PR: #xxx Problem Summary: Direct leak of 193504 byte(s) in 6047 object(s) allocated from: (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x2a0015fd) const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/memtable.cpp:243:38 doris::MemTableWriter::write(doris::vectorized::Block const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/memtable_writer.cpp:107:27 doris::DeltaWriterV2::write(doris::vectorized::Block const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/delta_writer_v2.cpp:164:30 doris::vectorized::VTabletWriterV2::_write_memtable(std::shared_ptr<doris::vectorized::Block>, long, doris::vectorized::Rows const&) /root/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:552:24 doris::vectorized::VTabletWriterV2::write(doris::RuntimeState*, doris::vectorized::Block&) /root/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:486:9 doris::vectorized::AsyncResultWriter::process_block(doris::RuntimeState*, doris::RuntimeProfile*) /root/doris/be/src/vec/sink/writer/async_result_writer.cpp:169:23 doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0::operator()() const /root/doris/be/src/vec/sink/writer/async_result_writer.cpp:103:5

liaoxin01 · 2025-07-08T13:27:12Z

run buildall

doris-robot · 2025-07-08T13:47:30Z

TPC-H: Total hot run time: 40145 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ceb462168da76760deea90d5019ffe7c58f7e716, data reload: false

------ Round 1 ----------------------------------
q1	17723	7447	6634	6634
q2	2105	179	162	162
q3	10921	1159	1236	1159
q4	11488	747	738	738
q5	7814	2952	2886	2886
q6	225	143	136	136
q7	1001	643	636	636
q8	9734	2121	2102	2102
q9	8436	6503	6467	6467
q10	7057	2311	2286	2286
q11	458	257	264	257
q12	403	214	213	213
q13	17807	3024	3001	3001
q14	251	208	212	208
q15	507	465	477	465
q16	460	378	372	372
q17	999	516	534	516
q18	7135	6727	6735	6727
q19	1232	1002	953	953
q20	485	199	195	195
q21	4071	3164	3061	3061
q22	1113	983	971	971
Total cold run time: 111425 ms
Total hot run time: 40145 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6678	6652	6604	6604
q2	338	241	245	241
q3	3144	3047	2949	2949
q4	1984	1775	1735	1735
q5	5630	5697	5761	5697
q6	215	130	125	125
q7	2207	1851	1781	1781
q8	3325	3520	3515	3515
q9	8885	8804	8958	8804
q10	3607	3549	3555	3549
q11	594	473	499	473
q12	847	628	655	628
q13	3825	3164	3154	3154
q14	295	283	296	283
q15	519	459	472	459
q16	485	443	442	442
q17	1842	1622	1619	1619
q18	8217	7793	7745	7745
q19	1684	1569	1590	1569
q20	2112	1905	1881	1881
q21	5114	4828	4955	4828
q22	1103	990	975	975
Total cold run time: 62650 ms
Total hot run time: 59056 ms

doris-robot · 2025-07-08T13:59:32Z

TPC-DS: Total hot run time: 189569 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ceb462168da76760deea90d5019ffe7c58f7e716, data reload: false

query1	991	371	359	359
query2	6512	1968	1948	1948
query3	6717	227	217	217
query4	33810	23453	23210	23210
query5	4393	448	453	448
query6	276	181	179	179
query7	4625	314	323	314
query8	301	235	230	230
query9	9581	2636	2638	2636
query10	499	263	259	259
query11	18241	15223	15180	15180
query12	155	107	107	107
query13	1661	434	428	428
query14	9291	6618	6530	6530
query15	214	176	183	176
query16	7937	454	481	454
query17	1632	578	567	567
query18	2045	307	320	307
query19	218	157	156	156
query20	118	110	108	108
query21	208	107	106	106
query22	4576	4132	4381	4132
query23	34451	33492	33333	33333
query24	12580	2847	2832	2832
query25	737	420	408	408
query26	1899	176	175	175
query27	2978	345	345	345
query28	7895	2151	2152	2151
query29	1144	442	458	442
query30	330	159	161	159
query31	1029	781	815	781
query32	105	61	63	61
query33	800	312	314	312
query34	925	502	518	502
query35	922	734	713	713
query36	1132	935	932	932
query37	293	75	77	75
query38	3995	3778	3832	3778
query39	1486	1418	1439	1418
query40	295	103	105	103
query41	54	52	52	52
query42	125	107	101	101
query43	538	492	473	473
query44	1212	810	810	810
query45	185	173	175	173
query46	1143	723	746	723
query47	1943	1861	1838	1838
query48	426	346	368	346
query49	1294	389	380	380
query50	806	429	404	404
query51	7236	7218	7163	7163
query52	102	91	89	89
query53	257	182	183	182
query54	1123	472	451	451
query55	79	79	77	77
query56	280	249	257	249
query57	1295	1177	1188	1177
query58	248	209	222	209
query59	3121	2860	2900	2860
query60	296	257	262	257
query61	115	115	117	115
query62	861	692	661	661
query63	214	187	187	187
query64	5267	640	640	640
query65	3279	3204	3182	3182
query66	1416	310	297	297
query67	16046	15537	15661	15537
query68	4404	573	596	573
query69	430	270	266	266
query70	1147	1079	1063	1063
query71	325	258	255	255
query72	6320	4223	3983	3983
query73	760	349	366	349
query74	10051	8907	8888	8888
query75	3373	2624	2637	2624
query76	2520	1075	1107	1075
query77	466	281	280	280
query78	10390	9686	9588	9588
query79	1322	597	598	597
query80	1025	439	429	429
query81	520	217	218	217
query82	1162	94	88	88
query83	224	150	149	149
query84	241	80	81	80
query85	1275	318	303	303
query86	359	308	283	283
query87	4346	4225	4290	4225
query88	3738	2391	2366	2366
query89	416	300	294	294
query90	1928	190	188	188
query91	143	108	109	108
query92	62	53	52	52
query93	1107	548	558	548
query94	956	298	297	297
query95	355	258	262	258
query96	611	281	281	281
query97	3292	3152	3164	3152
query98	207	201	201	201
query99	1516	1313	1305	1305
Total cold run time: 300949 ms
Total hot run time: 189569 ms

doris-robot · 2025-07-08T14:04:56Z

ClickBench: Total hot run time: 31.01 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ceb462168da76760deea90d5019ffe7c58f7e716, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.62	0.11	0.12
query5	0.53	0.52	0.51
query6	1.13	0.73	0.74
query7	0.03	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.51	0.50
query10	0.56	0.54	0.57
query11	0.15	0.10	0.10
query12	0.13	0.11	0.11
query13	0.60	0.60	0.59
query14	0.78	0.79	0.80
query15	0.85	0.83	0.82
query16	0.37	0.39	0.39
query17	1.02	1.03	1.00
query18	0.23	0.21	0.22
query19	1.88	1.81	1.89
query20	0.01	0.01	0.02
query21	15.40	0.60	0.60
query22	2.51	1.95	2.16
query23	16.98	0.84	0.74
query24	3.20	1.77	2.33
query25	0.38	0.13	0.14
query26	0.39	0.14	0.12
query27	0.05	0.04	0.04
query28	8.60	0.52	0.49
query29	12.55	3.16	3.17
query30	0.24	0.05	0.05
query31	2.85	0.39	0.37
query32	3.25	0.46	0.45
query33	2.98	3.00	3.02
query34	17.20	4.51	4.53
query35	4.52	4.52	4.49
query36	0.65	0.47	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.03
query40	0.16	0.12	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.06 s
Total hot run time: 31.01 s

doris-robot · 2025-07-08T14:15:50Z

BE UT Coverage Report

Increment line coverage 67.74% (21/31) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	45.18% (12477/27618)
Line Coverage	36.05% (110616/306833)
Region Coverage	35.23% (57366/162852)
Branch Coverage	32.29% (31094/96294)

liaoxin01 requested a review from morrySnow as a code owner July 8, 2025 07:18

liaoxin01 force-pushed the pick_52902_to_origin_branch-3.1 branch from 0acb870 to 809b77d Compare July 8, 2025 07:20

morrySnow changed the title ~~branch-3.1: [fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable (#52902)~~ branch-3.1: [fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable #52902 Jul 8, 2025

liaoxin01 force-pushed the pick_52902_to_origin_branch-3.1 branch from 809b77d to d98d6fb Compare July 8, 2025 13:05

liaoxin01 force-pushed the pick_52902_to_origin_branch-3.1 branch from d98d6fb to 73ece7a Compare July 8, 2025 13:14

liaoxin01 force-pushed the pick_52902_to_origin_branch-3.1 branch from 73ece7a to ceb4621 Compare July 8, 2025 13:24

morrySnow approved these changes Jul 9, 2025

View reviewed changes

morrySnow merged commit 1091a84 into apache:branch-3.1 Jul 9, 2025
19 of 21 checks passed

branch-3.1: [fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable #52902 #52928

branch-3.1: [fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable #52902 #52928

Uh oh!

Conversation

liaoxin01 commented Jul 8, 2025 • edited by morrySnow Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liaoxin01 commented Jul 8, 2025

Uh oh!

hello-stephen commented Jul 8, 2025

Uh oh!

liaoxin01 commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

BE UT Coverage Report

Uh oh!

liaoxin01 commented Jul 8, 2025

Uh oh!

liaoxin01 commented Jul 8, 2025

Uh oh!

liaoxin01 commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

Uh oh!

doris-robot commented Jul 8, 2025

BE UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liaoxin01 commented Jul 8, 2025 •

edited by morrySnow

Loading