[fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable #52902

liaoxin01 · 2025-07-07T12:25:28Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Direct leak of 193504 byte(s) in 6047 object(s) allocated from:
#0 0x559f1d2a45fd in operator new(unsigned long) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x2a0015fd)
#1 0x559f1f54c41d in doris::MemTable::insert(doris::vectorized::Block const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/memtable.cpp:243:38
#2 0x559f1f597c08 in doris::MemTableWriter::write(doris::vectorized::Block const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/memtable_writer.cpp:107:27
#3 0x559f467a48c0 in doris::DeltaWriterV2::write(doris::vectorized::Block const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/delta_writer_v2.cpp:164:30
#4 0x559f4711e2f5 in doris::vectorized::VTabletWriterV2::_write_memtable(std::shared_ptrdoris::vectorized::Block, long, doris::vectorized::Rows const&) /root/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:552:24
#5 0x559f4711b4bf in doris::vectorized::VTabletWriterV2::write(doris::RuntimeState*, doris::vectorized::Block&) /root/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:486:9
#6 0x559f4702bb4b in doris::vectorized::AsyncResultWriter::process_block(doris::RuntimeState*, doris::RuntimeProfile*) /root/doris/be/src/vec/sink/writer/async_result_writer.cpp:169:23
#7 0x559f4702e7e8 in doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0::operator()() const /root/doris/be/src/vec/sink/writer/async_result_writer.cpp:103:5

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-07-07T12:25:33Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

liaoxin01 · 2025-07-07T12:26:03Z

run buildall

… leaks in MemTable

liaoxin01 · 2025-07-07T12:27:36Z

run buildall

doris-robot · 2025-07-07T12:46:49Z

TPC-H: Total hot run time: 33250 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 318d2eae92c3a93c20bb123a3f0f26b74aac5591, data reload: false

------ Round 1 ----------------------------------
q1	17576	5202	5009	5009
q2	1941	287	188	188
q3	10302	1305	701	701
q4	10229	1018	531	531
q5	7529	2478	2323	2323
q6	180	156	126	126
q7	892	742	593	593
q8	9321	1331	1096	1096
q9	7022	5155	5132	5132
q10	6901	2398	1968	1968
q11	477	288	267	267
q12	346	350	221	221
q13	17798	3775	3132	3132
q14	226	235	225	225
q15	553	479	489	479
q16	431	422	382	382
q17	599	878	349	349
q18	7465	7330	7133	7133
q19	1083	974	541	541
q20	334	366	223	223
q21	3813	3110	2331	2331
q22	369	327	300	300
Total cold run time: 105387 ms
Total hot run time: 33250 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5046	5102	5034	5034
q2	241	345	215	215
q3	2168	2711	2264	2264
q4	1409	1776	1340	1340
q5	4219	4302	4546	4302
q6	219	169	132	132
q7	2064	1932	1828	1828
q8	2679	2723	2705	2705
q9	7415	7417	7300	7300
q10	3073	3295	2864	2864
q11	585	521	484	484
q12	704	758	625	625
q13	3678	4128	3448	3448
q14	300	293	267	267
q15	519	475	472	472
q16	442	499	467	467
q17	1187	1601	1369	1369
q18	7953	7765	7572	7572
q19	850	836	970	836
q20	2048	2068	2001	2001
q21	4971	4625	4490	4490
q22	662	607	540	540
Total cold run time: 52432 ms
Total hot run time: 50555 ms

doris-robot · 2025-07-07T12:58:30Z

TPC-DS: Total hot run time: 186691 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 318d2eae92c3a93c20bb123a3f0f26b74aac5591, data reload: false

query1	1000	398	381	381
query2	6529	1658	1669	1658
query3	6747	214	217	214
query4	25955	23267	22973	22973
query5	4331	590	441	441
query6	294	226	211	211
query7	4635	508	298	298
query8	279	231	228	228
query9	8608	2628	2643	2628
query10	443	332	263	263
query11	15734	15052	14895	14895
query12	162	111	102	102
query13	1674	524	388	388
query14	9378	5815	5790	5790
query15	195	187	178	178
query16	7633	440	256	256
query17	1325	724	603	603
query18	2021	406	299	299
query19	221	190	162	162
query20	124	120	118	118
query21	217	123	110	110
query22	4128	4195	4645	4195
query23	35031	33995	34091	33995
query24	8521	2377	2416	2377
query25	550	468	391	391
query26	1227	263	146	146
query27	2771	509	335	335
query28	4276	2130	2104	2104
query29	729	545	422	422
query30	280	228	199	199
query31	929	860	793	793
query32	100	62	64	62
query33	548	332	293	293
query34	801	853	511	511
query35	597	654	581	581
query36	932	987	873	873
query37	103	99	77	77
query38	4146	4187	4109	4109
query39	1462	1434	1391	1391
query40	208	120	108	108
query41	56	55	52	52
query42	131	116	114	114
query43	510	545	484	484
query44	1319	857	825	825
query45	178	168	166	166
query46	859	1028	644	644
query47	1750	1801	1721	1721
query48	389	416	311	311
query49	752	489	407	407
query50	651	692	420	420
query51	4275	4193	4139	4139
query52	104	107	99	99
query53	226	265	179	179
query54	584	581	514	514
query55	88	88	85	85
query56	330	293	292	292
query57	1185	1192	1123	1123
query58	265	263	254	254
query59	2548	2590	2482	2482
query60	338	354	326	326
query61	156	143	145	143
query62	807	718	648	648
query63	231	196	193	193
query64	4562	1233	828	828
query65	4274	4166	4199	4166
query66	1070	408	308	308
query67	15749	15728	15516	15516
query68	7921	916	537	537
query69	515	325	281	281
query70	1221	1122	1218	1122
query71	467	339	306	306
query72	5821	4686	4777	4686
query73	681	599	350	350
query74	9282	9228	8781	8781
query75	3861	3174	2757	2757
query76	3686	1150	726	726
query77	811	404	298	298
query78	10916	10982	10243	10243
query79	2047	886	581	581
query80	615	532	434	434
query81	489	257	221	221
query82	444	127	97	97
query83	254	312	238	238
query84	249	104	83	83
query85	786	365	317	317
query86	386	300	278	278
query87	4488	4511	4347	4347
query88	3528	2294	2306	2294
query89	385	323	279	279
query90	1920	214	202	202
query91	144	147	111	111
query92	82	71	55	55
query93	1720	975	601	601
query94	672	318	204	204
query95	385	301	289	289
query96	508	584	279	279
query97	2765	2902	2638	2638
query98	230	216	217	216
query99	1477	1400	1243	1243
Total cold run time: 276987 ms
Total hot run time: 186691 ms

doris-robot · 2025-07-07T13:03:47Z

ClickBench: Total hot run time: 29.81 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 318d2eae92c3a93c20bb123a3f0f26b74aac5591, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.05
query3	0.24	0.07	0.08
query4	1.61	0.12	0.11
query5	0.43	0.42	0.43
query6	1.17	0.66	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.61	0.51	0.52
query10	0.57	0.57	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.62	0.62
query14	0.80	0.81	0.82
query15	0.93	0.89	0.88
query16	0.38	0.39	0.40
query17	1.11	1.07	1.10
query18	0.23	0.22	0.21
query19	1.96	1.90	1.85
query20	0.02	0.01	0.01
query21	15.41	0.93	0.56
query22	0.75	1.30	0.74
query23	14.74	1.38	0.61
query24	7.37	1.11	0.77
query25	0.43	0.16	0.18
query26	0.65	0.18	0.14
query27	0.07	0.06	0.05
query28	8.99	0.90	0.45
query29	12.54	3.98	3.30
query30	0.26	0.09	0.06
query31	2.83	0.64	0.39
query32	3.23	0.56	0.47
query33	3.10	3.12	3.11
query34	16.06	5.45	4.79
query35	4.90	4.87	4.89
query36	0.69	0.52	0.49
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.7 s
Total hot run time: 29.81 s

doris-robot · 2025-07-07T14:28:34Z

BE UT Coverage Report

Increment line coverage 65.52% (19/29) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.32% (15497/27038)
Line Coverage	46.29% (140829/304240)
Region Coverage	45.55% (71212/156355)
Branch Coverage	40.27% (37530/93200)

dataroaring

LGTM

github-actions · 2025-07-08T01:48:30Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-07-08T01:48:32Z

PR approved by anyone and no changes requested.

… leaks in MemTable (apache#52902) Issue Number: close #xxx Related PR: #xxx Problem Summary: Direct leak of 193504 byte(s) in 6047 object(s) allocated from: (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x2a0015fd) const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/memtable.cpp:243:38 doris::MemTableWriter::write(doris::vectorized::Block const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/memtable_writer.cpp:107:27 doris::DeltaWriterV2::write(doris::vectorized::Block const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/delta_writer_v2.cpp:164:30 doris::vectorized::VTabletWriterV2::_write_memtable(std::shared_ptr<doris::vectorized::Block>, long, doris::vectorized::Rows const&) /root/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:552:24 doris::vectorized::VTabletWriterV2::write(doris::RuntimeState*, doris::vectorized::Block&) /root/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:486:9 doris::vectorized::AsyncResultWriter::process_block(doris::RuntimeState*, doris::RuntimeProfile*) /root/doris/be/src/vec/sink/writer/async_result_writer.cpp:169:23 doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0::operator()() const /root/doris/be/src/vec/sink/writer/async_result_writer.cpp:103:5

… leaks in MemTable (apache#52902)

… leaks in MemTable (apache#52902) Issue Number: close #xxx Related PR: #xxx Problem Summary: Direct leak of 193504 byte(s) in 6047 object(s) allocated from: (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be+0x2a0015fd) const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/memtable.cpp:243:38 doris::MemTableWriter::write(doris::vectorized::Block const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/memtable_writer.cpp:107:27 doris::DeltaWriterV2::write(doris::vectorized::Block const*, std::vector<unsigned int, doris::CustomStdAllocator<unsigned int, doris::Allocator<false, false, false, doris::DefaultMemoryAllocator> > > const&) /root/doris/be/src/olap/delta_writer_v2.cpp:164:30 doris::vectorized::VTabletWriterV2::_write_memtable(std::shared_ptr<doris::vectorized::Block>, long, doris::vectorized::Rows const&) /root/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:552:24 doris::vectorized::VTabletWriterV2::write(doris::RuntimeState*, doris::vectorized::Block&) /root/doris/be/src/vec/sink/writer/vtablet_writer_v2.cpp:486:9 doris::vectorized::AsyncResultWriter::process_block(doris::RuntimeState*, doris::RuntimeProfile*) /root/doris/be/src/vec/sink/writer/async_result_writer.cpp:169:23 doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0::operator()() const /root/doris/be/src/vec/sink/writer/async_result_writer.cpp:103:5

…ntial memory leaks in MemTable (#52902) (#52965) pick from: #52902

…ntial memory leaks in MemTable #52902 (#52928) pick from：#52902

[fix](load) Convert RowInBlock* to shared_ptr to fix potential memory…

318d2ea

… leaks in MemTable

liaoxin01 force-pushed the DORIS-21167 branch from ff41d93 to 318d2ea Compare July 7, 2025 12:27

dataroaring added the dev/3.0.x label Jul 7, 2025

hello-stephen added the dev/3.1.x label Jul 7, 2025

dataroaring approved these changes Jul 8, 2025

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 8, 2025

hello-stephen approved these changes Jul 8, 2025

View reviewed changes

hello-stephen merged commit da9c1cf into apache:master Jul 8, 2025
25 of 29 checks passed

github-actions bot added dev/3.0.x-conflict dev/3.1.x-conflict labels Jul 8, 2025

liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Jul 8, 2025

[fix](load) Convert RowInBlock* to shared_ptr to fix potential memory…

0acb870

… leaks in MemTable (apache#52902)

liaoxin01 mentioned this pull request Jul 8, 2025

branch-3.1: [fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable #52902 #52928

Merged

liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Jul 8, 2025

[fix](load) Convert RowInBlock* to shared_ptr to fix potential memory…

809b77d

… leaks in MemTable (apache#52902)

liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Jul 8, 2025

[fix](load) Convert RowInBlock* to shared_ptr to fix potential memory…

cd53a29

… leaks in MemTable (apache#52902)

liaoxin01 mentioned this pull request Jul 8, 2025

branch-3.0: [fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable (#52902) #52965

Merged

16 tasks

dataroaring pushed a commit that referenced this pull request Jul 9, 2025

branch-3.0: [fix](load) Convert RowInBlock* to shared_ptr to fix pote…

d51643f

…ntial memory leaks in MemTable (#52902) (#52965) pick from: #52902

dataroaring added dev/3.0.7-merged and removed dev/3.0.x dev/3.0.x-conflict labels Jul 9, 2025

morrySnow pushed a commit that referenced this pull request Jul 9, 2025

branch-3.1: [fix](load) Convert RowInBlock* to shared_ptr to fix pote…

1091a84

…ntial memory leaks in MemTable #52902 (#52928) pick from：#52902

morrySnow added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Jul 9, 2025

gavinchou mentioned this pull request Aug 15, 2025

Release Note 3.0.7 #54868

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable #52902

[fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable #52902

Uh oh!

liaoxin01 commented Jul 7, 2025 •

edited

Loading

Uh oh!

Thearas commented Jul 7, 2025

Uh oh!

liaoxin01 commented Jul 7, 2025

Uh oh!

liaoxin01 commented Jul 7, 2025

Uh oh!

doris-robot commented Jul 7, 2025

Uh oh!

doris-robot commented Jul 7, 2025

Uh oh!

doris-robot commented Jul 7, 2025

Uh oh!

doris-robot commented Jul 7, 2025

Uh oh!

dataroaring left a comment

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable #52902

[fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable #52902

Uh oh!

Conversation

liaoxin01 commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jul 7, 2025

Uh oh!

liaoxin01 commented Jul 7, 2025

Uh oh!

liaoxin01 commented Jul 7, 2025

Uh oh!

doris-robot commented Jul 7, 2025

Uh oh!

doris-robot commented Jul 7, 2025

Uh oh!

doris-robot commented Jul 7, 2025

Uh oh!

doris-robot commented Jul 7, 2025

BE UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

github-actions bot commented Jul 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

liaoxin01 commented Jul 7, 2025 •

edited

Loading