Skip to content

[fix](load) add lock in active_memtable_mem_consumption #27101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Nov 17, 2023
Merged

[fix](load) add lock in active_memtable_mem_consumption #27101

dataroaring merged 1 commit into from
Nov 17, 2023

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Nov 16, 2023

Proposed changes

Fix NPE in active_memtable_mem_consumption

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@kaijchen
Copy link
Member Author

kaijchen commented Nov 16, 2023

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Nov 16, 2023

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Nov 16, 2023

TeamCity be ut coverage result:
Function Coverage: 36.75% (8410/22883)
Line Coverage: 29.29% (68432/233645)
Region Coverage: 27.87% (35359/126869)
Branch Coverage: 24.63% (18070/73356)
Coverage Report: http://coverage.selectdb-in.cc/coverage/abfb80162511b5ce1f2823c109ec5897a3db88c5_abfb80162511b5ce1f2823c109ec5897a3db88c5/report/index.html

dataroaring
dataroaring approved these changes Nov 16, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 16, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Nov 16, 2023

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 16, 2023

PR approved by anyone and no changes requested.

sollhui
sollhui approved these changes Nov 16, 2023
View reviewed changes
Copy link
Contributor

@sollhui sollhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

doris-robot commented Nov 16, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.47 seconds
stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17098754575 Bytes

@doris-robot
Copy link

doris-robot commented Nov 16, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit abfb80162511b5ce1f2823c109ec5897a3db88c5, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4953	4625	4607	4607
q2	356	139	134	134
q3	2016	1886	1925	1886
q4	1395	1247	1228	1228
q5	3952	3959	4071	3959
q6	254	132	122	122
q7	1382	866	891	866
q8	2765	2787	2780	2780
q9	9758	9842	9612	9612
q10	3468	3538	3526	3526
q11	376	250	245	245
q12	431	292	295	292
q13	4573	3811	3851	3811
q14	326	273	280	273
q15	589	526	518	518
q16	665	575	588	575
q17	1141	972	935	935
q18	7852	7309	7495	7309
q19	1687	1695	1693	1693
q20	518	336	317	317
q21	4398	4002	3991	3991
q22	474	364	374	364
Total cold run time: 53329 ms
Total hot run time: 49043 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4642	4593	4554	4554
q2	327	237	278	237
q3	4010	4016	4001	4001
q4	2713	2692	2695	2692
q5	9640	9588	9697	9588
q6	246	117	122	117
q7	2587	2314	2256	2256
q8	4455	4466	4513	4466
q9	13293	13134	13091	13091
q10	4097	4168	4160	4160
q11	789	661	665	661
q12	972	808	820	808
q13	4293	3543	3563	3543
q14	369	347	343	343
q15	588	517	524	517
q16	731	663	682	663
q17	3924	3847	3949	3847
q18	9566	9163	9070	9070
q19	1844	1793	1773	1773
q20	2425	2069	2039	2039
q21	8770	8570	8515	8515
q22	923	853	759	759
Total cold run time: 81204 ms
Total hot run time: 77700 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 16, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Nov 16, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit abfb80162511b5ce1f2823c109ec5897a3db88c5, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4943	4656	4656	4656
q2	357	148	159	148
q3	2023	1902	1835	1835
q4	1374	1251	1218	1218
q5	3976	3947	4021	3947
q6	251	128	131	128
q7	1411	870	883	870
q8	2765	2803	2769	2769
q9	9736	9709	9542	9542
q10	3476	3514	3536	3514
q11	365	249	249	249
q12	442	293	297	293
q13	4538	3816	3837	3816
q14	321	284	281	281
q15	578	528	525	525
q16	667	581	580	580
q17	1138	983	939	939
q18	7877	7509	7544	7509
q19	1672	1680	1702	1680
q20	536	322	301	301
q21	4410	3979	3977	3977
q22	475	370	360	360
Total cold run time: 53331 ms
Total hot run time: 49137 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4608	4583	4601	4583
q2	343	217	263	217
q3	4016	3998	4000	3998
q4	2713	2695	2714	2695
q5	9584	9685	9684	9684
q6	244	122	121	121
q7	2585	2294	2252	2252
q8	4442	4441	4463	4441
q9	13231	13191	13174	13174
q10	4073	4176	4167	4167
q11	752	658	647	647
q12	990	817	809	809
q13	4269	3572	3577	3572
q14	380	343	341	341
q15	569	528	527	527
q16	741	682	672	672
q17	3884	3924	3852	3852
q18	9530	9078	9139	9078
q19	1815	1803	1770	1770
q20	2395	2029	2058	2029
q21	8755	8487	8369	8369
q22	881	773	830	773
Total cold run time: 80800 ms
Total hot run time: 77771 ms

@doris-robot
Copy link

doris-robot commented Nov 16, 2023

TeamCity be ut coverage result:
Function Coverage: 36.75% (8410/22883)
Line Coverage: 29.28% (68404/233645)
Region Coverage: 27.87% (35357/126871)
Branch Coverage: 24.63% (18070/73358)
Coverage Report: http://coverage.selectdb-in.cc/coverage/abfb80162511b5ce1f2823c109ec5897a3db88c5_abfb80162511b5ce1f2823c109ec5897a3db88c5/report/index.html

@doris-robot
Copy link

doris-robot commented Nov 16, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.58 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17099330926 Bytes

@dataroaring dataroaring merged commit 4d2fb1f into apache:master Nov 17, 2023
superdiaodiao pushed a commit to superdiaodiao/doris that referenced this pull request Nov 21, 2023
@kaijchen @superdiaodiao
[fix](load) add lock in active_memtable_mem_consumption (apache#27101)
bb319c5
@xiaokang xiaokang added the p0_b label Nov 22, 2023
@kaijchen kaijchen mentioned this pull request Nov 23, 2023
Merged
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
@kaijchen @seawinde
[fix](load) add lock in active_memtable_mem_consumption (apache#27101)
003e105
@xiaokang xiaokang mentioned this pull request Dec 4, 2023
Closed
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@kaijchen
[fix](load) add lock in active_memtable_mem_consumption (apache#27101)
c9261c8
@kaijchen kaijchen deleted the lock-active_memtable_mem_consumption branch July 8, 2025 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@sollhui sollhui sollhui approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.3-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@kaijchen @doris-robot @hello-stephen @sollhui @dataroaring @xiaokang