Skip to content

[Fix](point query) fix memleak by increasing scanReplicaIds when using prepared statement #28184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
eldenmoon merged 1 commit into from
Dec 8, 2023
Merged

[Fix](point query) fix memleak by increasing scanReplicaIds when using prepared statement #28184

eldenmoon merged 1 commit into from
Dec 8, 2023

Conversation

@eldenmoon
Copy link
Member

@eldenmoon eldenmoon commented Dec 8, 2023

…ing prepared statement

OlapScanNode should release memory for scanReplicaIds

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@eldenmoon
[Fix](point query) fix memleak by increasing scanReplicaIds when us…
061cdad 
…ing prepared statement

OlapScanNode should release memory for `scanReplicaIds`
@eldenmoon
Copy link
Member Author

eldenmoon commented Dec 8, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 8, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.46 seconds
stream load tsv: 583 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17212485628 Bytes

@doris-robot
Copy link

doris-robot commented Dec 8, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 061cdadeeefa4775913591222461aec03f1be5f3, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4701	4474	4455	4455
q2	363	152	154	152
q3	1447	1243	1203	1203
q4	1092	912	903	903
q5	3094	3069	3107	3069
q6	251	132	127	127
q7	968	495	475	475
q8	2205	2237	2157	2157
q9	6632	6610	6682	6610
q10	3452	3263	3263	3263
q11	324	204	210	204
q12	341	205	203	203
q13	4572	3820	3770	3770
q14	239	209	212	209
q15	554	531	522	522
q16	435	386	388	386
q17	990	569	566	566
q18	7506	7212	7033	7033
q19	1525	1346	1424	1346
q20	583	314	309	309
q21	3039	2606	2594	2594
q22	346	292	292	292
Total cold run time: 44659 ms
Total hot run time: 39848 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4359	4365	4375	4365
q2	261	161	174	161
q3	3502	3495	3489	3489
q4	2356	2343	2339	2339
q5	5710	5712	5706	5706
q6	238	122	123	122
q7	2370	1874	1912	1874
q8	3489	3490	3478	3478
q9	8911	8912	8869	8869
q10	3876	3942	3990	3942
q11	493	395	376	376
q12	756	598	593	593
q13	4274	3593	3564	3564
q14	286	254	263	254
q15	565	524	516	516
q16	508	441	462	441
q17	1853	1844	1831	1831
q18	8593	8219	8224	8219
q19	1702	1707	1709	1707
q20	2237	1930	1922	1922
q21	6466	6193	6131	6131
q22	491	434	421	421
Total cold run time: 63296 ms
Total hot run time: 60320 ms

@xiaokang xiaokang changed the title [Fix](point query) fix memleak by increasing scanReplicaIds when us… [Fix](point query) fix memleak by increasing scanReplicaIds when using prepared statement Dec 8, 2023
xiaokang
xiaokang reviewed Dec 8, 2023
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

xiaokang
xiaokang reviewed Dec 8, 2023
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

xiaokang
xiaokang approved these changes Dec 8, 2023
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 8, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Dec 8, 2023

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 8, 2023

PR approved by anyone and no changes requested.

qidaye
qidaye approved these changes Dec 8, 2023
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit 0640411 into apache:master Dec 8, 2023
@eldenmoon eldenmoon deleted the fix-mem-prep branch December 8, 2023 14:08
@wm1581066 wm1581066 added the usercase Important user case type label label Dec 11, 2023
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@eldenmoon
[Fix](point query) fix memleak by increasing scanReplicaIds when us…
f012963 
…ing prepared statement (apache#28184)

OlapScanNode should release memory for `scanReplicaIds`
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Dec 19, 2023
@eldenmoon
[Fix](point query) fix memleak by increasing scanReplicaIds when us…
4ecb5c8 
…ing prepared statement (apache#28184)

OlapScanNode should release memory for `scanReplicaIds`
@eldenmoon eldenmoon mentioned this pull request Dec 19, 2023
Merged
xiaokang pushed a commit that referenced this pull request Dec 19, 2023
@eldenmoon
[Pick](branch-2.0) pick some PRs about point query optimization (#28611)
c03c92a 
* [Fix](point query) fix memleak by increasing `scanReplicaIds` when using prepared statement (#28184)
* [Performance](point query)Opimize partition prune for point query (#28150)
* [Improve](sort) avoid too may tmp vectors for get_columns (#27734)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaokang xiaokang xiaokang approved these changes

@qidaye qidaye qidaye approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.4-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@eldenmoon @doris-robot @xiaokang @qidaye @wm1581066