[Improvement](shuffle) Reduce memory consumption in data stream sender #41676

Gabriel39 · 2024-10-10T12:57:19Z

Proposed changes

Before:

After:

doris-robot · 2024-10-10T12:57:25Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Gabriel39 · 2024-10-10T12:57:33Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-10-10T13:02:01Z

be/src/vec/runtime/vdata_stream_recvr.cpp

 }

-void VDataStreamRecvr::add_block(Block* block, int sender_id, bool use_move) {
+void VDataStreamRecvr::add_block(Block* block, int sender_id) {


warning: method 'add_block' can be made const [readability-make-member-function-const]

Suggested change

void VDataStreamRecvr::add_block(Block* block, int sender_id) {

void VDataStreamRecvr::add_block(Block* block, int sender_id) const {

be/src/vec/runtime/vdata_stream_recvr.h:88:

- void add_block(Block* block, int sender_id); + void add_block(Block* block, int sender_id) const;

doris-robot · 2024-10-10T13:41:09Z

TeamCity be ut coverage result:
Function Coverage: 37.41% (9661/25825)
Line Coverage: 28.69% (80175/279456)
Region Coverage: 28.12% (41447/147402)
Branch Coverage: 24.72% (21107/85390)
Coverage Report: http://coverage.selectdb-in.cc/coverage/656146a35845ffad277388df9735524395cfa526_656146a35845ffad277388df9735524395cfa526/report/index.html

HappenLee

LGTM

github-actions · 2024-10-11T02:10:14Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-10-11T02:10:16Z

PR approved by anyone and no changes requested.

github-actions · 2024-10-11T03:26:08Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-10-11T03:43:32Z

clang-tidy review says "All clean, LGTM! 👍"

Gabriel39 · 2024-10-11T03:48:55Z

run buildall

doris-robot · 2024-10-11T04:12:42Z

TeamCity be ut coverage result:
Function Coverage: 37.40% (9660/25827)
Line Coverage: 28.67% (80136/279497)
Region Coverage: 28.11% (41445/147457)
Branch Coverage: 24.71% (21105/85420)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bd85447159141deb697f2958cef258a369b49b0e_bd85447159141deb697f2958cef258a369b49b0e/report/index.html

github-actions · 2024-10-11T06:03:27Z

PR approved by at least one committer and no changes requested.

Follow-up for #41676

apache#41676)

) Follow-up for apache#41676

…1808)

apache#41676)

) Follow-up for apache#41676

pick #41676 #41740 #41857

) Follow-up for apache#41676

github-actions bot reviewed Oct 10, 2024

View reviewed changes

HappenLee previously approved these changes Oct 11, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 11, 2024

github-actions bot added the reviewed label Oct 11, 2024

update

0827bd0

Gabriel39 dismissed HappenLee’s stale review via 0827bd0 October 11, 2024 03:21

Gabriel39 force-pushed the opt_1010 branch from 656146a to 0827bd0 Compare October 11, 2024 03:21

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 11, 2024

update

bd85447

Mryange approved these changes Oct 11, 2024

View reviewed changes

BiteTheDDDDt approved these changes Oct 11, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 11, 2024

Gabriel39 merged commit af95818 into apache:master Oct 11, 2024

Gabriel39 added dev/2.1.x dev/3.0.x labels Oct 11, 2024

Gabriel39 mentioned this pull request Oct 12, 2024

[followup](memory) Block should be moved by hash shuffling #41740

Merged

Gabriel39 added a commit that referenced this pull request Oct 12, 2024

[followup](memory) Block should be moved by hash shuffling (#41740)

cce5425

Follow-up for #41676

Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 14, 2024

[Improvement](shuffle) Reduce memory consumption in data stream sender (

227cc8c

apache#41676)

Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 14, 2024

[followup](memory) Block should be moved by hash shuffling (apache#41740

4be7ee5

) Follow-up for apache#41676

Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 14, 2024

[followup](memory) Block should be moved by hash shuffling (apache#41740

58c9653

) Follow-up for apache#41676

dataroaring pushed a commit that referenced this pull request Oct 15, 2024

[pick](branch-3.0) pick #39982 #40576 #41555 #41592 #41676 #41740 (#4…

54b4fa3

…1808)

Gabriel39 added dev/3.0.3-merged and removed dev/3.0.x labels Oct 15, 2024

Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 15, 2024

[Improvement](shuffle) Reduce memory consumption in data stream sender (

43bb8b1

apache#41676)

Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 15, 2024

[followup](memory) Block should be moved by hash shuffling (apache#41740

e90a846

) Follow-up for apache#41676

Gabriel39 mentioned this pull request Oct 15, 2024

[pick](branch-2.1) pick #41676 #41740 #41857 #41904

Merged

Gabriel39 added a commit that referenced this pull request Oct 15, 2024

[pick](branch-2.1) pick #41676 #41740 #41857 (#41904)

b185dfc

pick #41676 #41740 #41857

Gabriel39 added dev/2.1.7-merged and removed dev/2.1.x labels Oct 15, 2024

qzsee pushed a commit to qzsee/incubator-doris that referenced this pull request Oct 16, 2024

[followup](memory) Block should be moved by hash shuffling (apache#41740

272a8ae

) Follow-up for apache#41676

	void VDataStreamRecvr::add_block(Block* block, int sender_id) {
	void VDataStreamRecvr::add_block(Block* block, int sender_id) const {

[Improvement](shuffle) Reduce memory consumption in data stream sender #41676

[Improvement](shuffle) Reduce memory consumption in data stream sender #41676

Uh oh!

Conversation

Gabriel39 commented Oct 10, 2024

Proposed changes

Uh oh!

doris-robot commented Oct 10, 2024

Uh oh!

Gabriel39 commented Oct 10, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Oct 10, 2024

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Oct 10, 2024

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 11, 2024

Uh oh!

github-actions bot commented Oct 11, 2024

Uh oh!

github-actions bot commented Oct 11, 2024

Uh oh!

github-actions bot commented Oct 11, 2024

Uh oh!

Gabriel39 commented Oct 11, 2024

Uh oh!

doris-robot commented Oct 11, 2024

Uh oh!

github-actions bot commented Oct 11, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants