Skip to content

Conversation

@Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Sep 25, 2024

…d in this pipeline

Proposed changes

Consider 3 pipelines in this fragment (... -> join -> shuffle) :
pipeline 0 : ... -> local exchange sink
pipeline 1 : ... -> join build (INNER JOIN)
pipeline 2 : local exchange source -> join probe (INNER JOIN) -> data stream sender

Assume the JoinBuild returned 0 rows, join probe can finish directly once join build finished and do not need to wait for the local exchange sink finished. In this case, if pipeline 0 is blocked by a dependency for a long time, pipeline 2 should notify pipeline 0 to finish.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.28% (9623/25811)
Line Coverage: 28.69% (79620/277520)
Region Coverage: 28.11% (41167/146443)
Branch Coverage: 24.75% (20973/84744)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6fc6b2f092702c28004cb24a7a0ca67e90c54eb9_6fc6b2f092702c28004cb24a7a0ca67e90c54eb9/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.29% (9625/25814)
Line Coverage: 28.70% (79650/277531)
Region Coverage: 28.13% (41194/146451)
Branch Coverage: 24.76% (20986/84748)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6e0870da08189aa867cde432595a6b3291954b04_6e0870da08189aa867cde432595a6b3291954b04/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

3 similar comments
@Gabriel39
Copy link
Contributor Author

run buildall

@Gabriel39
Copy link
Contributor Author

run buildall

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.28% (9625/25816)
Line Coverage: 28.71% (79688/277586)
Region Coverage: 28.13% (41199/146473)
Branch Coverage: 24.76% (20985/84756)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4b13401ef365528ac4a7bcbbce7f6beb42596f41_4b13401ef365528ac4a7bcbbce7f6beb42596f41/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.28% (9623/25816)
Line Coverage: 28.70% (79675/277609)
Region Coverage: 28.11% (41178/146488)
Branch Coverage: 24.75% (20977/84766)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0c6d47cff0cbf3b5382128720777cac466ac36e5_0c6d47cff0cbf3b5382128720777cac466ac36e5/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

1 similar comment
@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.29% (9626/25816)
Line Coverage: 28.70% (79671/277627)
Region Coverage: 28.12% (41197/146496)
Branch Coverage: 24.75% (20983/84772)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c6c8fc5fbee037bc08cf538fbc097122f29e0579_c6c8fc5fbee037bc08cf538fbc097122f29e0579/report/index.html

BiteTheDDDDt added a commit that referenced this pull request Oct 11, 2024
…ly (#41667)

## Proposed changes
send ignored rf when hash join build closed early to avoid runtime
filter sync/merge error
Follow-up : #41292
Gabriel39 added a commit that referenced this pull request Oct 11, 2024
cjj2010 pushed a commit to cjj2010/doris that referenced this pull request Oct 12, 2024
apache#41292)

Consider 3 pipelines in this fragment (... -> join -> shuffle) :
pipeline 0 : `... -> local exchange sink`
pipeline 1 : `... -> join build (INNER JOIN)`
pipeline 2 : `local exchange source -> join probe (INNER JOIN) -> data
stream sender `

Assume the JoinBuild returned 0 rows, join probe can finish directly
once join build finished and do not need to wait for the `local exchange
sink` finished. In this case, if pipeline 0 is blocked by a dependency
for a long time, pipeline 2 should notify pipeline 0 to finish.
amorynan pushed a commit to amorynan/doris that referenced this pull request Oct 12, 2024
apache#41292)

Consider 3 pipelines in this fragment (... -> join -> shuffle) :
pipeline 0 : `... -> local exchange sink`
pipeline 1 : `... -> join build (INNER JOIN)`
pipeline 2 : `local exchange source -> join probe (INNER JOIN) -> data
stream sender `

Assume the JoinBuild returned 0 rows, join probe can finish directly
once join build finished and do not need to wait for the `local exchange
sink` finished. In this case, if pipeline 0 is blocked by a dependency
for a long time, pipeline 2 should notify pipeline 0 to finish.
BiteTheDDDDt added a commit that referenced this pull request Oct 14, 2024
…41751)

## Proposed changes
Sometimes eos is true, the finish dependency is still not ready. 
Therefore, we need to use wake_up_by_downstream to determine whether it
was closed early.
Otherwise, it will enter the normal rf build process in this case and
generate related errors.
Follow-up : #41292
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 16, 2024
apache#41292)

Consider 3 pipelines in this fragment (... -> join -> shuffle) :
pipeline 0 : `... -> local exchange sink`
pipeline 1 : `... -> join build (INNER JOIN)`
pipeline 2 : `local exchange source -> join probe (INNER JOIN) -> data
stream sender `

Assume the JoinBuild returned 0 rows, join probe can finish directly
once join build finished and do not need to wait for the `local exchange
sink` finished. In this case, if pipeline 0 is blocked by a dependency
for a long time, pipeline 2 should notify pipeline 0 to finish.
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 16, 2024
Gabriel39 pushed a commit to Gabriel39/incubator-doris that referenced this pull request Oct 16, 2024
…ly (apache#41667)

send ignored rf when hash join build closed early to avoid runtime
filter sync/merge error
Follow-up : apache#41292
Gabriel39 pushed a commit to Gabriel39/incubator-doris that referenced this pull request Oct 16, 2024
…pache#41751)

Sometimes eos is true, the finish dependency is still not ready.
Therefore, we need to use wake_up_by_downstream to determine whether it
was closed early.
Otherwise, it will enter the normal rf build process in this case and
generate related errors.
Follow-up : apache#41292
qzsee pushed a commit to qzsee/incubator-doris that referenced this pull request Oct 16, 2024
…pache#41751)

## Proposed changes
Sometimes eos is true, the finish dependency is still not ready. 
Therefore, we need to use wake_up_by_downstream to determine whether it
was closed early.
Otherwise, it will enter the normal rf build process in this case and
generate related errors.
Follow-up : apache#41292
Gabriel39 added a commit that referenced this pull request Oct 16, 2024
 #41751 (#41927)

## Proposed changes

pick #41292 #41350 #41589 #41628 #41743 #41601 #41667 #41751

<!--Describe your changes.-->

---------

Co-authored-by: Pxl <pxl290@qq.com>
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 17, 2024
apache#41292)

Consider 3 pipelines in this fragment (... -> join -> shuffle) :
pipeline 0 : `... -> local exchange sink`
pipeline 1 : `... -> join build (INNER JOIN)`
pipeline 2 : `local exchange source -> join probe (INNER JOIN) -> data
stream sender `

Assume the JoinBuild returned 0 rows, join probe can finish directly
once join build finished and do not need to wait for the `local exchange
sink` finished. In this case, if pipeline 0 is blocked by a dependency
for a long time, pipeline 2 should notify pipeline 0 to finish.
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 17, 2024
Gabriel39 pushed a commit to Gabriel39/incubator-doris that referenced this pull request Oct 17, 2024
…ly (apache#41667)

## Proposed changes
send ignored rf when hash join build closed early to avoid runtime
filter sync/merge error
Follow-up : apache#41292
Gabriel39 pushed a commit to Gabriel39/incubator-doris that referenced this pull request Oct 17, 2024
…pache#41751)

## Proposed changes
Sometimes eos is true, the finish dependency is still not ready. 
Therefore, we need to use wake_up_by_downstream to determine whether it
was closed early.
Otherwise, it will enter the normal rf build process in this case and
generate related errors.
Follow-up : apache#41292
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants