[Dask.order] Remove non-runnable leaf nodes from ordering#10697
Merged
[Dask.order] Remove non-runnable leaf nodes from ordering#10697
Conversation
Member
Author
|
Note: this also includes a mypy fix that is breaking distributed linting |
Member
hendrikmakait
left a comment
There was a problem hiding this comment.
Generally looks good to me, though I do not fully understand the change.
hendrikmakait
approved these changes
Dec 13, 2023
Member
hendrikmakait
left a comment
There was a problem hiding this comment.
LGTM after a second pass.
Member
Author
|
About the change, this is a similar change as #10619 but for leaf nodes instead of root nodes. There is a more thorough argument over there. I might follow up with the same for root nodes later but that is a little more tricky to do well |
Co-authored-by: Hendrik Makait <hendrik@makait.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a follow up to #10660 and addresses the still failing test in dask/distributed#8255
Most notably, this removes a special case in the
process_runnablecode path that caused this kind of graph to be executed too eagerly. The special code path was not hit in the specific unit test I added intest_order.py. I extended the test logic to cover this now.This special branch was added to enable eager execution of some dangling code branches as observed in
test_array_store_final_order(The image shows the correct order)
The dangling branches I was referring to are those linear branches ending in P51 and P15. Without that special casing, those two branches would not have been executed causing the root to be left in memory until the end.
What this visualization is not showing is what causes the problem. This graph is actually reducing to a single task
"store-12345": ["store-map-1", "store-map-2", ...]that is effectively an alias. This alias is throwing off the critical path algorithm such that those thin, small branches are effectively ignored and deemed as not valuable. Without that final reducer, the algorithm is forced to finish the connected graph first.This weird alias is actually something that is not even properly understood by the visualization code. It's rendered as the box in the lower right corner but it doesn't know what to do with it.