Config array optimize to skip fusion and return a HLG#6751
Config array optimize to skip fusion and return a HLG#6751jrbourbeau merged 3 commits intodask:masterfrom
Conversation
a9c472b to
42bc8d3
Compare
|
Is this supposed to be user-facing and last long term, or is it just a temporary debugging thing? If it's user facing then we should add it to https://github.com/dask/dask/blob/master/dask/dask.yaml and https://github.com/dask/dask/blob/master/dask/dask-schema.yaml and our doc build will pick them up. |
I think it should be long term. The option is already in |
jrbourbeau
left a comment
There was a problem hiding this comment.
This seems sensible as we already have similar early-exit behavior for DataFrames and the fuse function in dask/optimization.py, thanks for the PR @madsbk! I pushed a small commit to move the new test to be alongside the existing array optimization tests.
One difference between the array and DataFrame case is that setting optimization.fuse.active to False for DataFrames just excludes low-level task fusion, while in the array case optimization.fuse.active = False excludes both low-level task fusion and an inline_functions optimization. This is probably okay, but something we should be aware of. #6083 seems like a good next step if we want to implement finer disabling options for different types of optimizations.
Additionally, as we build out our high-level graph optimizations (#5644) we might consider adding a config option to turn off all low-level graph optimizations -- which I think is what @madsbk is after here
|
Also, all the test failures here look to be related to #6754 and not the changes in this PR |
Good point and I agree #6083 would be great! |
This PR makes array optimize to exit early when
optimization.fuse.active=Falseand return a high level graph.black dask/flake8 dask