[REVIEW] HighLevelGraph validation and dependencies fixes#6588
[REVIEW] HighLevelGraph validation and dependencies fixes#6588mrocklin merged 20 commits intodask:masterfrom
Conversation
Follows on from dask#6508
| if hasattr(self, "_dask_layers"): | ||
| return self._dask_layers | ||
| else: | ||
| return (self.key,) |
There was a problem hiding this comment.
I would expect this to always be self.key. I'm curious in which cases this assumption is incorrect.
There was a problem hiding this comment.
This boils down to the discrepancy between the definition of __dask_layers__ that should return output layers and Delayed that might not have any layers.
Delayed.key is always a low level graph key and so this is kind of a hack that make Delayed return a layer if possible.
|
In general this seems like a very useful cleanup. Thank you @madsbk for doing this. I am curious about |
|
I apologize for the delay in reviewing this. This looks good to me. Thank you for this cleanup @madsbk . Merging. |
Follows on from dask#6508 * Added core.keys_in_tasks() * HLG: added more thorough validation check * lingalg: layer dependencies are now sets * linalg.lstsq(): fixed dependencies * linalg.sfqr(): fixed dependencies * linalg.tsqr(): removed dependency when all slices are known * linalg.tsqr(): name_q2bs are name_q2cs are not layers * added help function compute_layer_dependencies() * elemwise(): fixed dependencies * Delayed: fixed __dask_layers__() of HLGs * _loc_list(): empty dependencies on empty index * reformat: black * keys_in_tasks(): fixed doc example * optimize_blockwise(): fixed dependencies * tril() and triu(): fixed dependencies when empty result * added a doctest: +SKIP
Expanding on the work of #6509, this PR implement a complete HighLevelGraph validation by re-calculating the layer dependencies and fixes all incorrect dependencies.
black dask/flake8 dask