Conversation
| else: | ||
| # Common case, drop extra parameters | ||
| values = [(getitem, arr, x) for x in slices] | ||
| getter = partial(getitem, arr) |
There was a problem hiding this comment.
In general we prefer to avoid including partial, lambdas, closures, or other dynamically generated functions in task graphs. They're harder to serialize.
| return list(product(*slices)) | ||
|
|
||
|
|
||
| def getem( |
There was a problem hiding this comment.
We should check with Xarray tests to make sure that we don't break anything there. They may use this (I hope not though).
|
@rjzamora if you have a moment can you take a look at this? |
|
I think the thing which will be trickiest here is rewriting slicing to propagate down to the array access level. The blockwise IO layer is opaque, so any subsequent slicing can't get past it. If I understand the corresponding parquet code on the dataframe side, there is rewriting of the blockwise IO in the optimization phase to get around this. Is that right @rjzamora? |
|
Superseded by #7417 |
Follow-up to #6931. Switches
from_arrayand its ilk (zarr, hdf5) to useBlockwiseIO.black dask/flake8 dask