Add atop(concatenate=False) keyword argument#1609
Conversation
|
cc @shoyer |
dask/array/core.py
Outdated
| function should expect an iterator of blocks on any array that holds that | ||
| index. | ||
| function should expect an iterable of blocks on any array that holds that | ||
| index. To receieve an array concatenated along the dummy dimension pass |
There was a problem hiding this comment.
Instead:
To receive arrays concatenated along contracted dimensions instead pass concatenate=True
|
Yes, I found the nested lists of arrays passed previously quite unintuitive, and also a little misleading, because it suggested that the arrays could be processed in a streaming fashion (when in reality they're loaded into memory all at once). |
This allows atop to pre-concatenate arrays before sending to the user defined function.
c6b0288 to
1d7a855
Compare
Historically we used to use iterators and stream things through. This is no longer the case though. |
|
Merging. |
OK, that could explain part of my confusion... |
This allows atop to pre-concatenate arrays before sending to the user
defined function.
Previously we would send an iterable of chunks to the user-defined-function. This allows us to optionally pre-concatenate those arrays ahead of time, simplifying the effort on the user's side.