Document meta kwarg in map_blocks and map_overlap.#6763
Document meta kwarg in map_blocks and map_overlap.#6763TomAugspurger merged 4 commits intodask:masterfrom
meta kwarg in map_blocks and map_overlap.#6763Conversation
quasiben
left a comment
There was a problem hiding this comment.
Thank you @pentschev for adding these updates! I left a few small typo comments
meta kwarg in map_blocks and map_overlap.meta kwarg in map_blocks and map_overlap.
|
cc @TomAugspurger @oarcher for awareness. |
|
LGTM! Thanks @pentschev. I'll wait for @TomAugspurger before merging |
dask/array/core.py
Outdated
| Note that this function will attempt to automatically determine the output | ||
| array type before computing it, please refer to the ``meta`` keyword argument |
There was a problem hiding this comment.
| Note that this function will attempt to automatically determine the output | |
| array type before computing it, please refer to the ``meta`` keyword argument | |
| Note that ``map_blocks`` will attempt to automatically determine the output | |
| array type by calling ``func`` on 0-d versions of the inputs. Please refer to the ``meta`` keyword argument |
Might have made that line too long.
dask/array/core.py
Outdated
| will be determined by a hash of the arguments. | ||
| meta : array-like, optional | ||
| The ``meta`` of the output array, when specified is expected to be an | ||
| array of the same type of that returned when calling ``.compute()`` on |
dask/array/core.py
Outdated
| 0-d array. It's important to ensure that ``func`` can successfully complete | ||
| computation without raising exceptions when 0-d is passed to it. If the |
There was a problem hiding this comment.
It may not be feasible for the user to write func to work on 0-d arrays. So perhaps rephrase this as something like "Consider providing meta if your func does not work on 0-d arrays are has expensive side-effects."
dask/array/core.py
Outdated
| 0-d array. It's important to ensure that ``func`` can successfully complete | ||
| computation without raising exceptions when 0-d is passed to it. If the | ||
| output type is known beforehand (e.g., ``np.ndarray``, ``cupy.ndarray``), | ||
| an empty array of such type can be passed, for example: ``meta=np.array(())``. |
There was a problem hiding this comment.
Does the dtype need to mach, or just the type?
| dask.array<increment, shape=(100,), dtype=int64, chunksize=(10,), chunktype=numpy.ndarray> | ||
|
|
||
| For functions that may not handle 0-d arrays, it's also possible to specify | ||
| ``meta`` with an empty array matching the type of the expected result. In |
|
@TomAugspurger I believe all your suggestions are addressed in ba212ec , please take another look when you have a chance. |
TomAugspurger
left a comment
There was a problem hiding this comment.
LGTM once the doctest issues are resolved.
dask/array/core.py
Outdated
| >>> da.map_blocks(lambda x: x[2], rs.random(5, dtype=dt), meta=cupy.array((), dtype=dt)) | ||
| dask.array<lambda, shape=(5,), dtype=float64, chunksize=(5,), chunktype=cupy.ndarray> |
There was a problem hiding this comment.
This will need a doctest skip. https://travis-ci.org/github/dask/dask/jobs/738415644#L1274
dask/array/core.py
Outdated
| >>> y | ||
| dask.array<_trim, shape=(4, 4), dtype=float64, chunksize=(2, 2), chunktype=cupy.ndarray> | ||
| >>> y.compute() |
There was a problem hiding this comment.
Another pair of skips needed I think.
dask/array/overlap.py
Outdated
| >>> y = d.map_overlap(lambda x: x + x[2], depth=1, meta=cupy.array(())) | ||
| >>> y | ||
| dask.array<_trim, shape=(4, 4), dtype=float64, chunksize=(2, 2), chunktype=cupy.ndarray> | ||
| >>> y.compute() |
|
Thanks @TomAugspurger , I had no idea that we could parse docs with doctests, that's super neat! Running that locally I found some mistakes and added the skips in bff4584, hopefully it will succeed now. |
|
Everything passed now, could you take another look when you have a chance @TomAugspurger ? |
|
Thanks! |
|
Thanks for reviews and merging @TomAugspurger ! |
|
Many thanks @pentschev and @TomAugspurger , the documentation is now as I would have like to be ! |
* Documenta `meta` kwarg in `map_blocks` and `map_overlap`. * Small fixes to ``meta`` text * Document using `dtype` with `meta` in `map_blocks` * Skip CuPy doctests
This should address #6690 partially, documenting how/when one should need
meta=inmap_blocksandmap_overlap, as well as providing some very simple examples.