Add `shape=None` to *_like() array creation functions by andersy005 · Pull Request #6064 · dask/dask

andersy005 · 2020-04-03T18:58:54Z

Tests added / passed
Passes black dask / flake8 dask

Towards fixing #4875

…-like-functions

andersy005 · 2020-04-03T19:00:21Z

Cc @mrocklin, @pentschev

dask/array/creation.py

…-like-functions

mrocklin

Thanks for doing this @andersy005 . I left some minor comments, mostly stylistic.

dask/array/creation.py

dask/array/tests/test_creation.py

dask/array/creation.py

…-like-functions

andersy005 · 2020-04-14T21:12:28Z

It looks like the test failures here are unrelated to the changes I made.

jrbourbeau

Thanks @andersy005! I've restarted the failed CI builds here to see if any test failures persist

…n-like-functions

TomAugspurger · 2020-04-22T14:16:40Z

Merged master to hopefully fix the CI stuff.

…-like-functions

jsignell

Can you reproduce the failures locally?

dask/array/creation.py

…-like-functions

andersy005 · 2020-04-24T22:29:26Z

Can you reproduce the failures locally?

Yes. All tests in dask/array/tests are passing, and all failing tests are coming from dask/dataframe/tests. I am going to look into whether the changes I made are affecting existing functionality in dask/dataframe

…-like-functions

dask/array/creation.py

jsignell · 2020-06-09T13:49:49Z

I just ran the tests locally and there are genuine failures coming from the dask.dataframe tests. In particular, this test is faililng on most parameter values:

dask/dask/dataframe/tests/test_ufunc.py

Line 444 in ac38635

def test_2args_with_array(ufunc, pandas, darray):

I can't quite figure out why yet...

andersy005 · 2020-06-09T14:20:22Z

I can't quite figure out why yet...

Me either. I spent some time this morning looking into it, but I still don't know yet the connection between the changes I made and the dask.dataframe. I could use some help in figuring it out :)

jsignell · 2020-06-09T14:25:26Z

Oh! I think I have something. I don't think the logic in the helper function is quite right. If shape is None it gets overridden and then chunks ends up being auto. I'll add a suggestion in a sec.

dask/array/creation.py

jsignell · 2020-06-09T15:22:04Z

The test failures are related to the empty_like function accepting a shape kwarg. Maybe @jcrist has some ideas about what's going on here since he's been working on this area in #6276

jsignell · 2020-06-09T22:36:29Z

Ok. So I think what is happening is in ufunc we end up trying to generate a meta and when we do that we call empy_like_safe:

dask/dask/dataframe/core.py

Lines 4756 to 4763 in dd682cc

    
           parts = [ 
        
               d._meta 
        
               if _is_broadcastable(d) 
        
               else empty_like_safe(d, (), dtype=d.dtype) 
        
               if isinstance(d, Array) 
        
               else d._meta_nonempty 
        
               for d in dasks 
        
           ]

which in turn goes and calls np.empty_like(a, .. so now that this PR makes things NEP18, the result of that is a dask array rather than a numpy array. And that causes problems because metas are supposed to be non-dask objects.

So I think this is the only change that is needed:

diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
index 72851b66..f059bb6c 100644
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -4756,7 +4756,7 @@ def elemwise(op, *args, **kwargs):
         parts = [
             d._meta
             if _is_broadcastable(d)
-            else empty_like_safe(d, (), dtype=d.dtype)
+            else np.empty((), dtype=d.dtype)
             if isinstance(d, Array)
             else d._meta_nonempty
             for d in dasks

jsignell · 2020-06-09T22:40:27Z

With that diff the tests pass for me locally :)

andersy005 · 2020-06-09T23:17:20Z

With that diff the tests pass for me locally :)

Many thanks for hunting this down!

jsignell · 2020-06-10T12:47:49Z

Hooray!! @dask/maintenance I think this one is ready!

jakirkham · 2020-06-10T16:35:55Z

Thanks @andersy005 for the PR and everyone for the reviews! 😄

andersy005 added 2 commits April 3, 2020 01:58

Add shape=None to *_like() array creation funcs

6a11d7d

Merge branch 'master' of github.com:dask/dask into array/shape-arg-in…

9419009

…-like-functions

jakirkham reviewed Apr 3, 2020

View reviewed changes

dask/array/creation.py Outdated Show resolved Hide resolved

Merge branch 'master' of github.com:dask/dask into array/shape-arg-in…

8e8fad3

…-like-functions

mrocklin reviewed Apr 4, 2020

View reviewed changes

andersy005 added 5 commits April 12, 2020 13:05

Merge branch 'master' of github.com:dask/dask into array/shape-arg-in…

1a412f5

…-like-functions

Add helper function for finding shape/chunks

fe8f3be

Skipif NumPy <1.17

d6dcfa6

Fix typo

f7ae69f

Merge branch 'master' of github.com:dask/dask into array/shape-arg-in…

ed216c1

…-like-functions

jrbourbeau reviewed Apr 17, 2020

View reviewed changes

Merge remote-tracking branch 'upstream/master' into array/shape-arg-i…

015d000

…n-like-functions

Merge branch 'master' of github.com:dask/dask into array/shape-arg-in…

43899e7

…-like-functions

danielballan mentioned this pull request Apr 24, 2020

NEP-18 not working scikit-beam/autocorr#15

Open

jsignell reviewed Apr 24, 2020

View reviewed changes

dask/array/creation.py Outdated Show resolved Hide resolved

andersy005 added 2 commits April 24, 2020 15:27

Merge branch 'master' of github.com:dask/dask into array/shape-arg-in…

9255278

…-like-functions

Update empty_like. Thank you @jsignell

61ae6ea

andersy005 added 4 commits May 6, 2020 10:18

Merge branch 'master' of github.com:dask/dask into array/shape-arg-in…

504a712

…-like-functions

Merge branch 'master' of github.com:dask/dask into array/shape-arg-in…

5aef8ed

…-like-functions

Merge branch 'master' of github.com:dask/dask into array/shape-arg-in…

89b74d6

…-like-functions

Update signature

f962854

jsignell reviewed Jun 9, 2020

View reviewed changes

dask/array/creation.py Outdated Show resolved Hide resolved

Add "order" back (Thank you @jsignell)

a27ca93

jsignell reviewed Jun 9, 2020

View reviewed changes

dask/array/creation.py Outdated Show resolved Hide resolved

Simplify helper function logic

6440d56

Replace empty_like_safe call with an explicity np.empty call

1b6d167

martindurant merged commit 4eef8e2 into dask:master Jun 10, 2020

andersy005 deleted the array/shape-arg-in-like-functions branch June 10, 2020 13:36

jsignell mentioned this pull request Jun 10, 2020

Cleanup and modernize dask.array creation functions (e.g. da.ones) #6276

Open

2 tasks

jakirkham mentioned this pull request Aug 27, 2020

GPU Support dask/dask-image#133

Open

pentschev mentioned this pull request Feb 15, 2021

Implement shape argument in *_like functions #4875

Closed

Uh oh!

Conversation

andersy005 commented Apr 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andersy005 commented Apr 3, 2020

Uh oh!

Uh oh!

mrocklin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andersy005 commented Apr 14, 2020

Uh oh!

jrbourbeau left a comment

Choose a reason for hiding this comment

Uh oh!

TomAugspurger commented Apr 22, 2020

Uh oh!

jsignell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andersy005 commented Apr 24, 2020

Uh oh!

Uh oh!

jsignell commented Jun 9, 2020

Uh oh!

andersy005 commented Jun 9, 2020

Uh oh!

jsignell commented Jun 9, 2020

Uh oh!

Uh oh!

jsignell commented Jun 9, 2020

Uh oh!

jsignell commented Jun 9, 2020

Uh oh!

jsignell commented Jun 9, 2020

Uh oh!

andersy005 commented Jun 9, 2020

Uh oh!

jsignell commented Jun 10, 2020

Uh oh!

jakirkham commented Jun 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

andersy005 commented Apr 3, 2020 •

edited

Loading