Skip to content

Pass store's kwargs to compute call#3300

Merged
mrocklin merged 2 commits intodask:masterfrom
jakirkham:store_use_kwargs
Mar 19, 2018
Merged

Pass store's kwargs to compute call#3300
mrocklin merged 2 commits intodask:masterfrom
jakirkham:store_use_kwargs

Conversation

@jakirkham
Copy link
Copy Markdown
Member

Fixes #3299
Related #2980

Appears the kwargs intended for the compute call in Dask Array's store were getting dropped after a recent rewrite. This corrects it so that kwargs are passed to the compute call.

cc @mrocklin @rabernat

@jakirkham
Copy link
Copy Markdown
Member Author

Not sure of a good test here given we already passed kwargs in tests and they were just getting dropped. Suggestions welcome.

Appears the `kwargs` intended for the `compute` call in Dask Array's
`store` were getting dropped after a recent rewrite. This corrects it so
that `kwargs` are passed to the `compute` call.
@rabernat
Copy link
Copy Markdown
Contributor

While perhaps overkill, some version of my example from #3299 with the UnreliableStore target would be an effective test of store(..., retries=n).

@martindurant
Copy link
Copy Markdown
Member

For the student: how many retries with UnreliableStore and N partitions would it take to have a >99.9% chance of passing the test? :)

@rabernat
Copy link
Copy Markdown
Contributor

For the student: how many retries with UnreliableStore and N partitions would it take to have a >99.9% chance of passing the test? :)

For this reason, the UnreliableStore should probably be modified to just fail once and then succeed after, via some internal state memory.

@mrocklin
Copy link
Copy Markdown
Member

mrocklin commented Mar 19, 2018 via email

@jakirkham
Copy link
Copy Markdown
Member Author

We probably could create a custom get function and then pass some keyword argument(s) in and check for them in the custom get function. There might be something easier than this though.

@jcrist
Copy link
Copy Markdown
Member

jcrist commented Mar 19, 2018

We probably could create a custom get function and then pass some keyword argument(s) in and check for them in the custom get function. There might be something easier than this though.

That's what I would do (we've done similar things elsewhere). Something like:

def test_forwarded_kwargs()
    called = [False]
    def myget(*args, **kwargs):
        called[0] = True
        assert 'foo' in kwargs
        return dask.get(*args, **kwargs)
    store([source], [target], get=myget, foo='fake keyword')
    assert called[0]

@jakirkham jakirkham force-pushed the store_use_kwargs branch 3 times, most recently from c322877 to a723c0b Compare March 19, 2018 17:04
@jakirkham
Copy link
Copy Markdown
Member Author

Thanks. Went ahead and did exactly that.

Please let me know if there is anything else.

@mrocklin mrocklin merged commit 1e56992 into dask:master Mar 19, 2018
@jakirkham jakirkham deleted the store_use_kwargs branch March 19, 2018 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants