Skip to content

map_partitions with review comments#6776

Merged
jsignell merged 4 commits intodask:masterfrom
kumarprabhu1988:map_partition_info2
Jan 15, 2021
Merged

map_partitions with review comments#6776
jsignell merged 4 commits intodask:masterfrom
kumarprabhu1988:map_partition_info2

Conversation

@kumarprabhu1988
Copy link
Contributor

@kumarprabhu1988 kumarprabhu1988 commented Oct 29, 2020

This PR adds functionality to return partition_info in a call to map_partitions. It is useful for distributed implementations of many algorithms. This PR includes changes for review comments in this PR: #6755.

@kumarprabhu1988 kumarprabhu1988 mentioned this pull request Oct 29, 2020
2 tasks
@jsignell
Copy link
Member

Thanks for keeping at it @kumarprabhu1988 - I think this is getting close. The missing piece now is docs. Could you add a section to the examples here:

def map_partitions(self, func, *args, **kwargs):

@kumarprabhu1988
Copy link
Contributor Author

@jsignell Apologies for the delay. I've updated the docs. Let me know if this looks good.

@jsignell
Copy link
Member

jsignell commented Dec 1, 2020

I really think the doc string should include an example using partition_info. Would you be able to add that?

@jsignell
Copy link
Member

I'll merge this when it passes.

@jrbourbeau
Copy link
Member

Just merged master to bring in some CI fixes (cc @jsignell)

@jrbourbeau
Copy link
Member

Just checking in @jsignell, is this still good to go?

@jsignell jsignell merged commit d9fbc42 into dask:master Jan 15, 2021
abduhbm pushed a commit to abduhbm/dask that referenced this pull request Jan 19, 2021
* map_partitions with review comments

* update doc

* Add example to docstring

Co-authored-by: Julia Signell <jsignell@gmail.com>
Co-authored-by: James Bourbeau <jrbourbeau@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add magic partition_info keyword to dd.map_partitions function

3 participants