map_overlap : multiple dataframe handling#9145

Merged

rjzamora merged 20 commits intodask:mainfrom

faulaire:map_overlap_multiple_dataframe

Jun 29, 2022

Contributor

faulaire commented May 30, 2022 •

edited

Loading

Closes map_overlap with multiple dask dataframes #9042
Tests added / passed
Passes pre-commit run --all-files


          map_overlap : multiple dataframe handling

f999f02

Collaborator

GPUtester commented May 30, 2022

Can one of the admins verify this patch?

github-actions Bot added the dataframe label

faulaire mentioned this pull request

map_overlap with multiple dask dataframes #9042

Closed

faulaire added 2 commits

May 30, 2022 17:07


          map_overlap : Refactoring to use partitionwise_graph function + add e…

7718b74

…nforce_metadata parameter


          map_overlap : Add partition_info + fix documentation

39ab9a1

Member

quasiben commented May 31, 2022

add to allowlist

pavithraes reviewed

View reviewed changes

dask/dataframe/tests/test_rolling.py

pavithraes reviewed

View reviewed changes

dask/dataframe/tests/test_rolling.py Outdated

pavithraes reviewed

View reviewed changes

dask/dataframe/tests/test_rolling.py Outdated

pavithraes reviewed

View reviewed changes

dask/dataframe/rolling.py

pavithraes reviewed

View reviewed changes

dask/dataframe/rolling.py Outdated

pavithraes reviewed

View reviewed changes

dask/dataframe/rolling.py

pavithraes reviewed

View reviewed changes

dask/dataframe/rolling.py Outdated

pavithraes reviewed

View reviewed changes

dask/dataframe/core.py Outdated

pavithraes reviewed

View reviewed changes

dask/dataframe/core.py Outdated

pavithraes reviewed

View reviewed changes

dask/dataframe/core.py Outdated

pavithraes reviewed

View reviewed changes

dask/dataframe/rolling.py Outdated

pavithraes requested a review from ian-r-rose

June 1, 2022 19:09

Member

pavithraes commented Jun 1, 2022

@faulaire Thank you for working on this!

@ian-r-rose and I took a look at it together, and we've shared some initial thoughts. :)

faulaire added 7 commits

June 3, 2022 08:48


          [map_overlap][tests] : use a parametrize arg instead of a loop for be…

18a1d78

…fore/after arguments

+ Fix a type on npartitions parameter


          [map_overlap] add $META at the end of the documentation

1d8e0d7


          Mention align_dataframes=False option in map_overlap error

f654853


          Rename _get_meta to _get_meta_map_partitions

7c5f2d2


          Rename _get_divisions to _get_divisions_map_partitions

e4466f9


          [map_overlap] do not mutate dsk in _get_nexts_partitions and _get_pre…

d500871

…vs_partitions


          [map_overlap] blake / pep8

bf44e5a

Contributor Author

faulaire commented Jun 3, 2022

@pavithraes / @ian-r-rose I performed a few commits to match your comments.
One commit is missing : the rollback of API change. I'll do it if you are sure that we want to keep the existing API.

faulaire requested a review from pavithraes

June 3, 2022 15:19

pavithraes reviewed

View reviewed changes

dask/dataframe/rolling.py Outdated

faulaire added 6 commits

June 9, 2022 12:06


          - Remove is_only_scalar helper function

4f82d50


          - Move imports at top level of the file

49a7a04


          - Update map_overlap documentation

3798b76


          - Linting

dec01a0


          - Rollback API changes

a94d5ff


          - Fix map_overlap tokenization

b9d5c09

faulaire requested a review from pavithraes

June 9, 2022 11:08

pavithraes reviewed

View reviewed changes

dask/dataframe/rolling.py Outdated

pavithraes reviewed

View reviewed changes

dask/dataframe/tests/test_rolling.py Outdated

faulaire added 2 commits

June 11, 2022 16:20


          - Simplify tuple construction


          - Add transform_divisions and align_dataframes as tests parameters

6ff9162

faulaire requested a review from pavithraes

June 11, 2022 14:34

Member

pavithraes commented Jun 16, 2022

@faulaire This PR is looking good, and I think we can mark it ready for review. :)

@jrbourbeau Would you mind taking a look when you get a chance?

faulaire marked this pull request as ready for review

June 16, 2022 13:45

Member

ncclementi commented Jun 22, 2022

This PR looks ready for a final review, @rjzamora would you be able to pick this one up for a final review since
James is OOO for a couple of weeks?

rjzamora reviewed

View reviewed changes

Member

rjzamora left a comment

This is looking really nice @faulaire - Thanks for the work!

Only noticed a couple of minor things.

dask/dataframe/rolling.py Outdated

dask/dataframe/tests/test_rolling.py

faulaire added 2 commits

June 24, 2022 15:24


          - Improve map_overlap docstring

7954ea9

- Improve map_overlap unit tests to cover cases where after and before are datetime.timedelta


          - Add "overlap" prefix to name in order be more consistent with the o…

f3516fc

…riginal default name

jsignell mentioned this pull request

Release 2022.6.1 dask/community#258

Closed

9 tasks

faulaire mentioned this pull request

dask.dataframe.map_partitions docs out of date compared to dask.dataframe.DataFrame.map_partitions #9223

Closed

rjzamora approved these changes

View reviewed changes

Member

rjzamora left a comment

LGTM - Thanks @faulaire !

rjzamora merged commit 352d934 into dask:main

Collaborator

ian-r-rose commented Jun 29, 2022

Thanks @faulaire !

Contributor Author

faulaire commented Jun 29, 2022

Thanks !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels