Skip to content

Move DataFrame demo IO to Blockwise#7237

Closed
rjzamora wants to merge 121 commits intodask:mainfrom
rjzamora:make-timeseries-blockwise
Closed

Move DataFrame demo IO to Blockwise#7237
rjzamora wants to merge 121 commits intodask:mainfrom
rjzamora:make-timeseries-blockwise

Conversation

@rjzamora
Copy link
Member

@rjzamora rjzamora commented Feb 17, 2021 •
edited
Loading

Blocked by #7042

Continuation of #7042

  • Refactors make_timeseries to use Blockwise.
  • Refactors daily_stock to use Blockwise.

rjzamora added 30 commits January 7, 2021 12:04
@rjzamora
remove blockwiseio
387bacc
@rjzamora
cleanup
9a7e2e1
@rjzamora
orc support
4105ed0
@rjzamora
support csv
cc0cd7d
@rjzamora
cleanup
593cb9a
@rjzamora
add ensure_tuple_key
1485c1f
@rjzamora
add test_parquet_distributed_roundtrip
c46da61
@rjzamora
remove unnecessary test
f982f99
@rjzamora
incorperate dask#6989 fix
5d61bf4
@rjzamora
Add back Ian's blockwise array-creation
6736adb
@rjzamora
change CreateArraySubgraph name to CreateArrayWrapper
8c305e9
@rjzamora
include test for dask#7017
f21ab5c
@rjzamora
add test for dask#7005
5e6fca8
@rjzamora
fix repr bug
f41a68e
@rjzamora
progress on reduced graph size
91ace2e
@rjzamora
read_parquet working
3cd5656
@rjzamora
save csv attempt
0b97ed9
@rjzamora
avoid using Blockwise for csv for now (will take more effort than par…
3c3ad48 
…quet/orc
@rjzamora
support array creation
bc5fd00
@rjzamora
improve docstring
6e05a9d
@rjzamora
support parquet serialization - creating new kwargs_bcast object to a…
5d1ab56 
…void duplicating the same kwargs for every output partition
@rjzamora
improve docstrings a bit
78f67c1
@rjzamora
change kwargs_bcast to kwargs_engine
6fc652e
@rjzamora
starting to remove path_or_frag - need to regenerate fragment on work…
18797a4 
…er for row-wise filtering
@rjzamora
save state
b8b50d8
@rjzamora
never pass fragment objects in graph
f6331ae
@rjzamora
clean out use of read_from_paths throughout read_parquet
c9c5ab0
@rjzamora
remove read_from_paths from read_parquet kwargs
cfab192
@rjzamora
remove all serialization from Blockwise+IO
ee3cbf5
@rjzamora
merge with master
176aa50
@rjzamora rjzamora changed the title Move datasets.timeseries to Blockwise and generalize getitem optimization Blockwise-layer improvements for DataFrame IO Feb 17, 2021
@rjzamora rjzamora mentioned this pull request Feb 17, 2021
Closed
29 tasks
@rjzamora rjzamora changed the title Blockwise-layer improvements for DataFrame IO Move DataFrame demo IO to Blockwise Feb 26, 2021
@rjzamora rjzamora mentioned this pull request Feb 26, 2021
Closed
Base automatically changed from master to main March 8, 2021 20:19
@rjzamora rjzamora mentioned this pull request Apr 27, 2021
Merged
@rjzamora
Copy link
Member Author

rjzamora commented Apr 27, 2021

Closing in favor of #7615

@rjzamora rjzamora closed this Apr 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@rjzamora