Skip to content

[DataFrame] Implement IO for ray_df#1599

Merged
devin-petersohn merged 29 commits intoray-project:masterfrom
simon-mo:df_io
Feb 27, 2018
Merged

[DataFrame] Implement IO for ray_df#1599
devin-petersohn merged 29 commits intoray-project:masterfrom
simon-mo:df_io

Conversation

@simon-mo
Copy link
Copy Markdown
Contributor

@simon-mo simon-mo commented Feb 24, 2018 •
edited
Loading

What do these changes do?

This PR adds support for

  • read_csv
  • read_parquet

Related issue number

None

Note

simon-mo and others added 24 commits February 19, 2018 21:11
@simon-mo
Add parquet-cpp to gitignore
a28e092
@simon-mo
Add read_csv and read_parquet
9678cc4
@simon-mo
Gitignore pytest_cache
184b544
@simon-mo
Fix flake8
91a5d7b
@simon-mo
Add io to __init__
a6f27e6
@simon-mo
Merge branch 'master' of https://github.com/ray-project/ray into df_io
9734308
@devin-petersohn
Changing Index. Currently running tests, but so far untested.
bba597c
@devin-petersohn
Removing issue of reassigning DF in from_pandas
82f2bd7
@devin-petersohn
Fixing lint
48dc6fc
@devin-petersohn
Fix bug
bf42a26
@devin-petersohn
Fix bug
f0b2069
@devin-petersohn
Fix bug
52e269a
@devin-petersohn
Better performance
ac717b2
@devin-petersohn
Fixing index issue with sum
7efea34
@simon-mo
Address comments
15a5271
@simon-mo
Merge branch 'df_patch03' of https://github.com/devin-petersohn/ray i…
ef80f9f 
…nto df_io
@simon-mo
Update io with index
32dbd9e
@devin-petersohn
Updating performance and implementation. Adding tests
c1d781e
@devin-petersohn
Fixing off-by-1
f5ee1bd
@devin-petersohn
Fix lint
aa9aca8
@simon-mo
Address Comments
a268c40
@simon-mo
Merge branch 'df_patch03' of https://github.com/devin-petersohn/ray i…
3f1d958 
…nto df_io
@simon-mo
Make pop compatible with new to_pandas
223f3ed
@simon-mo
Format Code
5e583e6
@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Feb 25, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3930/
Test PASSed.

@simon-mo
Copy link
Copy Markdown
Contributor Author

simon-mo commented Feb 25, 2018

This PR is ready for review now.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Feb 26, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3947/
Test PASSed.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Feb 26, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3972/
Test PASSed.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Feb 26, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3973/
Test PASSed.

devin-petersohn
devin-petersohn approved these changes Feb 26, 2018
View reviewed changes
Copy link
Copy Markdown
Member

@devin-petersohn devin-petersohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, just a couple of comments.

python/ray/dataframe/dataframe.py
# reindex here because we want a pd.RangeIndex within the partitions.
# It is smaller and sometimes faster.
t_df.reindex()
# reset_index here because we want a pd.RangeIndex
Copy link
Copy Markdown
Member

@devin-petersohn devin-petersohn Feb 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch! Thanks @simon-mo.

python/ray/dataframe/dataframe.py Outdated
level=level,
numeric_only=numeric_only),
index=temp_index)._df))
# collapsed_df.index = self.columns
Copy link
Copy Markdown
Member

@devin-petersohn devin-petersohn Feb 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be left in?

Copy link
Copy Markdown
Contributor Author

@simon-mo simon-mo Feb 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. Removing it

@simon-mo simon-mo mentioned this pull request Feb 26, 2018
Merged
3 tasks
@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Feb 26, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3974/
Test PASSed.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Feb 26, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3975/
Test PASSed.

@devin-petersohn
Copy link
Copy Markdown
Member

devin-petersohn commented Feb 27, 2018

Passed private-travis build. OK to merge. Thanks @simon-mo!

@devin-petersohn devin-petersohn merged commit d78a22f into ray-project:master Feb 27, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@devin-petersohn devin-petersohn devin-petersohn approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@simon-mo @AmplabJenkins @devin-petersohn