[DataFrame] Implements mode, to_datetime, and get_dummies by kunalgosar · Pull Request #1956 · ray-project/ray

kunalgosar · 2018-04-27T08:02:43Z

Import unique and value_counts directly from pandas.

This PR implements mode, to_datetime, and get_dummies and includes a fix for __getitem__.

AmplabJenkins · 2018-04-27T09:05:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5083/
Test PASSed.

AmplabJenkins · 2018-04-27T09:34:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5084/
Test PASSed.

AmplabJenkins · 2018-04-27T17:01:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5087/
Test FAILed.

devin-petersohn

Left a few comments, overall looks good.

devin-petersohn · 2018-04-27T18:20:01Z

python/ray/dataframe/dataframe.py

add numeric_only to remove the check.

devin-petersohn · 2018-04-27T18:32:35Z

python/ray/dataframe/datetime.py

For the error checking, do you need all of the other parameters?

devin-petersohn · 2018-04-27T18:41:47Z

python/ray/dataframe/reshape.py

from future

AmplabJenkins · 2018-04-27T19:17:54Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5089/
Test PASSed.

p-yang · 2018-04-27T21:59:23Z

python/ray/dataframe/dataframe.py

You can probably reuse the code across both if cases, since they're largely the same

p-yang · 2018-04-27T22:00:31Z

python/ray/dataframe/dataframe.py

Rename this as it's too similar to utils._reindex_helper

kunalgosar · 2018-04-27T23:07:38Z

Thanks @Veryku @devin-petersohn for reviewing! I addressed the comments above.

AmplabJenkins · 2018-04-28T00:11:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5093/
Test PASSed.

p-yang · 2018-04-28T01:23:42Z

python/ray/dataframe/reshape.py

Missing a return statement?

oh good catch, thanks!

p-yang · 2018-04-28T01:30:02Z

python/ray/dataframe/reshape.py

This name is pretty undescriptive

AmplabJenkins · 2018-04-28T02:19:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5095/
Test PASSed.

AmplabJenkins · 2018-04-28T02:29:15Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5096/
Test PASSed.

AmplabJenkins · 2018-04-28T05:40:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5100/
Test PASSed.

AmplabJenkins · 2018-04-28T05:56:54Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5102/
Test PASSed.

devin-petersohn · 2018-05-02T04:48:26Z

Jenkins, retest this please.

AmplabJenkins · 2018-05-02T05:16:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5142/
Test FAILed.

devin-petersohn · 2018-05-02T15:05:09Z

Jenkins, retest this please.

AmplabJenkins · 2018-05-02T16:12:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5148/
Test PASSed.

devin-petersohn · 2018-05-03T06:21:10Z

Merged, thanks @kunalgosar

* magic-methods: fmt Fix IndentationError Write magic methods for SampleBatch/PartialRollout Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) add pthread linking (ray-project#1986) [DataFrame] Add layer of abstraction to allow OID instantiation (ray-project#1984) [DataFrame] Fix blocking issue on _IndexMetadata passing (ray-project#1965)

* master: (21 commits) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) add pthread linking (ray-project#1986) [DataFrame] Add layer of abstraction to allow OID instantiation (ray-project#1984) ...

* master: (25 commits) [DataFrame] Add direct pandas imports for MVP (ray-project#1960) Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) ...

devin-petersohn reviewed Apr 27, 2018

View reviewed changes

p-yang reviewed Apr 27, 2018

View reviewed changes

p-yang reviewed Apr 28, 2018

View reviewed changes

kunalgosar force-pushed the mode branch from b6fb3c6 to 2243b8a Compare April 28, 2018 04:38

kunalgosar added 14 commits May 1, 2018 00:17

implement mode and fix getitem

92c1ac5

mode broken on misaligned partitions

5c017f0

fully implement mode

d69d60b

implement to_datetime

61e16b3

implement get_dummies

9c8f8a9

implement tests

a621d8f

fix __getitem__

430360e

fix python2 compatibility

40f66ea

fix getitem bug

767dc8f

resolving comments

7f89d9f

Adding documentation

102e27d

resolving comment

bc2a443

resolve name change

5a0b4b5

speeding up getitem

928aed3

kunalgosar force-pushed the mode branch from 8001718 to 928aed3 Compare May 1, 2018 07:24

complete rebase

44d0a0d

devin-petersohn approved these changes May 2, 2018

View reviewed changes

devin-petersohn merged commit d85ee0b into ray-project:master May 3, 2018

Conversation

kunalgosar commented Apr 27, 2018

Uh oh!

AmplabJenkins commented Apr 27, 2018

Uh oh!

AmplabJenkins commented Apr 27, 2018

Uh oh!

AmplabJenkins commented Apr 27, 2018

Uh oh!

devin-petersohn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Apr 27, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

p-yang Apr 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kunalgosar commented Apr 27, 2018

Uh oh!

AmplabJenkins commented Apr 28, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Apr 28, 2018

Uh oh!

AmplabJenkins commented Apr 28, 2018

Uh oh!

AmplabJenkins commented Apr 28, 2018

Uh oh!

AmplabJenkins commented Apr 28, 2018

Uh oh!

devin-petersohn commented May 2, 2018

Uh oh!

AmplabJenkins commented May 2, 2018

Uh oh!

devin-petersohn commented May 2, 2018

Uh oh!

AmplabJenkins commented May 2, 2018

Uh oh!

devin-petersohn commented May 3, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

p-yang Apr 27, 2018 •

edited

Loading