[DataFrame] Implementing write methods by kunalgosar · Pull Request #1918 · ray-project/ray

kunalgosar · 2018-04-17T21:16:41Z

This PR is rebased on #1712

This PR implements the write IO functions by falling back on pandas. This is a temporary placeholder and throws a warning for performance degradation.

Opening in favor of #1713

AmplabJenkins · 2018-04-17T22:09:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4976/
Test PASSed.

AmplabJenkins · 2018-04-17T22:26:56Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4978/
Test PASSed.

AmplabJenkins · 2018-04-18T05:26:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4983/
Test FAILed.

kunalgosar · 2018-04-18T07:23:52Z

Jenkins, retest this please.

AmplabJenkins · 2018-04-19T20:29:31Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5006/
Test FAILed.

AmplabJenkins · 2018-04-20T02:29:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5008/
Test FAILed.

AmplabJenkins · 2018-04-21T04:15:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5024/
Test PASSed.

AmplabJenkins · 2018-04-22T08:27:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5029/
Test FAILed.

AmplabJenkins · 2018-04-22T17:57:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5030/
Test FAILed.

devin-petersohn

Looks great, just a quick formatting change.

devin-petersohn · 2018-04-22T19:18:28Z

python/ray/dataframe/test/test_io.py

 import pandas as pd
 from ray.dataframe.utils import to_pandas
-import ray.dataframe.io as io
+import ray.dataframe as rpd


Please change to import ray.dataframe as pd.

devin-petersohn · 2018-04-22T19:18:51Z

python/ray/dataframe/test/test_io.py

@@ -6,7 +6,7 @@
 import numpy as np
 import pandas as pd


Please change to import pandas.

Formatting done.

AmplabJenkins · 2018-04-22T22:08:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5031/
Test PASSed.

AmplabJenkins · 2018-04-22T22:37:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5032/
Test FAILed.

devin-petersohn · 2018-04-22T22:39:06Z

Jenkins, retest this please.

AmplabJenkins · 2018-04-22T23:42:02Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5033/
Test PASSed.

devin-petersohn

Thanks @kunalgosar !

devin-petersohn · 2018-04-23T04:44:54Z

Merged, thanks @kunalgosar!

* master: updates (ray-project#1958) Pin Cython in autoscaler development example. (ray-project#1951) Incorporate C++ Buffer management and Seal global threadpool fix from arrow (ray-project#1950) [XRay] Add consistency check for protocol between node_manager and local_scheduler_client (ray-project#1944) Remove smart_open install. (ray-project#1943) [DataFrame] Fully implement append, concat and join (ray-project#1932) [DataFrame] Fix for __getitem__ string indexing (ray-project#1939) [DataFrame] Implementing write methods (ray-project#1918) [rllib] arr[end] was excluded when end is not None (ray-project#1931) [DataFrame] Implementing API correct groupby with aggregation methods (ray-project#1914) Handle interrupts correctly for ASIO synchronous reads and writes. (ray-project#1929) [DataFrame] Adding read methods and tests (ray-project#1712) Allow task_table_update to fail when tasks are finished. (ray-project#1927) [rllib] Contribute DDPG to RLlib (ray-project#1877) [xray] Workers blocked in a `ray.get` release their resources (ray-project#1920) Raylet task dispatch and throttling worker startup (ray-project#1912) [DataFrame] Eval fix (ray-project#1903)

* 'master' of https://github.com/ray-project/ray: [rllib] Fix broken link in docs (ray-project#1967) [DataFrame] Sample implement (ray-project#1954) [DataFrame] Implement Inter-DataFrame operations (ray-project#1937) remove UniqueIDHasher (ray-project#1957) [rllib] Add DDPG documentation, rename DDPG2 <=> DDPG (ray-project#1946) updates (ray-project#1958) Pin Cython in autoscaler development example. (ray-project#1951) Incorporate C++ Buffer management and Seal global threadpool fix from arrow (ray-project#1950) [XRay] Add consistency check for protocol between node_manager and local_scheduler_client (ray-project#1944) Remove smart_open install. (ray-project#1943) [DataFrame] Fully implement append, concat and join (ray-project#1932) [DataFrame] Fix for __getitem__ string indexing (ray-project#1939) [DataFrame] Implementing write methods (ray-project#1918) [rllib] arr[end] was excluded when end is not None (ray-project#1931) [DataFrame] Implementing API correct groupby with aggregation methods (ray-project#1914)

devin-petersohn mentioned this pull request Apr 17, 2018

[DataFrame] Adding write methods and tests #1713

Closed

kunalgosar force-pushed the cleanWrite branch from f608366 to 0200f04 Compare April 17, 2018 21:27

kunalgosar force-pushed the cleanWrite branch from 49eba3e to 78517cf Compare April 19, 2018 20:24

kunalgosar force-pushed the cleanWrite branch from 78517cf to d3be2cb Compare April 20, 2018 02:25

kunalgosar force-pushed the cleanWrite branch from d3be2cb to 95c7076 Compare April 21, 2018 03:11

kunalgosar added 2 commits April 22, 2018 00:56

Add in write methods and functionality

a88fa84

infer highest available pickle version

d83356c

kunalgosar force-pushed the cleanWrite branch from 2a93757 to d83356c Compare April 22, 2018 07:58

Fix import rebase artifact

83bb7fc

devin-petersohn requested changes Apr 22, 2018

View reviewed changes

formatting changes to test

125c00a

fix lint

c93972f

devin-petersohn approved these changes Apr 23, 2018

View reviewed changes

devin-petersohn merged commit 7c9f392 into ray-project:master Apr 23, 2018

kunalgosar mentioned this pull request Apr 24, 2018

[DataFrames] ADD dataframe.to_sql functionality, using sql alchemy #1689

Closed

Conversation

kunalgosar commented Apr 17, 2018

Uh oh!

AmplabJenkins commented Apr 17, 2018

Uh oh!

AmplabJenkins commented Apr 17, 2018

Uh oh!

AmplabJenkins commented Apr 18, 2018

Uh oh!

kunalgosar commented Apr 18, 2018

Uh oh!

AmplabJenkins commented Apr 19, 2018

Uh oh!

AmplabJenkins commented Apr 20, 2018

Uh oh!

AmplabJenkins commented Apr 21, 2018

Uh oh!

AmplabJenkins commented Apr 22, 2018

Uh oh!

AmplabJenkins commented Apr 22, 2018

Uh oh!

devin-petersohn left a comment

Choose a reason for hiding this comment

Uh oh!

devin-petersohn Apr 22, 2018

Choose a reason for hiding this comment

Uh oh!

devin-petersohn Apr 22, 2018

Choose a reason for hiding this comment

Uh oh!

kunalgosar Apr 22, 2018

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Apr 22, 2018

Uh oh!

AmplabJenkins commented Apr 22, 2018

Uh oh!

devin-petersohn commented Apr 22, 2018

Uh oh!

AmplabJenkins commented Apr 22, 2018

Uh oh!

devin-petersohn left a comment

Choose a reason for hiding this comment

Uh oh!

devin-petersohn commented Apr 23, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants