[DataFrame] Adding read methods and tests by adgirish · Pull Request #1712 · ray-project/ray

adgirish · 2018-03-13T20:36:57Z

What do these changes do?

Add temporary IO wrappers that default to the Pandas implementation. Will be deprecated to actual IO functionality.

AmplabJenkins · 2018-03-13T21:37:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4306/
Test PASSed.

AmplabJenkins · 2018-03-16T19:54:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4361/
Test FAILed.

AmplabJenkins · 2018-03-18T03:13:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4379/
Test PASSed.

AmplabJenkins · 2018-03-18T21:28:00Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4388/
Test PASSed.

devin-petersohn

Please implement the two tests listed as TODO.

devin-petersohn · 2018-03-20T22:14:37Z

python/ray/dataframe/test/test_io.py

Please implement these.

devin-petersohn · 2018-03-20T22:14:48Z

python/ray/dataframe/test/test_io.py

Please implement these.

AmplabJenkins · 2018-04-05T23:30:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4677/
Test PASSed.

AmplabJenkins · 2018-04-10T22:45:02Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4786/
Test PASSed.

AmplabJenkins · 2018-04-13T17:04:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4873/
Test FAILed.

AmplabJenkins · 2018-04-13T17:33:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4876/
Test FAILed.

AmplabJenkins · 2018-04-13T17:44:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4874/
Test FAILed.

AmplabJenkins · 2018-04-13T17:53:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4878/
Test FAILed.

devin-petersohn · 2018-04-14T04:41:40Z

Please rebase on master

AmplabJenkins · 2018-04-14T19:46:34Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4898/
Test FAILed.

adgirish · 2018-04-14T19:47:39Z

Jenkins, retest this please.

AmplabJenkins · 2018-04-14T20:48:23Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4900/
Test PASSed.

AmplabJenkins · 2018-04-16T16:13:11Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4939/
Test FAILed.

AmplabJenkins · 2018-04-16T16:47:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4941/
Test FAILed.

AmplabJenkins · 2018-04-16T18:43:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4942/
Test PASSed.

AmplabJenkins · 2018-04-16T20:21:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4948/
Test FAILed.

adgirish · 2018-04-16T23:02:58Z

Jenkins, retest this please.

AmplabJenkins · 2018-04-16T23:31:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4954/
Test FAILed.

AmplabJenkins · 2018-04-17T01:47:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4959/
Test FAILed.

AmplabJenkins · 2018-04-17T02:59:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4962/
Test PASSed.

AmplabJenkins · 2018-04-17T20:39:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4972/
Test PASSed.

…dded io tests to travis

AmplabJenkins · 2018-04-19T20:04:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5004/
Test FAILed.

AmplabJenkins · 2018-04-19T20:19:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5005/
Test FAILed.

AmplabJenkins · 2018-04-19T20:59:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5003/
Test PASSed.

AmplabJenkins · 2018-04-20T00:14:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5007/
Test FAILed.

devin-petersohn · 2018-04-20T17:33:47Z

Jenkins, retest this please

AmplabJenkins · 2018-04-20T18:36:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5013/
Test PASSed.

AmplabJenkins · 2018-04-20T19:45:02Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5017/
Test PASSed.

devin-petersohn · 2018-04-21T01:33:29Z

Merged, Thanks @adgirish and @kunalgosar.

* master: Handle interrupts correctly for ASIO synchronous reads and writes. (ray-project#1929) [DataFrame] Adding read methods and tests (ray-project#1712) Allow task_table_update to fail when tasks are finished. (ray-project#1927) [rllib] Contribute DDPG to RLlib (ray-project#1877) [xray] Workers blocked in a `ray.get` release their resources (ray-project#1920) Raylet task dispatch and throttling worker startup (ray-project#1912) [DataFrame] Eval fix (ray-project#1903) [tune] Polishing docs (ray-project#1846) [tune] [rllib] Automatically determine RLlib resources and add queueing mechanism for autoscaling (ray-project#1848) Preemptively push local arguments for actor tasks (ray-project#1901) [tune] Allow fetching pinned objects from trainable functions (ray-project#1895) Multithreading refactor for ObjectManager. (ray-project#1911) Add slice functionality (ray-project#1832) [DataFrame] Pass read_csv kwargs to _infer_column (ray-project#1894) Addresses missed comments from multichunk object transfer PR. (ray-project#1908) Allow numpy arrays to be passed by value into tasks (and inlined in the task spec). (ray-project#1816) [xray] Lineage cache requests notifications from the GCS about remote tasks (ray-project#1834) Fix UI issue for non-json-serializable task arguments. (ray-project#1892) Remove unnecessary calls to .hex() for object IDs. (ray-project#1910) Allow multiple raylets to be started on a single machine. (ray-project#1904) # Conflicts: # python/ray/rllib/__init__.py # python/ray/rllib/dqn/dqn.py

* master: updates (ray-project#1958) Pin Cython in autoscaler development example. (ray-project#1951) Incorporate C++ Buffer management and Seal global threadpool fix from arrow (ray-project#1950) [XRay] Add consistency check for protocol between node_manager and local_scheduler_client (ray-project#1944) Remove smart_open install. (ray-project#1943) [DataFrame] Fully implement append, concat and join (ray-project#1932) [DataFrame] Fix for __getitem__ string indexing (ray-project#1939) [DataFrame] Implementing write methods (ray-project#1918) [rllib] arr[end] was excluded when end is not None (ray-project#1931) [DataFrame] Implementing API correct groupby with aggregation methods (ray-project#1914) Handle interrupts correctly for ASIO synchronous reads and writes. (ray-project#1929) [DataFrame] Adding read methods and tests (ray-project#1712) Allow task_table_update to fail when tasks are finished. (ray-project#1927) [rllib] Contribute DDPG to RLlib (ray-project#1877) [xray] Workers blocked in a `ray.get` release their resources (ray-project#1920) Raylet task dispatch and throttling worker startup (ray-project#1912) [DataFrame] Eval fix (ray-project#1903)

adgirish force-pushed the resolvingMerge branch from 9366fa4 to 208f8a9 Compare March 16, 2018 19:27

devin-petersohn requested changes Mar 20, 2018

View reviewed changes

adgirish force-pushed the resolvingMerge branch from 50f0236 to 01d5e35 Compare April 13, 2018 17:20

adgirish force-pushed the resolvingMerge branch from fabd8fe to adfb8be Compare April 14, 2018 19:14

devin-petersohn mentioned this pull request Apr 15, 2018

[DataFrame] support read_json #1902

Closed

adgirish force-pushed the resolvingMerge branch from 338e38c to 2cdd6e8 Compare April 16, 2018 16:14

kunalgosar force-pushed the resolvingMerge branch from 3fce4fb to ec75071 Compare April 17, 2018 20:06

adgirish and others added 13 commits April 19, 2018 13:02

updating

e4ef5a3

Temporarily phasing out read_csv code for wrapper while diagnosing, a…

6eb9b9e

…dded io tests to travis

Adding travis

4972b03

restoring distributed read csv

a3dcbc6

resolving rebases

dd47940

lint

4e65aae

Sampling out HD test

d2c5dff

adding dep

6f0aaab

fix pathing

c21c4d3

Flagging out tests

a716b37

resolving read_method issues

effa1b2

fix build issue

05d302d

move additional dependencies to extras

2a43e58

kunalgosar force-pushed the resolvingMerge branch from bdfdf50 to 2a43e58 Compare April 19, 2018 20:03

fixing lint

d60ed33

removing IO dependencies

e06ae97

updated requirements doc

1c1dfa6

devin-petersohn approved these changes Apr 21, 2018

View reviewed changes

devin-petersohn merged commit 3c48783 into ray-project:master Apr 21, 2018

kunalgosar mentioned this pull request May 5, 2018

[DataFrames] Feature request: ray.dataframe should be able to read from sql table and query #1680

Closed

Conversation

adgirish commented Mar 13, 2018

What do these changes do?

Uh oh!

AmplabJenkins commented Mar 13, 2018

Uh oh!

AmplabJenkins commented Mar 16, 2018

Uh oh!

AmplabJenkins commented Mar 18, 2018

Uh oh!

AmplabJenkins commented Mar 18, 2018

Uh oh!

devin-petersohn left a comment

Choose a reason for hiding this comment

Uh oh!

devin-petersohn Mar 20, 2018

Choose a reason for hiding this comment

Uh oh!

kunalgosar Apr 17, 2018

Choose a reason for hiding this comment

Uh oh!

devin-petersohn Mar 20, 2018

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Apr 5, 2018

Uh oh!

AmplabJenkins commented Apr 10, 2018

Uh oh!

AmplabJenkins commented Apr 13, 2018

Uh oh!

AmplabJenkins commented Apr 13, 2018

Uh oh!

AmplabJenkins commented Apr 13, 2018

Uh oh!

AmplabJenkins commented Apr 13, 2018

Uh oh!

devin-petersohn commented Apr 14, 2018

Uh oh!

AmplabJenkins commented Apr 14, 2018

Uh oh!

adgirish commented Apr 14, 2018

Uh oh!

AmplabJenkins commented Apr 14, 2018

Uh oh!

AmplabJenkins commented Apr 16, 2018

Uh oh!

AmplabJenkins commented Apr 16, 2018

Uh oh!

AmplabJenkins commented Apr 16, 2018

Uh oh!

AmplabJenkins commented Apr 16, 2018

Uh oh!

adgirish commented Apr 16, 2018

Uh oh!

AmplabJenkins commented Apr 16, 2018

Uh oh!

AmplabJenkins commented Apr 17, 2018

Uh oh!

AmplabJenkins commented Apr 17, 2018

Uh oh!

AmplabJenkins commented Apr 17, 2018

Uh oh!

AmplabJenkins commented Apr 19, 2018

Uh oh!

AmplabJenkins commented Apr 19, 2018

Uh oh!

AmplabJenkins commented Apr 19, 2018

Uh oh!

AmplabJenkins commented Apr 20, 2018

Uh oh!

devin-petersohn commented Apr 20, 2018

Uh oh!

AmplabJenkins commented Apr 20, 2018

Uh oh!

AmplabJenkins commented Apr 20, 2018

Uh oh!

devin-petersohn commented Apr 21, 2018

Uh oh!

Reviewers

Assignees