Skip to content

Allow numpy arrays to be passed by value into tasks (and inlined in the task spec).#1816

Merged
pcmoritz merged 8 commits intoray-project:masterfrom
robertnishihara:inlineargs
Apr 16, 2018
Merged

Allow numpy arrays to be passed by value into tasks (and inlined in the task spec).#1816
pcmoritz merged 8 commits intoray-project:masterfrom
robertnishihara:inlineargs

Conversation

@robertnishihara
Copy link
Copy Markdown
Collaborator

@robertnishihara robertnishihara commented Apr 3, 2018

This is related to #1813.

For some reason, this PR doesn't seem to work. E.g., If I do

import ray
import numpy as np

ray.local_scheduler.check_simple_value(np.ones(3))

The last line returns False.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Apr 3, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4623/
Test FAILed.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Apr 12, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4837/
Test FAILed.

@robertnishihara
Copy link
Copy Markdown
Collaborator Author

robertnishihara commented Apr 12, 2018

retest this please

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Apr 12, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4836/
Test PASSed.

pcmoritz
pcmoritz reviewed Apr 12, 2018
View reviewed changes
src/common/lib/python/common_extension.cc Outdated
Copy link
Copy Markdown
Contributor

@pcmoritz pcmoritz Apr 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks a little confusing, why isn't this subsumed by

+    *num_elements_contained += PyArray_NBYTES(array);
+    return (*num_elements_contained <
+            RayConfig::instance().num_elements_limit());

below?

Copy link
Copy Markdown
Collaborator Author

@robertnishihara robertnishihara Apr 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, thx.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Apr 12, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4839/
Test PASSed.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Apr 13, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4849/
Test FAILed.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Apr 13, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4853/
Test FAILed.

@pcmoritz
Copy link
Copy Markdown
Contributor

pcmoritz commented Apr 13, 2018

retest this please

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Apr 13, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4871/
Test FAILed.

@robertnishihara
Copy link
Copy Markdown
Collaborator Author

robertnishihara commented Apr 13, 2018

retest this please

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Apr 13, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4877/
Test FAILed.

@robertnishihara
Copy link
Copy Markdown
Collaborator Author

robertnishihara commented Apr 15, 2018

For the differences between import_array and import_array1 see https://github.com/numpy/numpy/blob/4cc8cb27e792098875edea9fa000528ca1b34f02/numpy/core/code_generators/generate_numpy_api.py#L130-L134.

@AmplabJenkins
Copy link
Copy Markdown

AmplabJenkins commented Apr 15, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/4927/
Test PASSed.

@pcmoritz pcmoritz merged commit 6ca2c2a into ray-project:master Apr 16, 2018
@pcmoritz pcmoritz deleted the inlineargs branch April 16, 2018 03:36
@robertnishihara robertnishihara mentioned this pull request Apr 16, 2018
Closed
royf added a commit to royf/ray that referenced this pull request Apr 22, 2018
@royf
Merge branch 'master' into softq
eb2b27e 
* master:
  Handle interrupts correctly for ASIO synchronous reads and writes. (ray-project#1929)
  [DataFrame] Adding read methods and tests (ray-project#1712)
  Allow task_table_update to fail when tasks are finished. (ray-project#1927)
  [rllib] Contribute DDPG to RLlib (ray-project#1877)
  [xray] Workers blocked in a `ray.get` release their resources (ray-project#1920)
  Raylet task dispatch and throttling worker startup (ray-project#1912)
  [DataFrame] Eval fix (ray-project#1903)
  [tune] Polishing docs (ray-project#1846)
  [tune] [rllib] Automatically determine RLlib resources and add queueing mechanism for autoscaling (ray-project#1848)
  Preemptively push local arguments for actor tasks (ray-project#1901)
  [tune] Allow fetching pinned objects from trainable functions (ray-project#1895)
  Multithreading refactor for ObjectManager. (ray-project#1911)
  Add slice functionality (ray-project#1832)
  [DataFrame] Pass read_csv kwargs to _infer_column (ray-project#1894)
  Addresses missed comments from multichunk object transfer PR. (ray-project#1908)
  Allow numpy arrays to be passed by value into tasks (and inlined in the task spec). (ray-project#1816)
  [xray] Lineage cache requests notifications from the GCS about remote tasks (ray-project#1834)
  Fix UI issue for non-json-serializable task arguments. (ray-project#1892)
  Remove unnecessary calls to .hex() for object IDs. (ray-project#1910)
  Allow multiple raylets to be started on a single machine. (ray-project#1904)

# Conflicts:
#	python/ray/rllib/__init__.py
#	python/ray/rllib/dqn/dqn.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pcmoritz pcmoritz pcmoritz approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@robertnishihara @AmplabJenkins @pcmoritz