Remove dependencies from TaskExecutionSpecification by edoakes · Pull Request #5166 · ray-project/ray

edoakes · 2019-07-10T23:59:19Z

What do these changes do?

Removes the dependencies field from TaskExecutionSpecification in favor of a new PreviousActorTaskDummyObjectId field in TaskSpecification.

This was an artifact left over from a previous design, and the dependencies were never actually updated during runtime. Rather, it was only used to specify dependencies for actor tasks on the previously run actor task. Now, there is a dedicated field for this use case that is only specified as part of actor task submissions.

The change has a nice byproduct of simplifying the task submission interfaces in the core worker.

Related issue number

Linter

I've run scripts/format.sh to lint the changes in this PR.

edoakes · 2019-07-11T00:15:43Z

Note that this touches many of the same files as #5147

AmplabJenkins · 2019-07-11T00:50:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1599/
Test FAILed.

AmplabJenkins · 2019-07-11T02:33:23Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15289/
Test PASSed.

edoakes · 2019-07-11T16:22:34Z

Updated to rebase off of the changes merged in #5147

stephanie-wang

A few tiny things, but looks great!

src/ray/core_worker/context.cc

src/ray/core_worker/task_interface.cc

python/ray/includes/task.pxi

AmplabJenkins · 2019-07-11T18:17:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1622/
Test FAILed.

AmplabJenkins · 2019-07-11T18:44:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1623/
Test FAILed.

AmplabJenkins · 2019-07-11T19:03:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15313/
Test PASSed.

AmplabJenkins · 2019-07-11T19:13:34Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15314/
Test PASSed.

AmplabJenkins · 2019-07-11T21:29:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1627/
Test FAILed.

stephanie-wang

Thanks, looks good!

Please try to avoid force-pushing in the future, as it makes it hard to see what the new changes are.

AmplabJenkins · 2019-07-11T23:47:48Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15324/
Test FAILed.

AmplabJenkins · 2019-07-11T23:47:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15318/
Test PASSed.

AmplabJenkins · 2019-07-12T00:10:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1633/
Test FAILed.

src/ray/protobuf/common.proto

AmplabJenkins · 2019-07-12T20:49:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15346/
Test PASSed.

AmplabJenkins · 2019-07-12T23:30:20Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15353/
Test PASSed.

pcmoritz · 2019-07-12T23:44:14Z

This is almost ready, there is another Travis error:

Use --sandbox_debug to see verbose messages from the sandbox
src/ray/raylet/lib/java/org_ray_runtime_raylet_RayletClientImpl.cc: In function 'void Java_org_ray_runtime_raylet_RayletClientImpl_nativeSubmitTask(JNIEnv*, jclass, jlong, jbyteArray, jbyteArray)':
src/ray/raylet/lib/java/org_ray_runtime_raylet_RayletClientImpl.cc:78:76: error: no matching function for call to 'RayletClient::SubmitTask(std::vector<ray::ObjectID>&, ray::TaskSpecification&)'
   auto status = raylet_client->SubmitTask(execution_dependencies, task_spec);
                                                                            ^
In file included from src/ray/raylet/lib/java/org_ray_runtime_raylet_RayletClientImpl.cc:6:0:
src/ray/raylet/raylet_client.h:85:15: note: candidate: ray::Status RayletClient::SubmitTask(const ray::TaskSpecification&)
   ray::Status SubmitTask(const ray::TaskSpecification &task_spec);
               ^
src/ray/raylet/raylet_client.h:85:15: note:   candidate expects 1 argument, 2 provided
INFO: Elapsed time: 327.299s, Critical Path: 26.18s
INFO: 2021 processes: 1947 remote cache hit, 73 linux-sandbox, 1 worker.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

AmplabJenkins · 2019-07-13T05:00:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1654/
Test FAILed.

AmplabJenkins · 2019-07-13T14:03:07Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1657/
Test FAILed.

AmplabJenkins · 2019-07-15T05:25:59Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15377/
Test PASSed.

src/ray/raylet/lib/java/org_ray_runtime_raylet_RayletClientImpl.cc

AmplabJenkins · 2019-07-15T18:26:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15388/
Test PASSed.

AmplabJenkins · 2019-07-15T20:31:16Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15390/
Test PASSed.

AmplabJenkins · 2019-07-16T01:09:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15393/
Test PASSed.

AmplabJenkins · 2019-07-16T03:21:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1681/
Test FAILed.

edoakes changed the title ~~[wip] Remove dependencies from TaskExecutionSpecification~~ Remove dependencies from TaskExecutionSpecification Jul 10, 2019

stephanie-wang self-assigned this Jul 11, 2019

edoakes force-pushed the actor-task-dep branch 2 times, most recently from 7a0803d to 559f42f Compare July 11, 2019 16:15

stephanie-wang requested changes Jul 11, 2019

View reviewed changes

src/ray/core_worker/context.cc Outdated Show resolved Hide resolved

src/ray/core_worker/task_interface.cc Outdated Show resolved Hide resolved

python/ray/includes/task.pxi Outdated Show resolved Hide resolved

edoakes added 3 commits July 11, 2019 13:40

Remove dependencies from TaskExecutionSpecification

28e9bdf

Remove TaskSpec from core worker

fb692c9

Address style comments

7556035

edoakes force-pushed the actor-task-dep branch from a324afe to 7556035 Compare July 11, 2019 20:58

stephanie-wang approved these changes Jul 11, 2019

View reviewed changes

formatting

0e5abd7

raulchen approved these changes Jul 12, 2019

View reviewed changes

src/ray/protobuf/common.proto Outdated Show resolved Hide resolved

edoakes added 4 commits July 12, 2019 10:39

Remove outdated comment

93f4c5d

formatting

63e6398

Merge branch 'master' into actor-task-dep

e75bf72

force travis

ebfe1c5

edoakes added 2 commits July 14, 2019 19:01

Merge branch 'master' into actor-task-dep

e46811d

Java task submission

b241c4a

raulchen reviewed Jul 15, 2019

View reviewed changes

src/ray/raylet/lib/java/org_ray_runtime_raylet_RayletClientImpl.cc Show resolved Hide resolved

edoakes added 2 commits July 15, 2019 08:43

Java fixes

0efc680

Regenerate header file

ad68f01

Fix object id ready check in mock client

8fb480a

pcmoritz merged commit e5be5fd into ray-project:master Jul 16, 2019

Conversation

edoakes commented Jul 10, 2019

What do these changes do?

Related issue number

Linter

Uh oh!

edoakes commented Jul 11, 2019

Uh oh!

AmplabJenkins commented Jul 11, 2019

Uh oh!

AmplabJenkins commented Jul 11, 2019

Uh oh!

edoakes commented Jul 11, 2019

Uh oh!

stephanie-wang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AmplabJenkins commented Jul 11, 2019

Uh oh!

AmplabJenkins commented Jul 11, 2019

Uh oh!

AmplabJenkins commented Jul 11, 2019

Uh oh!

AmplabJenkins commented Jul 11, 2019

Uh oh!

AmplabJenkins commented Jul 11, 2019

Uh oh!

stephanie-wang left a comment

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jul 11, 2019

Uh oh!

AmplabJenkins commented Jul 11, 2019

Uh oh!

AmplabJenkins commented Jul 12, 2019

Uh oh!

Uh oh!

AmplabJenkins commented Jul 12, 2019

Uh oh!

AmplabJenkins commented Jul 12, 2019

Uh oh!

pcmoritz commented Jul 12, 2019

Uh oh!

AmplabJenkins commented Jul 13, 2019

Uh oh!

AmplabJenkins commented Jul 13, 2019

Uh oh!

AmplabJenkins commented Jul 15, 2019

Uh oh!

Uh oh!

AmplabJenkins commented Jul 15, 2019

Uh oh!

AmplabJenkins commented Jul 15, 2019

Uh oh!

AmplabJenkins commented Jul 16, 2019

Uh oh!

AmplabJenkins commented Jul 16, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants