[core] Refactor task arguments and attach owner address by stephanie-wang · Pull Request #9152 · ray-project/ray

stephanie-wang · 2020-06-25T21:56:37Z

Why are these changes needed?

The owner address will be needed for ObjectIDs passed to the raylet for several reasons:

To contact the owner and check if a required object cannot be fetched, due to some failure.
To contact the owner for an object's locations, once the object directory is moved into the owner.

The owner address is available for ObjectIDs that the process has a reference to, but not for task arguments that are passed by reference. This PR attaches the owner's address to all arguments passed by reference in the task spec.

This PR also contains two refactors:

Changes TaskArg to a virtual class, implemented by either TaskArgByReference or TaskArgByValue.
Changes TaskArg to only contain one ID (as far as I could tell, the comment about it needing to contain multiple IDs is very outdated and there is no current plan to support multiple IDs in a single argument).

Checks

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/latest/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested (please justify below)

…owers

AmplabJenkins · 2020-06-25T21:59:20Z

Can one of the admins verify this patch?

AmplabJenkins · 2020-06-26T04:05:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27583/
Test FAILed.

AmplabJenkins · 2020-06-26T07:03:31Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27584/
Test FAILed.

zhuohan123

LGTM

zhuohan123 · 2020-07-01T00:14:25Z

src/ray/protobuf/common.proto

+  // ObjectID that the worker has a reference to.
+  bytes object_id = 1;
+  // The address of the object's owner.
+  Address owner_address = 3;


why is this 3?

AmplabJenkins · 2020-07-01T00:47:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27735/
Test FAILed.

AmplabJenkins · 2020-07-01T04:29:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27727/
Test FAILed.

AmplabJenkins · 2020-07-02T02:29:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27802/
Test PASSed.

AmplabJenkins · 2020-07-02T03:05:48Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27808/
Test PASSed.

AmplabJenkins · 2020-07-02T05:57:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27810/
Test PASSed.

AmplabJenkins · 2020-07-02T08:11:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27820/
Test PASSed.

AmplabJenkins · 2020-07-02T10:18:00Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27818/
Test PASSed.

This reverts commit 275da2e.

AmplabJenkins · 2020-07-06T05:13:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27881/
Test PASSed.

This reverts commit 4a326fc.

AmplabJenkins · 2020-07-07T00:08:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27908/
Test FAILed.

AmplabJenkins · 2020-07-07T01:23:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27919/
Test PASSed.

AmplabJenkins · 2020-07-07T10:06:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27944/
Test FAILed.

…project#9152)" This reverts commit b42d6a1.

…ss (ray-project#9152)"" This reverts commit 3aaf1aa.

stephanie-wang added 7 commits June 25, 2020 10:49

Add intended worker ID to GetObjectStatus, tests

34c6088

Remove TaskID owner_id

00bcb8e

Merge remote-tracking branch 'upstream/master' into ref-counting-borr…

bff4c11

…owers

lint

ba184b3

Add owner address to task args

d2297f5

Make TaskArg a virtual class, remove multi args

f277e82

Set owner address for task args

8cea621

stephanie-wang changed the title ~~[core] Refactor task arguments and attach owner address~~ [WIP][core] Refactor task arguments and attach owner address Jun 25, 2020

stephanie-wang changed the title ~~[WIP][core] Refactor task arguments and attach owner address~~ [core] Refactor task arguments and attach owner address Jun 26, 2020

stephanie-wang added 2 commits June 25, 2020 17:56

Merge remote-tracking branch 'upstream/master' into refactor-task-arg

3c654cc

merge

f11ed5a

stephanie-wang added 2 commits June 30, 2020 15:25

Fix tests

de1b140

Fix

2b91193

zhuohan123 approved these changes Jul 1, 2020

View reviewed changes

stephanie-wang added 4 commits July 1, 2020 16:02

build

1db869c

update

3b1e04e

build

7c144c1

java

877382c

stephanie-wang added 2 commits July 1, 2020 22:19

Move code

3e86d44

build

24a099a

stephanie-wang added 4 commits July 2, 2020 09:48

Merge remote-tracking branch 'upstream/master' into refactor-task-arg

4ed1141

Revert "Fix Google log directory again (ray-project#9063)"

4a326fc

This reverts commit 275da2e.

Fix free

cfe61ea

x

309eac2

ericl approved these changes Jul 3, 2020

View reviewed changes

build

b8c22a7

stephanie-wang force-pushed the refactor-task-arg branch from 31808fb to b8c22a7 Compare July 3, 2020 23:39

stephanie-wang mentioned this pull request Jul 3, 2020

[core] Pass owner address from the workers to the raylet #9299

Merged

6 tasks

stephanie-wang added 2 commits July 6, 2020 11:01

Fix java

1e82b40

Revert "Revert "Fix Google log directory again (ray-project#9063)""

bbd23b2

This reverts commit 4a326fc.

lint

7f2986f

stephanie-wang merged commit b42d6a1 into ray-project:master Jul 7, 2020

stephanie-wang deleted the refactor-task-arg branch July 7, 2020 04:25

kfstorm mentioned this pull request Jul 22, 2020

[Java] Attach owner address for pass-by-reference task arguments #9634

Merged

6 tasks

chaokunyang mentioned this pull request Jul 28, 2020

[Streaming] fix streaming ci #9675

Merged

6 tasks

lixin-wei added a commit to lixin-wei/ray that referenced this pull request Jul 30, 2020

Revert "[core] Refactor task arguments and attach owner address (ray-…

3aaf1aa

…project#9152)" This reverts commit b42d6a1.

lixin-wei added a commit to lixin-wei/ray that referenced this pull request Aug 5, 2020

Revert "Revert "[core] Refactor task arguments and attach owner addre…

ba43d8a

…ss (ray-project#9152)"" This reverts commit 3aaf1aa.

Conversation

stephanie-wang commented Jun 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Checks

Uh oh!

AmplabJenkins commented Jun 25, 2020

Uh oh!

AmplabJenkins commented Jun 26, 2020

Uh oh!

AmplabJenkins commented Jun 26, 2020

Uh oh!

zhuohan123 left a comment

Choose a reason for hiding this comment

Uh oh!

zhuohan123 Jul 1, 2020

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jul 1, 2020

Uh oh!

AmplabJenkins commented Jul 1, 2020

Uh oh!

AmplabJenkins commented Jul 2, 2020

Uh oh!

AmplabJenkins commented Jul 2, 2020

Uh oh!

AmplabJenkins commented Jul 2, 2020

Uh oh!

AmplabJenkins commented Jul 2, 2020

Uh oh!

AmplabJenkins commented Jul 2, 2020

Uh oh!

AmplabJenkins commented Jul 6, 2020

Uh oh!

AmplabJenkins commented Jul 7, 2020

Uh oh!

AmplabJenkins commented Jul 7, 2020

Uh oh!

AmplabJenkins commented Jul 7, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

stephanie-wang commented Jun 25, 2020 •

edited

Loading