GCS server error handling for actor creation by ffbin · Pull Request #8899 · ray-project/ray

ffbin · 2020-06-11T09:39:29Z

Why are these changes needed?

Design document:
https://docs.google.com/document/d/1Cuqxlw53abEZPVYVF-pUWpbnwrFEgVTel_sQGwt8DmU/edit#heading=h.s3ogmk8m7308
This PR implements actor management error handling. Lease worker resource recovery will be implemented in the next pr.

Related issue number

Checks

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/latest/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested (please justify below)

AmplabJenkins · 2020-06-11T09:46:37Z

Can one of the admins verify this patch?

AmplabJenkins · 2020-06-11T11:40:23Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27030/
Test PASSed.

AmplabJenkins · 2020-06-11T19:45:06Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27039/
Test PASSed.

AmplabJenkins · 2020-06-12T03:40:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27056/
Test FAILed.

AmplabJenkins · 2020-06-12T04:47:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27054/
Test FAILed.

AmplabJenkins · 2020-06-12T04:58:10Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27059/
Test PASSed.

AmplabJenkins · 2020-06-12T04:58:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27060/
Test FAILed.

AmplabJenkins · 2020-06-12T09:04:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27064/
Test PASSed.

AmplabJenkins · 2020-06-12T09:19:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27069/
Test PASSed.

AmplabJenkins · 2020-06-12T09:40:14Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27063/
Test FAILed.

AmplabJenkins · 2020-06-12T10:17:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27071/
Test PASSed.

AmplabJenkins · 2020-06-12T11:24:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27074/
Test PASSed.

AmplabJenkins · 2020-06-12T12:46:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27073/
Test PASSed.

raulchen · 2020-06-12T12:06:24Z

src/ray/core_worker/transport/direct_actor_transport.cc

No need to record every completed task. We just need to check: 1) the new task is an actor creation task; 2) this worker is already an actor; 3) the actor id matches.

src/ray/gcs/gcs_server/gcs_actor_manager.cc

raulchen · 2020-06-15T09:52:19Z

src/ray/gcs/gcs_server/gcs_actor_manager.cc

src/ray/raylet/node_manager.cc

raulchen · 2020-06-15T12:39:25Z

src/ray/gcs/gcs_server/gcs_actor_scheduler.cc

Now we can put the code for leasing worker (Line 65) right here. And should update DB after leasing a worker.

AmplabJenkins · 2020-06-16T15:26:03Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27219/
Test FAILed.

AmplabJenkins · 2020-06-18T08:58:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27284/
Test FAILed.

raulchen · 2020-06-18T12:02:17Z

python/ray/tests/test_gcs_fault_tolerance.py

Suggested change

def test_gcs_server_re_schedule_actor():

def test_gcs_server_restart_during_actor_creation():

raulchen · 2020-06-18T12:04:12Z

python/ray/tests/test_gcs_fault_tolerance.py

Use ray_start_regular

raulchen · 2020-06-18T12:04:33Z

python/ray/tests/test_gcs_fault_tolerance.py

this is not needed any more?

raulchen · 2020-06-18T12:05:11Z

src/ray/core_worker/core_worker.cc

Suggested change

RAY_LOG(DEBUG) << "Execute task, task info = " << task_spec.DebugString();

RAY_LOG(DEBUG) << "Executing task, task info = " << task_spec.DebugString();

raulchen · 2020-06-18T12:06:00Z

src/ray/core_worker/transport/direct_actor_transport.cc

Suggested change

// If GCS server is restarted after sending create actor task to the core worker, the

// If GCS server is restarted after sending an actor creation task to this core worker, the

raulchen · 2020-06-18T12:12:58Z

src/ray/gcs/gcs_server/gcs_actor_scheduler.cc

Can you add a check to make sure the actor isn't associated with any node id and work id?

raulchen · 2020-06-18T12:15:20Z

src/ray/gcs/gcs_server/gcs_actor_scheduler.cc

Don't need to call Schedule here, just call LeaseWorkerFromNode

raulchen · 2020-06-18T12:18:05Z

src/ray/gcs/gcs_server/gcs_server.cc

Suggested change

auto on_done = [this]() {

auto actor_manager_load_initial_data_callback = [this]() {

Avoid using the same var name. And document that actor data needs be to loaded lastly.

raulchen · 2020-06-18T12:19:20Z

src/ray/raylet/node_manager.cc

Suggested change

local_queues_.RemoveTasks(task_ids);

local_queues_.RemoveTasks({task_id});

raulchen · 2020-06-18T12:20:41Z

src/ray/raylet/node_manager.cc

Just realize that we should only do this for actor creation tasks when gcs service is enabled.

Submitted actor creation task ... is already queue. This is most likely due to a GCS restart. We will remove the old one from the queue, and enqueue the new one.

AmplabJenkins · 2020-06-19T11:01:06Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27335/
Test FAILed.

AmplabJenkins · 2020-06-22T05:03:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27392/
Test FAILed.

AmplabJenkins · 2020-06-22T07:17:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27397/
Test PASSed.

raulchen · 2020-06-30T03:28:47Z

src/ray/core_worker/transport/direct_actor_transport.cc

Suggested change

// the restarted GCS server will send the same create actor task to the core worker

// the restarted GCS server will send the same actor creation task to the core worker

raulchen · 2020-06-30T03:38:26Z

src/ray/gcs/gcs_server/gcs_actor_manager.cc

If this is only for testing, it'd be better to just provide a GetCreatedActors method and do the loop in test code. So people won't accidentally use this.

raulchen · 2020-06-30T03:42:36Z

src/ray/gcs/gcs_server/gcs_actor_scheduler.cc

Suggested change

<< " owners a leased worker. Create actor directly on worker.";

<< " is already tied to a leased worker. Create actor directly on worker.";

raulchen · 2020-06-30T03:59:44Z

src/ray/raylet/node_manager.cc

if (spec.IsActorCreationTask()) {
// ...
} else {
// same as before.
}

AmplabJenkins · 2020-06-30T17:26:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27690/
Test FAILed.

wumuzi520 · 2020-07-01T08:42:27Z

src/ray/gcs/gcs_server/gcs_actor_scheduler.cc

@@ -226,20 +226,16 @@ void GcsActorScheduler::HandleWorkerLeasedReply(
    // node, and then try again on the new node.
    RAY_CHECK(!retry_at_raylet_address.raylet_id().empty());
    actor->UpdateAddress(retry_at_raylet_address);


if (auto node = gcs_node_manager_.GetNode(actor->GetNodeID())) {
} else {
}

AmplabJenkins · 2020-07-01T10:54:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27764/
Test PASSed.

wumuzi520 · 2020-07-01T12:22:34Z

LGTM!

AmplabJenkins · 2020-07-01T13:11:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27767/
Test FAILed.

AmplabJenkins · 2020-07-01T13:18:56Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27770/
Test PASSed.

raulchen · 2020-07-02T03:22:55Z

src/ray/protobuf/gcs.proto

  double timestamp = 13;
  // The task specification of this actor's creation task.
  TaskSpec task_spec = 14;
+  // Resource mapping ids assigned to the worker executing the task.


// Resource mapping ids acquired by the leased worker. This field is only set when this actor already has a leased worker.

raulchen · 2020-07-02T03:39:01Z

src/ray/gcs/gcs_server/test/gcs_actor_scheduler_test.cc

  ASSERT_EQ(actor->GetWorkerID(), worker_id);
 }

+TEST_F(GcsActorSchedulerTest, TestReschedule) {


should test 2 cases: 1) actor is already tied to a leased worker; 2) not tied.

AmplabJenkins · 2020-07-02T07:44:57Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/27816/
Test FAILed.

ffbin requested a review from raulchen June 11, 2020 09:39

rkooo567 requested a review from stephanie-wang June 11, 2020 16:54

ffbin force-pushed the dev_gcs_server_re_schedule branch 3 times, most recently from 892fe29 to ab9d2be Compare June 12, 2020 03:06

raulchen reviewed Jun 15, 2020

View reviewed changes

ffbin force-pushed the dev_gcs_server_re_schedule branch from 86689f2 to 90d0919 Compare June 18, 2020 07:38

raulchen reviewed Jun 18, 2020

View reviewed changes

ffbin changed the title ~~[WIP]Support gcs server fault tolerance~~ [WIP]GCS server error handling for actor creation Jun 19, 2020

ffbin force-pushed the dev_gcs_server_re_schedule branch from 90d0919 to 8eeff6f Compare June 19, 2020 09:52

ffbin force-pushed the dev_gcs_server_re_schedule branch from 8eeff6f to 09a936a Compare June 22, 2020 03:49

ffbin changed the title ~~[WIP]GCS server error handling for actor creation~~ GCS server error handling for actor creation Jun 22, 2020

灵洵 added 8 commits June 28, 2020 17:54

add debug code

c7e01d9

add debug code

ed95ade

fix review comments

b72e8f8

fix gcs_actor_manager_test bug

7771529

fix review comments

af8c828

fix ut bug

54b39a8

fix ut bug

d9616ca

add testcase

7a5abeb

raulchen reviewed Jun 30, 2020

View reviewed changes

fix review comments

8f86e29

ffbin force-pushed the dev_gcs_server_re_schedule branch from d2169c6 to 8f86e29 Compare June 30, 2020 08:03

wumuzi520 reviewed Jul 1, 2020

View reviewed changes

灵洵 added 2 commits July 1, 2020 17:08

fix spill back node dead bug

a648596

fix python ut

99fac1d

add spill back an invalid node testcase

81b0126

raulchen reviewed Jul 2, 2020

View reviewed changes

fix review comments

47818ee

rkooo567 mentioned this pull request Jul 2, 2020

GCS Actor Management On Failing Tests #9266

Closed

raulchen approved these changes Jul 2, 2020

View reviewed changes

raulchen merged commit 8fcfcc4 into ray-project:master Jul 2, 2020

raulchen deleted the dev_gcs_server_re_schedule branch July 2, 2020 08:27

raulchen mentioned this pull request Jul 12, 2020

[GCS] Fix the bug about raylet receiving duplicate actor creation tasks #9422

Merged

6 tasks

rkooo567 added this to the Turn On GCS Actor Management By Default milestone Jul 18, 2020

israbbani mentioned this pull request Jan 9, 2026

[core] (1/n) [Removing GCS Centralized Scheduling] Removing the feature flag, GCS, and Raylet code. #59979

Merged

	def test_gcs_server_re_schedule_actor():
	def test_gcs_server_restart_during_actor_creation():

	RAY_LOG(DEBUG) << "Execute task, task info = " << task_spec.DebugString();
	RAY_LOG(DEBUG) << "Executing task, task info = " << task_spec.DebugString();

	// If GCS server is restarted after sending create actor task to the core worker, the
	// If GCS server is restarted after sending an actor creation task to this core worker, the

	auto on_done = [this]() {
	auto actor_manager_load_initial_data_callback = [this]() {

	local_queues_.RemoveTasks(task_ids);
	local_queues_.RemoveTasks({task_id});

	// the restarted GCS server will send the same create actor task to the core worker
	// the restarted GCS server will send the same actor creation task to the core worker

	<< " owners a leased worker. Create actor directly on worker.";
	<< " is already tied to a leased worker. Create actor directly on worker.";

Conversation

ffbin commented Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

AmplabJenkins commented Jun 11, 2020

Uh oh!

AmplabJenkins commented Jun 11, 2020

Uh oh!

AmplabJenkins commented Jun 11, 2020

Uh oh!

AmplabJenkins commented Jun 12, 2020

Uh oh!

AmplabJenkins commented Jun 12, 2020

Uh oh!

AmplabJenkins commented Jun 12, 2020

Uh oh!

AmplabJenkins commented Jun 12, 2020

Uh oh!

AmplabJenkins commented Jun 12, 2020

Uh oh!

AmplabJenkins commented Jun 12, 2020

Uh oh!

AmplabJenkins commented Jun 12, 2020

Uh oh!

AmplabJenkins commented Jun 12, 2020

Uh oh!

AmplabJenkins commented Jun 12, 2020

Uh oh!

AmplabJenkins commented Jun 12, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jun 16, 2020

Uh oh!

AmplabJenkins commented Jun 18, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

ffbin commented Jun 11, 2020 •

edited

Loading