Why are these changes needed?

Here's one example of this test failing -- https://buildkite.com/ray-project/postmerge/builds/10876#01977721-a652-4d5d-be35-1c0ec445f459/177-1290

The test expects the task error message to contain "Socket closed", but gets different error messages depending on timing:

• Expected: "Socket closed"
• Actual: "RpcError: RPC Error message: recvmsg:Connection reset by peer; RPC Error details: rpc_code: 14"

From the logs, this probably happens because:

1. The worker exit information propagates through the raylet --> GCS --> driver path
2. The task failure is detected through the direct RPC connection failure
3. These two information flows have different latencies and can arrive in different orders

Fix

Removed error message assertion from the task cancellation verification in test_worker_exit_intended_user_exit. This eliminates flakiness without compromising test coverage. Maintains essential verification.

The test still validates:

• Worker exit type: INTENDED_USER_EXIT
• Exit detail contains: "ray.cancel"
• Task failure type: WORKER_DIED

Failure Log Analysis

Cancellation initiated

[2025-06-17 00:22:43,508 I 7865 7865] normal_task_submitter.cc:705: Cancelling a task: 71b133a11e1c461cffffffffffffffffffffffff01000000 force_kill: 1 recursive: 1

• Driver calls ray.cancel(t, force=True)
• NormalTaskSubmitter::CancelTask sends RPC with force_kill=1

Worker Receives Cancellation (Worker Side)

[2025-06-17 00:22:43,508 I 9326 9510] core_worker.cc:4453: Cancel a normal task task_id=71b133a11e1c461cffffffffffffffffffffffff01000000
[2025-06-17 00:22:43,508 W 9326 9510] core_worker.cc:1262: Force exit the process. Details: The worker exits because the task cancel-f has received a force ray.cancel request.

• Worker's CoreWorker::HandleCancelTask processes the request
• Since force_kill=true, it immediately calls ExitWorker()

Worker Clean Shutdown

[2025-06-17 00:22:43,511 I 9326 9510] raylet_client.cc:73: RayletClient::Disconnect, exit_type=INTENDED_USER_EXIT, exit_detail=The worker exits because the task cancel-f has received a force ray.cancel request.

• Worker cleanly disconnects from raylet with proper exit information
• Exit type is correctly set to INTENDED_USER_EXIT

Raylet Can't Find Failure Cause

[2025-06-17 00:22:43,521 I 9263 9263] (raylet) node_manager.cc:635: didn't find failure cause for task 986822d6e5815953e511f19e1e76e9bf47154b8101000000

[2025-06-17 00:22:43,526 W 7865 9324] normal_task_submitter.cc:660: Task failure cause for task 71b133a11e1c461cffffffffffffffffffffffff01000000: Error type 0 exception string  fail immedediately: 0
[2025-06-17 00:22:43,526 W 7865 9324] task_manager.cc:1119: Task attempt 71b133a11e1c461cffffffffffffffffffffffff01000000 failed with error WORKER_DIED Fail immediately? 0, status RpcError: RPC Error message: recvmsg:Connection reset by peer; RPC Error details:  rpc_code: 14

• Driver's task submitter detects the broken connection
• Reports it as WORKER_DIED with RpcError: recvmsg:Connection reset by peer; rpc_code: 14