Skip to content

[Core] Instance Preemption Causing Application Level Broken Pipe Issue  #48628

@MengjinYan

Description

@MengjinYan

What happened + What you expected to happen

In a recent investigation, we found that when a task writes message to the local object store at the same time the local object store is being shut down, the Broken pipe error thrown from the write message operation will fail the current task as an application level error. This caused the task to fail without retry.

The expected behavior should be the Broken pipe error should be thrown as a system error so that the task can further retry as configured.

Versions / Dependencies

N/A

Reproduction script

N/A

Issue Severity

None

Metadata

Metadata

Assignees

Labels

P1Issue that should be fixed within a few weeksbugSomething that is supposed to be working; but isn'tcommunity-backlogcoreIssues that should be addressed in Ray Core

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions