Skip to content

Object store thrashing if it runs ray.get in a non-main thread. #12906

@rkooo567

Description

@rkooo567

What is the problem?

object store thrashes when ray.get is called in a non-main thread because ray.get wouldn't unsubscribe the get dependency because the RPC to unsubscribe the get dependency is not called (NotifyDirectCallUnblock), resulting in that the give object is pulled / evicted until the get dependency is unsubscribed by the task termination.

Reproduction (REQUIRED)

Please provide a short code snippet (less than 50 lines if possible) that can be copy-pasted to reproduce the issue. The snippet should have no external library dependencies (i.e., use fake or mock data / environments):

@rkooo567 will prepare the simple reproduction script

  • I have verified my script runs in a clean environment and reproduces the issue.
  • I have verified the issue also occurs with the latest wheels.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Issue moderate in impact or severitybugSomething that is supposed to be working; but isn'tcoreIssues that should be addressed in Ray Corepending-cleanupThis issue is pending cleanup. It will be removed in 2 weeks after being assigned.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions