-
Notifications
You must be signed in to change notification settings - Fork 7.4k
Closed
Description
One of the issues in #4256 appears to be that the lineage cache grows larger and larger. This causes more and more data to be sent between nodes when tasks are forwarded and the raylets receiving tasks have to do more processing.
Currently tasks are flushed from the lineage cache when they are assigned to a worker (by the raylet that owns the worker). Instead, we should try to flush the tasks as soon as the task is first submitted to the raylet.
One change that will be required is that now whoever receives a forwarded task will have to listen for GCS notifications before it can evict the task from the lineage cache. Before, whoever forwarded the task had to listen.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels