Skip to content

Flush tasks from lineage cache much more quickly. #4359

@robertnishihara

Description

@robertnishihara

One of the issues in #4256 appears to be that the lineage cache grows larger and larger. This causes more and more data to be sent between nodes when tasks are forwarded and the raylets receiving tasks have to do more processing.

Currently tasks are flushed from the lineage cache when they are assigned to a worker (by the raylet that owns the worker). Instead, we should try to flush the tasks as soon as the task is first submitted to the raylet.

One change that will be required is that now whoever receives a forwarded task will have to listen for GCS notifications before it can evict the task from the lineage cache. Before, whoever forwarded the task had to listen.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions