Skip to content

[data] [streaming] Limit operator shouldn't materialize stream #34234

@ericl

Description

@ericl

The following shouldn't materialize all previous blocks:

ray.data.range(1000, parallelism=1000).map(lambda x: x).limit(1).materialize()

This can be implemented with a special Limit operator that terminates once it receives enough elements.

Metadata

Metadata

Assignees

Labels

P1Issue that should be fixed within a few weeksbugSomething that is supposed to be working; but isn't

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions