[data] New executor backend [1/n]--- Add basic interfaces#31216
[data] New executor backend [1/n]--- Add basic interfaces#31216ericl merged 6 commits intoray-project:masterfrom
Conversation
Signed-off-by: Eric Liang <ekhliang@gmail.com>
c21
left a comment
There was a problem hiding this comment.
LGTM, cc @clarkzinzow as well.
clarkzinzow
left a comment
There was a problem hiding this comment.
Apologies for not getting this review in before the merge, mostly nits about type annotations and docstrings.
| input. For most operators, this is always `0` since there is only | ||
| one upstream input operator. | ||
| """ | ||
| raise NotImplementedError |
There was a problem hiding this comment.
How do we plan to propagate whether we need to preserve order down to operators at execution time?
Right now, the MapOperator in the BulkExecutor PR has hardcoded order preservation into its execution state, without a convenient PhysicalOperator API to change that behavior at execution time (not instantiation time). It looks like we need a PhysicalOperator API for registering/binding execution options, maybe an API that's symmetrical to PhysicalOperator.shutdown().
There was a problem hiding this comment.
Probably it will be added to execute()... maybe we should just omit it for now.
ericl
left a comment
There was a problem hiding this comment.
Thanks--- comments incorporated into the main PR.
This PR adds the basic interfaces and feature flags; split out from https://github.com/ray-project/ray/pull/30903/files See REP ray-project/enhancements#18 for more details.
…t#31216) This PR adds the basic interfaces and feature flags; split out from https://github.com/ray-project/ray/pull/30903/files See REP ray-project/enhancements#18 for more details. Signed-off-by: tmynn <hovhannes.tamoyan@gmail.com>
Why are these changes needed?
This PR adds the basic interfaces and feature flags; split out from https://github.com/ray-project/ray/pull/30903/files
See REP ray-project/enhancements#18 for more details.