[Data] Make execution callback dataflow explicit to prevent state leakage#61405
Merged
bveeramani merged 25 commits intoray-project:masterfrom Mar 25, 2026
Conversation
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Contributor
There was a problem hiding this comment.
Code Review
This pull request successfully refactors the execution callback system to make the dataflow more explicit, moving from dynamic injection to eager initialization. The changes are consistent across the modified files, with the planner now instantiating and returning the list of callbacks, which is then propagated up to the executor. The overall implementation is clean and achieves the stated goal. I have one minor suggestion to improve code clarity in plan.py.
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Contributor
Author
|
@bveeramani will need your review on this |
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
…efactor Signed-off-by: Limark Dcunha <83493294+limarkdcunha@users.noreply.github.com>
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
…efactor Signed-off-by: Limark Dcunha <83493294+limarkdcunha@users.noreply.github.com>
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
Signed-off-by: Limark Dcunha <limarkdcunha@gmail.com>
This was referenced Mar 25, 2026
bveeramani
added a commit
that referenced
this pull request
Mar 25, 2026
…62053) ## Description `planner.plan()` now returns a tuple `(physical_plan, callbacks)` after #61405. In this PR, I've updates the planner unit tests to unpack the tuple instead of indexing into the result. I'm doing this to make the return type a bit more explicit and readable. ## Related issues Related to #61405 Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
bveeramani
added a commit
that referenced
this pull request
Mar 25, 2026
## Description In this PR, I've removed old callback registration APIs (`get_execution_callbacks`, `add_execution_callback`, `remove_execution_callback`) in favor of the new `execution_callback_classes` property on `DataContext`. This is part of a larger effort to avoid storing callback instances in `DataContext` since they might be shared across dataset executions. The new implementation re-constructs the callbacks for each execution and directly passes them into the executor as a constructor argument. This makes the data flow much more explicit and easier to reason about. ## Related issues Follow-up to #61405. Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
JasonLi1909
pushed a commit
to JasonLi1909/ray
that referenced
this pull request
Mar 26, 2026
…62055) ## Description In this PR, I've removed old callback registration APIs (`get_execution_callbacks`, `add_execution_callback`, `remove_execution_callback`) in favor of the new `execution_callback_classes` property on `DataContext`. This is part of a larger effort to avoid storing callback instances in `DataContext` since they might be shared across dataset executions. The new implementation re-constructs the callbacks for each execution and directly passes them into the executor as a constructor argument. This makes the data flow much more explicit and easier to reason about. ## Related issues Follow-up to ray-project#61405. Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Larger Issue
Refactored the execution callback system from dynamic, per-job injection to static, eager initialization. DataContext now serves as a stateless registry of callback factories, enabling the StreamingExecutor to load all callbacks at startup. Each callback then self-configures or no-ops based on the runtime environment.
Context
This is Part 3 of a four-part change set intended to resolve the issue described above.
Part 1 diff: #60480
Part 2 diff: #61293
Closes #60279.