Add DataContext + LogicalOp Args to Dataset Export#53554
Merged
raulchen merged 15 commits intoray-project:masterfrom Jun 9, 2025
Merged
Conversation
87694b0 to
6a6455f
Compare
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
6a6455f to
8fb69eb
Compare
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
eac92c1 to
384d5fb
Compare
iamjustinhsu
commented
Jun 4, 2025
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
68610f1 to
fb7131a
Compare
85999e9 to
0b4b974
Compare
…u/add-ctx-and-logical-ops-to-dataset-metadata-exporter
20d1bf5 to
c12a2c9
Compare
raulchen
reviewed
Jun 5, 2025
raulchen
reviewed
Jun 6, 2025
7db0d59 to
ec730b9
Compare
a9bbd8e to
df0af64
Compare
iamjustinhsu
commented
Jun 6, 2025
Comment on lines
149
to
156
Contributor
Author
There was a problem hiding this comment.
this is where we catch -all
df0af64 to
778c8f4
Compare
778c8f4 to
89549c8
Compare
Signed-off-by: iamjustinhsu <jhsu@anyscale.com>
89549c8 to
5f6bbed
Compare
raulchen
reviewed
Jun 6, 2025
|
|
||
| def _get_args(self) -> Dict[str, Any]: | ||
| """This Dict must be serializable""" | ||
| return {k: sanitize_for_struct(v) for k, v in vars(self).items()} |
Contributor
There was a problem hiding this comment.
Previously sanitize_for_struct was called on the metadata exporter.
And that looks cleaner than putting it here.
Was there any issues?
Contributor
Author
There was a problem hiding this comment.
yes, the issue was sometimes the args are not pickable, so i was encountering this error
!!! FAIL serialization: cannot pickle '_thread.lock' object
This is because we call register_dataset_for_stats_actor, which rpc serializes and fails. honestly i could case handle this, but didn't want to risk
| data_context: DataContext | ||
|
|
||
|
|
||
| def sanitize_for_struct(obj): |
Contributor
There was a problem hiding this comment.
actually should we just json.dumps the entire obj?
this function seems to be just reinventing json.dumps.
Contributor
Author
There was a problem hiding this comment.
that sounds smart, im gonna try that Oh wait, the reason why is because if there is a deeply nested json, and one argument is not serialzable, then it fails the entire thing. The sanitize_for_struct gives a best guess estimate of the obj itself. lmk if that doesn't make sense
raulchen
approved these changes
Jun 6, 2025
Contributor
raulchen
left a comment
There was a problem hiding this comment.
LGTM. a few final comments.
80ffca0 to
2e29c7a
Compare
bveeramani
pushed a commit
that referenced
this pull request
Nov 20, 2025
… op args when refreshing metadata (#58755) ## Description We export dataset and operator metadata whenever there is a state change. However, the size of the export file can be very large because the metadata also includes the DataContext config and operator args, which does not change over time, and they will be written multiple times to the file. To reduce the file size and remove redundant info, we can only export DataContext and operator args when the dataset is [registered](https://github.com/ray-project/ray/blob/d1cce8c9dc8411fad7cfbd619350bec6f19839a3/python/ray/data/_internal/stats.py#L621), and avoid them in the later state updates. ## Related issues Related previous PRs: [55355](#55355), [53554](#53554) Signed-off-by: cong.qian <cong.qian@anyscale.com>
400Ping
pushed a commit
to 400Ping/ray
that referenced
this pull request
Nov 21, 2025
… op args when refreshing metadata (ray-project#58755) ## Description We export dataset and operator metadata whenever there is a state change. However, the size of the export file can be very large because the metadata also includes the DataContext config and operator args, which does not change over time, and they will be written multiple times to the file. To reduce the file size and remove redundant info, we can only export DataContext and operator args when the dataset is [registered](https://github.com/ray-project/ray/blob/d1cce8c9dc8411fad7cfbd619350bec6f19839a3/python/ray/data/_internal/stats.py#L621), and avoid them in the later state updates. ## Related issues Related previous PRs: [55355](ray-project#55355), [53554](ray-project#53554) Signed-off-by: cong.qian <cong.qian@anyscale.com>
ykdojo
pushed a commit
to ykdojo/ray
that referenced
this pull request
Nov 27, 2025
… op args when refreshing metadata (ray-project#58755) ## Description We export dataset and operator metadata whenever there is a state change. However, the size of the export file can be very large because the metadata also includes the DataContext config and operator args, which does not change over time, and they will be written multiple times to the file. To reduce the file size and remove redundant info, we can only export DataContext and operator args when the dataset is [registered](https://github.com/ray-project/ray/blob/d1cce8c9dc8411fad7cfbd619350bec6f19839a3/python/ray/data/_internal/stats.py#L621), and avoid them in the later state updates. ## Related issues Related previous PRs: [55355](ray-project#55355), [53554](ray-project#53554) Signed-off-by: cong.qian <cong.qian@anyscale.com> Signed-off-by: YK <1811651+ykdojo@users.noreply.github.com>
SheldonTsen
pushed a commit
to SheldonTsen/ray
that referenced
this pull request
Dec 1, 2025
… op args when refreshing metadata (ray-project#58755) ## Description We export dataset and operator metadata whenever there is a state change. However, the size of the export file can be very large because the metadata also includes the DataContext config and operator args, which does not change over time, and they will be written multiple times to the file. To reduce the file size and remove redundant info, we can only export DataContext and operator args when the dataset is [registered](https://github.com/ray-project/ray/blob/d1cce8c9dc8411fad7cfbd619350bec6f19839a3/python/ray/data/_internal/stats.py#L621), and avoid them in the later state updates. ## Related issues Related previous PRs: [55355](ray-project#55355), [53554](ray-project#53554) Signed-off-by: cong.qian <cong.qian@anyscale.com>
peterxcli
pushed a commit
to peterxcli/ray
that referenced
this pull request
Feb 25, 2026
… op args when refreshing metadata (ray-project#58755) ## Description We export dataset and operator metadata whenever there is a state change. However, the size of the export file can be very large because the metadata also includes the DataContext config and operator args, which does not change over time, and they will be written multiple times to the file. To reduce the file size and remove redundant info, we can only export DataContext and operator args when the dataset is [registered](https://github.com/ray-project/ray/blob/d1cce8c9dc8411fad7cfbd619350bec6f19839a3/python/ray/data/_internal/stats.py#L621), and avoid them in the later state updates. ## Related issues Related previous PRs: [55355](ray-project#55355), [53554](ray-project#53554) Signed-off-by: cong.qian <cong.qian@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
event_EXPORT_DATASET_METADATA.log
^ Here is an example file.
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.