[Data][Docs] Add job-level checkpointing documentation#60921
Merged
bveeramani merged 5 commits intoray-project:masterfrom Feb 11, 2026
Merged
[Data][Docs] Add job-level checkpointing documentation#60921bveeramani merged 5 commits intoray-project:masterfrom
bveeramani merged 5 commits intoray-project:masterfrom
Conversation
…ointing Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com>
Signed-off-by: “Alex <alexchien130@gmail.com>
Contributor
There was a problem hiding this comment.
Code Review
This PR adds valuable documentation for the new job-level checkpointing feature. The explanations are clear and the example is helpful. I've suggested a few minor wording improvements for clarity and proposed adding a note about an important requirement for the id_column to ensure users can use this feature correctly.
| id_column="id", | ||
| checkpoint_path="s3://my-bucket/ray-data-checkpoints", # Must be accessible by all nodes | ||
| delete_checkpoint_on_success=False, # Preserves checkpoints after successful runs | ||
| ) |
Contributor
There was a problem hiding this comment.
The id_column has a critical requirement: its values must be unique for each row, and the column must be present throughout the entire pipeline. This information is crucial for users to correctly use checkpointing. I recommend adding a .. note:: block after the example to highlight this.
For example:
.. note::
The ``id_column`` must contain unique values for each row across the entire dataset. This column must also be present throughout all transformations in the pipeline.
Member
There was a problem hiding this comment.
Sounds reasonable, want to add?
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
preneond
pushed a commit
to preneond/ray
that referenced
this pull request
Feb 15, 2026
…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com> Signed-off-by: Ondrej Prenek <ondra.prenek@gmail.com>
limarkdcunha
pushed a commit
to limarkdcunha/ray
that referenced
this pull request
Feb 17, 2026
…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com>
preneond
pushed a commit
to preneond/ray
that referenced
this pull request
Feb 17, 2026
…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com>
MuhammadSaif700
pushed a commit
to MuhammadSaif700/ray
that referenced
this pull request
Feb 17, 2026
…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com> Signed-off-by: Muhammad Saif <2024BBIT200@student.Uet.edu.pk>
Kunchd
pushed a commit
to Kunchd/ray
that referenced
this pull request
Feb 17, 2026
…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com>
ans9868
pushed a commit
to ans9868/ray
that referenced
this pull request
Feb 18, 2026
…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com> Signed-off-by: Adel Nour <ans9868@nyu.edu>
Aydin-ab
pushed a commit
to kunling-anyscale/ray
that referenced
this pull request
Feb 20, 2026
…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com>
peterxcli
pushed a commit
to peterxcli/ray
that referenced
this pull request
Feb 25, 2026
…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>
peterxcli
pushed a commit
to peterxcli/ray
that referenced
this pull request
Feb 25, 2026
…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>
preneond
pushed a commit
to preneond/ray
that referenced
this pull request
Mar 23, 2026
…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR documents Ray Data job-level checkpointing functionality added in #59409.
Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples.
Sections modified:
Supersedes #60289
Fixes #60250