[Data][Docs](draft): Document job-level checkpointing by anonihunter · Pull Request #60289 · ray-project/ray

anonihunter · 2026-01-19T08:59:13Z

This draft PR documents Ray Data job-level checkpointing that was added in #59409.

The documentation adds a short explanation of job-level checkpointing and how it
applies to offline batch inference, with a small configuration example.

Sections touched:

Execution Configurations
End-to-end: Offline Batch Inference

Feedback welcome.

Fixes #60250

gemini-code-assist

Code Review

This pull request adds documentation for the job-level checkpointing feature in Ray Data. The changes include adding a new section to the batch inference guide and a more detailed section with a code example to the execution configurations guide. The documentation is clear and well-written. I have one suggestion to improve the code example to better reflect best practices for checkpoint storage in a distributed environment.

gemini-code-assist · 2026-01-19T09:00:29Z

doc/source/data/execution-configurations.rst

+        checkpoint_path="/tmp/ray_data_checkpoint",
+        delete_checkpoint_on_success=False,


The example for checkpoint_path uses /tmp/ray_data_checkpoint, which can be misleading. Checkpoints must be stored in a location accessible by all nodes in the cluster, such as a cloud storage path (e.g., S3) or a network file system (NFS). Using /tmp will only work on a single-node cluster and is not a good practice for production workloads. I suggest updating the example to use a more realistic path and adding comments to clarify the requirements for checkpoint_path and the purpose of delete_checkpoint_on_success.

Suggested change

checkpoint_path="/tmp/ray_data_checkpoint",

delete_checkpoint_on_success=False,

# The checkpoint path must be accessible from all nodes in the cluster

# (e.g., a cloud storage path).

checkpoint_path="s3://my-bucket/ray_data_checkpoints",

# Keep checkpoint files after job success for inspection.

delete_checkpoint_on_success=False,

…ointing Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com>

bveeramani

ty for the contribution!

bveeramani · 2026-01-27T00:06:38Z

doc/source/data/batch_inference.rst

+Job-level checkpointing can be used to make offline batch inference jobs
+resilient to failures such as node restarts or transient execution errors.


Active voice

Suggested change

Job-level checkpointing can be used to make offline batch inference jobs

resilient to failures such as node restarts or transient execution errors.

Use job-level checkpointing to make offline batch inference jobs resilient to failures

like node restarts or transient execution errors.

bveeramani · 2026-01-27T00:07:42Z

doc/source/data/batch_inference.rst

+When enabled, Ray Data records progress during execution. If a batch inference
+job fails partway through processing, rerunning the same pipeline with the same
+checkpoint configuration will resume from the last completed checkpoint instead
+of reprocessing all records.


Suggested change

When enabled, Ray Data records progress during execution. If a batch inference

job fails partway through processing, rerunning the same pipeline with the same

checkpoint configuration will resume from the last completed checkpoint instead

of reprocessing all records.

When enabled, Ray Data records progress during execution. If a batch inference

job fails partway through processing, rerunning the same pipeline with the same

checkpoint configuration resumes from the last completed checkpoint instead

of reprocessing all records.

bveeramani · 2026-01-27T00:08:27Z

doc/source/data/execution-configurations.rst

+Job-level checkpointing is configured through the
+:class:`~ray.data.checkpoint.CheckpointConfig` and is set on the current
+:class:`~ray.data.DataContext`.


Active voice

Suggested change

Job-level checkpointing is configured through the

:class:`~ray.data.checkpoint.CheckpointConfig` and is set on the current

:class:`~ray.data.DataContext`.

To configure job-level checkpointing, specify a

:class:`~ray.data.checkpoint.CheckpointConfig` on the current

:class:`~ray.data.DataContext`.

bveeramani · 2026-01-27T00:09:20Z

doc/source/data/execution-configurations.rst

+.. The checkpoint path must be accessible by all nodes in the Ray cluster.
+.. delete_checkpoint_on_success=False preserves checkpoints after successful runs.


Is this intended to be visible to the reader?

bveeramani · 2026-01-27T00:10:14Z

doc/source/data/execution-configurations.rst

+Ray Data supports job-level checkpointing to improve fault tolerance for
+long-running batch pipelines. When enabled, Ray Data can resume a failed job
+from the last successfully processed records instead of restarting from the
+beginning.


Here and elsewhere -- for consistency, could we use the term "rows" instead of "records"? I think we typically use "row" in the Ray Data documentation

Suggested change

Ray Data supports job-level checkpointing to improve fault tolerance for

long-running batch pipelines. When enabled, Ray Data can resume a failed job

from the last successfully processed records instead of restarting from the

beginning.

Ray Data supports job-level checkpointing to improve fault tolerance for

long-running batch pipelines. When enabled, Ray Data can resume a failed job

from the last successfully processed rows instead of restarting from the

beginning.

github-actions · 2026-02-10T00:58:27Z

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

bveeramani · 2026-02-10T07:57:41Z

Hey @anonihunter , this PR has been stale for a while, so I've asked @yuhuan130 to take this over the finish line.

When I merge the PR, I'll include you as a co-author so you get credit

anonihunter · 2026-02-10T09:05:30Z

Thanks for the update, really appreciate you taking this forward 🙌
I wasn’t at my working desk for a bit, so I couldn’t quickly apply the requested tweaks.
Happy to be included as a co-author — thanks for that!

This PR documents Ray Data job-level checkpointing functionality added in #59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes #60289 Fixes #60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com>

…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com> Signed-off-by: Ondrej Prenek <ondra.prenek@gmail.com>

…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com>

…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com> Signed-off-by: Muhammad Saif <2024BBIT200@student.Uet.edu.pk>

…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com>

…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com> Signed-off-by: Adel Nour <ans9868@nyu.edu>

…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com>

…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>

…0921) This PR documents Ray Data job-level checkpointing functionality added in ray-project#59409. Adds documentation explaining job-level checkpointing and its application to offline batch inference, including configuration examples. **Sections modified:** - Execution Configurations - End-to-end: Offline Batch Inference Supersedes ray-project#60289 Fixes ray-project#60250 --------- Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com> Signed-off-by: “Alex <alexchien130@gmail.com> Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: Abhishek Kumar <anonyomoushunter@gmail.com> Co-authored-by: Balaji Veeramani <bveeramani@berkeley.edu> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Balaji Veeramani <balaji@anyscale.com>

anonihunter requested a review from a team as a code owner January 19, 2026 08:59

gemini-code-assist bot reviewed Jan 19, 2026

View reviewed changes

anonihunter force-pushed the docs-job-level-checkpointing branch from 07dcbb5 to 26d3139 Compare January 19, 2026 09:09

docs(data): clarify checkpoint_path requirements for job-level checkp…

0e4b599

…ointing Signed-off-by: Abhishek Kumar <anonyomoushunter@gmail.com>

anonihunter force-pushed the docs-job-level-checkpointing branch from 26d3139 to 0e4b599 Compare January 19, 2026 09:58

ray-gardener bot added docs An issue or change related to documentation data Ray Data-related issues community-contribution Contributed by the community labels Jan 19, 2026

anonihunter mentioned this pull request Jan 19, 2026

[Data] Add documentation for Ray Data checkpointing #60250

Closed

bveeramani reviewed Jan 27, 2026

View reviewed changes

bveeramani changed the title ~~docs(draft): document job-level checkpointing~~ [Data][Docs](draft): Document job-level checkpointing Jan 27, 2026

github-actions bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Feb 10, 2026

bveeramani closed this Feb 10, 2026

yuhuan130 mentioned this pull request Feb 10, 2026

[Data][Docs] Add job-level checkpointing documentation #60921

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Data][Docs](draft): Document job-level checkpointing#60289

[Data][Docs](draft): Document job-level checkpointing#60289
anonihunter wants to merge 1 commit intoray-project:masterfrom
anonihunter:docs-job-level-checkpointing

anonihunter commented Jan 19, 2026 •

edited by bveeramani

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jan 19, 2026

Uh oh!

bveeramani left a comment

Uh oh!

bveeramani Jan 27, 2026

Uh oh!

bveeramani Jan 27, 2026

Uh oh!

bveeramani Jan 27, 2026

Uh oh!

bveeramani Jan 27, 2026

Uh oh!

bveeramani Jan 27, 2026

Uh oh!

github-actions bot commented Feb 10, 2026

Uh oh!

bveeramani commented Feb 10, 2026

Uh oh!

anonihunter commented Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		checkpoint_path="/tmp/ray_data_checkpoint",
		delete_checkpoint_on_success=False,

-        checkpoint_path="/tmp/ray_data_checkpoint",
-        delete_checkpoint_on_success=False,
+        # The checkpoint path must be accessible from all nodes in the cluster
+        # (e.g., a cloud storage path).
+        checkpoint_path="s3://my-bucket/ray_data_checkpoints",
+        # Keep checkpoint files after job success for inspection.
+        delete_checkpoint_on_success=False,

		Job-level checkpointing can be used to make offline batch inference jobs
		resilient to failures such as node restarts or transient execution errors.

		.. The checkpoint path must be accessible by all nodes in the Ray cluster.
		.. delete_checkpoint_on_success=False preserves checkpoints after successful runs.

Conversation

anonihunter commented Jan 19, 2026 • edited by bveeramani Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

bveeramani left a comment

Choose a reason for hiding this comment

Uh oh!

bveeramani Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

bveeramani Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

bveeramani Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

bveeramani Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

bveeramani Jan 27, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 10, 2026

Uh oh!

bveeramani commented Feb 10, 2026

Uh oh!

anonihunter commented Feb 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

anonihunter commented Jan 19, 2026 •

edited by bveeramani

Loading