[train][doc] Add validation and other details to checkpoint doc by TimothySeah · Pull Request #57065 · ray-project/ray

TimothySeah · 2025-10-01T00:01:28Z

Summary

Add validation to checkpoint doc. Also mention that async checkpoint uploading requires you to create a long-lasting checkpoint directory.

Testing

docbuild

Note

Documents checkpoint upload modes and async validation with new examples and API refs, adds s3torchconnector deps, and updates CI conda install to use the classic solver.

Docs (Train):
- Checkpoints guide: Rename to "Saving, Validating, and Loading Checkpoints"; add sections on CheckpointUploadMode (sync/async/custom) and async checkpoint validation with validate_fn; include new examples and figure.
- API refs: Add train.CheckpointUploadMode to doc/source/train/api/api.rst.
- Doc examples: Add checkpoints.py snippets for CheckpointUploadMode.SYNC/ASYNC/NO_UPLOAD, Torch/XLA validation via TorchTrainer, Ray Data map_batches, and reporting with validate_function.
- Monitoring & logging: Note validating checkpoints as a primary reporting use case.
Dependencies:
- Add s3torchconnector==1.4.3 (and transitively s3torchconnectorclient==1.4.3) to python/requirements_* for custom S3 uploads.
CI:
- Include ci/env/install-miniforge.sh in ci/docker/base.ml.wanda.yaml build context.
- In ci/env/install-miniforge.sh, install mpi4py with conda ... --solver classic for Python <3.12.

^{Written by Cursor Bugbot for commit c5741d9. This will update automatically on new commits. Configure here.}

gemini-code-assist

Code Review

This pull request significantly enhances the documentation for Ray Train checkpoints by adding details on validation and various upload modes. The new code examples are valuable, but I've identified critical issues in the asynchronous checkpointing examples concerning the lifecycle of temporary directories, which could lead to runtime failures. I've also pointed out a couple of inaccuracies in the documentation text. My feedback includes suggestions to correct these issues to ensure the examples are robust and the documentation is accurate.

doc/source/train/doc_code/checkpoints.py

doc/source/train/user-guides/checkpoints.rst

aslonnie

@elliot-barn to review the dependency change.

python/requirements/ml/train-test-requirements.txt

Signed-off-by: Timothy Seah <tseah@anyscale.com>

doc/source/train/user-guides/checkpoints.rst

Signed-off-by: Timothy Seah <tseah@anyscale.com>

justinvyu

Thanks!

Questions to answer in the user guide:

How do I set resources on the validation group?
- Emphasize that we can set resources differently on the validation group to leverage a heterogeneous cluster. Ex: show that you can use a different GPU types (ex: A100 for training and A10G for inference, which has lower GRAM requirement). I think we should also show a best practice to schedule even CPU tasks onto different nodes to prevent validation from interfering with training.
What's the lifecycle of "metrics" attached to the checkpoint? Partial metrics -> validation -> Full metrics
How can I access my validation results at the end of training?
Followup: How can I report my validation results to wandb?

doc/source/train/user-guides/checkpoints.rst

…do to request gpu Signed-off-by: Timothy Seah <tseah@anyscale.com>

Signed-off-by: Timothy Seah <tseah@anyscale.com>

TimothySeah · 2025-10-09T02:27:39Z

Thanks!

Questions to answer in the user guide:

How do I set resources on the validation group?

Emphasize that we can set resources differently on the validation group to leverage a heterogeneous cluster. Ex: show that you can use a different GPU types (ex: A100 for training and A10G for inference, which has lower GRAM requirement). I think we should also show a best practice to schedule even CPU tasks onto different nodes to prevent validation from interfering with training.

What's the lifecycle of "metrics" attached to the checkpoint? Partial metrics -> validation -> Full metrics

How can I access my validation results at the end of training?

Followup: How can I report my validation results to wandb?

Done. A few comments:

Looks like https://docs.ray.io/en/latest/data/api/doc/ray.data.Dataset.map_batches.html cannot request specific gpu types. CC @ray-project/ray-data .
Once I modify get_all_reported_checkpoints to work with async validation, I will add it to the checkpoint metrics lifecycle section.
Will document experiment tracking in a followup PR since that might take a few steps
I think we should also show a best practice to schedule even CPU tasks onto different nodes to prevent validation from interfering with training. Discussed offline - will do this in a followup.

Signed-off-by: Timothy Seah <tseah@anyscale.com>

doc/source/train/doc_code/asynchronous_validation.py

Signed-off-by: Timothy Seah <tseah@anyscale.com>

doc/source/train/doc_code/asynchronous_validation.py

Signed-off-by: Timothy Seah <tseah@anyscale.com>

…checkpoints

doc/source/train/images/checkpoint_metrics_lifecycle.png

doc/source/train/user-guides/asynchronous-validation.rst

doc/source/train/doc_code/asynchronous_validation.py

doc/source/train/user-guides/asynchronous-validation.rst

Signed-off-by: Timothy Seah <tseah@anyscale.com>

angelinalg

Just some style nits.
Also, in the diagram, there should be a comma after the latin abbreviation i.e..

doc/source/train/user-guides/asynchronous-validation.rst

doc/source/train/user-guides/checkpoints.rst

doc/source/train/user-guides/monitoring-logging.rst

doc/source/train/user-guides/asynchronous-validation.rst

justinvyu · 2025-10-15T21:07:30Z

doc/source/train/user-guides/asynchronous-validation.rst

+.. literalinclude:: ../doc_code/asynchronous_validation.py
+    :language: python
+    :start-after: __validate_fn_simple_start__
+    :end-before: __validate_fn_simple_end__


This example is not too practical because we don't allow the user to specify to run this validate_fn as a GPU task. So they can only do CPU inference here. What about just keeping this as a skeleton function and point the user to the section below on how to write a distributed validate function.=

Done - @matthewdeng FYI.

doc/source/train/user-guides/asynchronous-validation.rst

doc/source/train/images/checkpoint_metrics_lifecycle.png

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Timothy Seah <tseah@anyscale.com>

Signed-off-by: Timothy Seah <tseah@anyscale.com>

TimothySeah · 2025-10-15T23:19:28Z

Just some style nits. Also, in the diagram, there should be a comma after the latin abbreviation i.e..

Done. Ended up using : {} at @justinvyu 's suggestion instead of .e.g.

…project#57065) # Summary Add validation to checkpoint doc. Also mention that async checkpoint uploading requires you to create a long-lasting checkpoint directory. # Testing docbuild  --- > [!NOTE] > Documents checkpoint upload modes and async validation with new examples and API refs, adds s3torchconnector deps, and updates CI conda install to use the classic solver. > > - **Docs (Train)**: > - **Checkpoints guide**: Rename to "Saving, Validating, and Loading Checkpoints"; add sections on `CheckpointUploadMode` (sync/async/custom) and async checkpoint validation with `validate_fn`; include new examples and figure. > - **API refs**: Add `train.CheckpointUploadMode` to `doc/source/train/api/api.rst`. > - **Doc examples**: Add `checkpoints.py` snippets for `CheckpointUploadMode.SYNC/ASYNC/NO_UPLOAD`, Torch/XLA validation via `TorchTrainer`, Ray Data `map_batches`, and reporting with `validate_function`. > - **Monitoring & logging**: Note validating checkpoints as a primary reporting use case. > - **Dependencies**: > - Add `s3torchconnector==1.4.3` (and transitively `s3torchconnectorclient==1.4.3`) to `python/requirements_*` for custom S3 uploads. > - **CI**: > - Include `ci/env/install-miniforge.sh` in `ci/docker/base.ml.wanda.yaml` build context. > - In `ci/env/install-miniforge.sh`, install `mpi4py` with `conda ... --solver classic` for Python <3.12. > > Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit c5741d9. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).  --------- Signed-off-by: Timothy Seah <tseah@anyscale.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>

…project#57065) # Summary Add validation to checkpoint doc. Also mention that async checkpoint uploading requires you to create a long-lasting checkpoint directory. # Testing docbuild  --- > [!NOTE] > Documents checkpoint upload modes and async validation with new examples and API refs, adds s3torchconnector deps, and updates CI conda install to use the classic solver. > > - **Docs (Train)**: > - **Checkpoints guide**: Rename to "Saving, Validating, and Loading Checkpoints"; add sections on `CheckpointUploadMode` (sync/async/custom) and async checkpoint validation with `validate_fn`; include new examples and figure. > - **API refs**: Add `train.CheckpointUploadMode` to `doc/source/train/api/api.rst`. > - **Doc examples**: Add `checkpoints.py` snippets for `CheckpointUploadMode.SYNC/ASYNC/NO_UPLOAD`, Torch/XLA validation via `TorchTrainer`, Ray Data `map_batches`, and reporting with `validate_function`. > - **Monitoring & logging**: Note validating checkpoints as a primary reporting use case. > - **Dependencies**: > - Add `s3torchconnector==1.4.3` (and transitively `s3torchconnectorclient==1.4.3`) to `python/requirements_*` for custom S3 uploads. > - **CI**: > - Include `ci/env/install-miniforge.sh` in `ci/docker/base.ml.wanda.yaml` build context. > - In `ci/env/install-miniforge.sh`, install `mpi4py` with `conda ... --solver classic` for Python <3.12. > > Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit c5741d9. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).  --------- Signed-off-by: Timothy Seah <tseah@anyscale.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: xgui <xgui@anyscale.com>

# Summary Add validation to checkpoint doc. Also mention that async checkpoint uploading requires you to create a long-lasting checkpoint directory. # Testing docbuild  --- > [!NOTE] > Documents checkpoint upload modes and async validation with new examples and API refs, adds s3torchconnector deps, and updates CI conda install to use the classic solver. > > - **Docs (Train)**: > - **Checkpoints guide**: Rename to "Saving, Validating, and Loading Checkpoints"; add sections on `CheckpointUploadMode` (sync/async/custom) and async checkpoint validation with `validate_fn`; include new examples and figure. > - **API refs**: Add `train.CheckpointUploadMode` to `doc/source/train/api/api.rst`. > - **Doc examples**: Add `checkpoints.py` snippets for `CheckpointUploadMode.SYNC/ASYNC/NO_UPLOAD`, Torch/XLA validation via `TorchTrainer`, Ray Data `map_batches`, and reporting with `validate_function`. > - **Monitoring & logging**: Note validating checkpoints as a primary reporting use case. > - **Dependencies**: > - Add `s3torchconnector==1.4.3` (and transitively `s3torchconnectorclient==1.4.3`) to `python/requirements_*` for custom S3 uploads. > - **CI**: > - Include `ci/env/install-miniforge.sh` in `ci/docker/base.ml.wanda.yaml` build context. > - In `ci/env/install-miniforge.sh`, install `mpi4py` with `conda ... --solver classic` for Python <3.12. > > Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit c5741d9. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).  --------- Signed-off-by: Timothy Seah <tseah@anyscale.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…project#57065) # Summary Add validation to checkpoint doc. Also mention that async checkpoint uploading requires you to create a long-lasting checkpoint directory. # Testing docbuild  --- > [!NOTE] > Documents checkpoint upload modes and async validation with new examples and API refs, adds s3torchconnector deps, and updates CI conda install to use the classic solver. > > - **Docs (Train)**: > - **Checkpoints guide**: Rename to "Saving, Validating, and Loading Checkpoints"; add sections on `CheckpointUploadMode` (sync/async/custom) and async checkpoint validation with `validate_fn`; include new examples and figure. > - **API refs**: Add `train.CheckpointUploadMode` to `doc/source/train/api/api.rst`. > - **Doc examples**: Add `checkpoints.py` snippets for `CheckpointUploadMode.SYNC/ASYNC/NO_UPLOAD`, Torch/XLA validation via `TorchTrainer`, Ray Data `map_batches`, and reporting with `validate_function`. > - **Monitoring & logging**: Note validating checkpoints as a primary reporting use case. > - **Dependencies**: > - Add `s3torchconnector==1.4.3` (and transitively `s3torchconnectorclient==1.4.3`) to `python/requirements_*` for custom S3 uploads. > - **CI**: > - Include `ci/env/install-miniforge.sh` in `ci/docker/base.ml.wanda.yaml` build context. > - In `ci/env/install-miniforge.sh`, install `mpi4py` with `conda ... --solver classic` for Python <3.12. > > Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit c5741d9. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).  --------- Signed-off-by: Timothy Seah <tseah@anyscale.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>

…project#57065) # Summary Add validation to checkpoint doc. Also mention that async checkpoint uploading requires you to create a long-lasting checkpoint directory. # Testing docbuild  --- > [!NOTE] > Documents checkpoint upload modes and async validation with new examples and API refs, adds s3torchconnector deps, and updates CI conda install to use the classic solver. > > - **Docs (Train)**: > - **Checkpoints guide**: Rename to "Saving, Validating, and Loading Checkpoints"; add sections on `CheckpointUploadMode` (sync/async/custom) and async checkpoint validation with `validate_fn`; include new examples and figure. > - **API refs**: Add `train.CheckpointUploadMode` to `doc/source/train/api/api.rst`. > - **Doc examples**: Add `checkpoints.py` snippets for `CheckpointUploadMode.SYNC/ASYNC/NO_UPLOAD`, Torch/XLA validation via `TorchTrainer`, Ray Data `map_batches`, and reporting with `validate_function`. > - **Monitoring & logging**: Note validating checkpoints as a primary reporting use case. > - **Dependencies**: > - Add `s3torchconnector==1.4.3` (and transitively `s3torchconnectorclient==1.4.3`) to `python/requirements_*` for custom S3 uploads. > - **CI**: > - Include `ci/env/install-miniforge.sh` in `ci/docker/base.ml.wanda.yaml` build context. > - In `ci/env/install-miniforge.sh`, install `mpi4py` with `conda ... --solver classic` for Python <3.12. > > Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit c5741d9. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).  --------- Signed-off-by: Timothy Seah <tseah@anyscale.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…project#57065) # Summary Add validation to checkpoint doc. Also mention that async checkpoint uploading requires you to create a long-lasting checkpoint directory. # Testing docbuild  --- > [!NOTE] > Documents checkpoint upload modes and async validation with new examples and API refs, adds s3torchconnector deps, and updates CI conda install to use the classic solver. > > - **Docs (Train)**: > - **Checkpoints guide**: Rename to "Saving, Validating, and Loading Checkpoints"; add sections on `CheckpointUploadMode` (sync/async/custom) and async checkpoint validation with `validate_fn`; include new examples and figure. > - **API refs**: Add `train.CheckpointUploadMode` to `doc/source/train/api/api.rst`. > - **Doc examples**: Add `checkpoints.py` snippets for `CheckpointUploadMode.SYNC/ASYNC/NO_UPLOAD`, Torch/XLA validation via `TorchTrainer`, Ray Data `map_batches`, and reporting with `validate_function`. > - **Monitoring & logging**: Note validating checkpoints as a primary reporting use case. > - **Dependencies**: > - Add `s3torchconnector==1.4.3` (and transitively `s3torchconnectorclient==1.4.3`) to `python/requirements_*` for custom S3 uploads. > - **CI**: > - Include `ci/env/install-miniforge.sh` in `ci/docker/base.ml.wanda.yaml` build context. > - In `ci/env/install-miniforge.sh`, install `mpi4py` with `conda ... --solver classic` for Python <3.12. > > Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit c5741d9. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).  --------- Signed-off-by: Timothy Seah <tseah@anyscale.com> Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

TimothySeah requested review from a team as code owners October 1, 2025 00:01

This comment was marked as outdated.

Sign in to view

gemini-code-assist bot reviewed Oct 1, 2025

View reviewed changes

ray-gardener bot added docs An issue or change related to documentation train Ray Train Related Issue labels Oct 1, 2025

elliot-barn requested a review from a team as a code owner October 2, 2025 00:00

TimothySeah force-pushed the tseah/validating-checkpoints branch from c5741d9 to ef44dd8 Compare October 2, 2025 18:04

This comment was marked as outdated.

Sign in to view

aslonnie requested a review from elliot-barn October 3, 2025 04:29

aslonnie reviewed Oct 3, 2025

View reviewed changes

python/requirements/ml/train-test-requirements.txt Show resolved Hide resolved

TimothySeah added 5 commits October 3, 2025 12:10

[train][doc] Add validation and other details to checkpoint doc

b4ded6b

Signed-off-by: Timothy Seah <tseah@anyscale.com>

address cursor comments + fix ci

3d473a9

Signed-off-by: Timothy Seah <tseah@anyscale.com>

fix indentation

49bcfdf

Signed-off-by: Timothy Seah <tseah@anyscale.com>

fix formatting

c6ae97b

Signed-off-by: Timothy Seah <tseah@anyscale.com>

wrap around function to avoid error

636cb68

Signed-off-by: Timothy Seah <tseah@anyscale.com>

TimothySeah force-pushed the tseah/validating-checkpoints branch from 9c28ce9 to 636cb68 Compare October 3, 2025 19:11

elliot-barn approved these changes Oct 4, 2025

View reviewed changes

JasonLi1909 reviewed Oct 6, 2025

View reviewed changes

address comments

66c5f2d

Signed-off-by: Timothy Seah <tseah@anyscale.com>

This comment was marked as outdated.

Sign in to view

justinvyu reviewed Oct 7, 2025

View reviewed changes

matthewdeng reviewed Oct 8, 2025

View reviewed changes

doc/source/train/user-guides/checkpoints.rst Outdated Show resolved Hide resolved

doc/source/train/user-guides/checkpoints.rst Outdated Show resolved Hide resolved

doc/source/train/user-guides/checkpoints.rst Outdated Show resolved Hide resolved

TimothySeah added 2 commits October 8, 2025 18:32

1) fix dummy checkpoint upload 2) introduce single node task + add to…

6ed8661

…do to request gpu Signed-off-by: Timothy Seah <tseah@anyscale.com>

call out heterogeneous cluster advantage + metrics lifecycle

9291aeb

Signed-off-by: Timothy Seah <tseah@anyscale.com>

TimothySeah added 4 commits October 8, 2025 19:33

address simple comments

8ad1d6c

Signed-off-by: Timothy Seah <tseah@anyscale.com>

best practice: name validation runs after train run + epoch

0bec645

Signed-off-by: Timothy Seah <tseah@anyscale.com>

reorder validation doc

fabbdf4

Signed-off-by: Timothy Seah <tseah@anyscale.com>

split into separate guide, add additional notes as suggested

efbd6d3

Signed-off-by: Timothy Seah <tseah@anyscale.com>

cursor bot reviewed Oct 11, 2025

View reviewed changes

doc/source/train/doc_code/asynchronous_validation.py Outdated Show resolved Hide resolved

TimothySeah requested review from JasonLi1909, justinvyu and matthewdeng October 11, 2025 01:34

TimothySeah added 2 commits October 10, 2025 18:49

fix literalincludes

acab35d

Signed-off-by: Timothy Seah <tseah@anyscale.com>

fix doc warnings again

3b37c2a

Signed-off-by: Timothy Seah <tseah@anyscale.com>

cursor bot reviewed Oct 11, 2025

View reviewed changes

doc/source/train/doc_code/asynchronous_validation.py Outdated Show resolved Hide resolved

TimothySeah added the go add ONLY when ready to merge, run all tests label Oct 13, 2025

TimothySeah added 2 commits October 14, 2025 14:09

simplify local checkpoint

d178424

Signed-off-by: Timothy Seah <tseah@anyscale.com>

Merge remote-tracking branch 'upstream/master' into tseah/validating-…

5a5d791

…checkpoints

matthewdeng reviewed Oct 14, 2025

View reviewed changes

TimothySeah added 6 commits October 14, 2025 15:54

run formatting

3337799

Signed-off-by: Timothy Seah <tseah@anyscale.com>

address simple comments

ad8b4c6

Signed-off-by: Timothy Seah <tseah@anyscale.com>

reorganize writing a dsitributed validation function section

720609e

Signed-off-by: Timothy Seah <tseah@anyscale.com>

call out metrics replacement

2eb0a67

Signed-off-by: Timothy Seah <tseah@anyscale.com>

set dataset name as well

e1b6271

Signed-off-by: Timothy Seah <tseah@anyscale.com>

fix docbuild

574d02a

Signed-off-by: Timothy Seah <tseah@anyscale.com>

matthewdeng requested a review from angelinalg October 15, 2025 18:37

angelinalg reviewed Oct 15, 2025

View reviewed changes

justinvyu approved these changes Oct 15, 2025

View reviewed changes

TimothySeah and others added 2 commits October 15, 2025 15:53

Apply suggestions from code review

cdd8954

Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com> Signed-off-by: Timothy Seah <tseah@anyscale.com>

address more comments

ec2a9d5

Signed-off-by: Timothy Seah <tseah@anyscale.com>

matthewdeng merged commit 67c610b into ray-project:master Oct 16, 2025
6 checks passed

Conversation

TimothySeah commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

This comment was marked as outdated.

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

aslonnie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

justinvyu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TimothySeah commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

angelinalg left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

justinvyu Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

TimothySeah Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

TimothySeah commented Oct 15, 2025

Uh oh!

Uh oh!

TimothySeah commented Oct 1, 2025 •

edited

Loading

TimothySeah commented Oct 9, 2025 •

edited

Loading

angelinalg left a comment •

edited

Loading