[train][doc] Add checkpoint_upload_mode to checkpoint docs by TimothySeah · Pull Request #56860 · ray-project/ray

TimothySeah · 2025-09-24T00:17:11Z

Summary

Add checkpoint_upload_mode to checkpoint docs

Testing

docbuild

Note

Adds docs and examples for checkpoint upload modes (SYNC/ASYNC/NO_UPLOAD), updates API refs, includes s3torchconnector deps, and tweaks CI to use classic conda solver for mpi4py.

Docs (Train Checkpoints)
- Add CheckpointUploadMode with guidance and examples for SYNC, ASYNC, and NO_UPLOAD in doc/source/train/user-guides/checkpoints.rst and doc_code/checkpoints.py.
- Reference new figures and code snippets; update API refs to include ~train.CheckpointUploadMode in doc/source/train/api/api.rst.
Dependencies
- Add s3torchconnector==1.4.3 (and s3torchconnectorclient==1.4.3) in python/requirements_compiled.txt; include s3torchconnector in python/requirements/ml/train-test-requirements.txt.
CI
- Use classic conda solver for mpi4py install in ci/env/install-miniforge.sh.
- Include ci/env/install-miniforge.sh in ci/docker/base.ml.wanda.yaml build context.

^{Written by Cursor Bugbot for commit 9dcf683. This will update automatically on new commits. Configure here.}

justinvyu

can you add an explicit "Asynchronous checkpointing" header?

doc/source/train/user-guides/checkpoints.rst

doc/source/train/doc_code/checkpoints.py

doc/source/train/user-guides/checkpoints.rst

JasonLi1909 · 2025-09-29T18:49:25Z

doc/source/train/user-guides/checkpoints.rst

+the next training step can start in parallel. If so, you should use
+``ray.train.CheckpointUploadMode.ASYNC``. This is helpful for larger
+checkpoints that might take longer to upload, but might add unnecessary
+complexity if you want to immediately upload a small checkpoint.


It might be helpful to elaborate more on what the tradeoffs are between async and sync. Such as what the complexities might be, is there a significant performance or memory tradeoff, or is async usually just the better option

Added which kicks off a new thread to upload the checkpoint and moved Each ``report`` blocks until the previous ``report``\'s checkpoint upload completes before starting a new checkpoint upload thread. Ray Train does this to avoid accumulating too many upload threads and potentially running out of memory. to the main body.

What I'm going for is:

complexities: kicking off a thread + the report blocking thing above

performance: only worthwhile if checkpoint is big

memory: more memory from new thread, but this is capped at 1 thread as per the comment above

Lmk if the new wording addresses these points.

justinvyu

Just one comment about the figure

doc/source/train/images/sync_vs_async_checkpointing.png

doc/source/train/user-guides/checkpoints.rst

Signed-off-by: Timothy Seah <tseah@anyscale.com>

This reverts commit f073cef. Signed-off-by: Timothy Seah <tseah@anyscale.com>

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Signed-off-by: Timothy Seah <tseah@anyscale.com>

Co-authored-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: Timothy Seah <tseah@anyscale.com>

Signed-off-by: Timothy Seah <tseah@anyscale.com>

…ct#56860) Add checkpoint_upload_mode to checkpoint docs --------- Signed-off-by: Timothy Seah <tseah@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: Douglas Strodtman <douglas@anyscale.com>

…ct#56860) Add checkpoint_upload_mode to checkpoint docs --------- Signed-off-by: Timothy Seah <tseah@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: Seiji Eicher <seiji@anyscale.com>

…ct#56860) Add checkpoint_upload_mode to checkpoint docs --------- Signed-off-by: Timothy Seah <tseah@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Justin Yu <justinvyu@anyscale.com>

…ct#56860) Add checkpoint_upload_mode to checkpoint docs --------- Signed-off-by: Timothy Seah <tseah@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: Josh Kodi <joshkodi@gmail.com>

…ct#56860) Add checkpoint_upload_mode to checkpoint docs --------- Signed-off-by: Timothy Seah <tseah@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Justin Yu <justinvyu@anyscale.com>

…ct#56860) Add checkpoint_upload_mode to checkpoint docs --------- Signed-off-by: Timothy Seah <tseah@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…ct#56860) Add checkpoint_upload_mode to checkpoint docs --------- Signed-off-by: Timothy Seah <tseah@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

TimothySeah added the go add ONLY when ready to merge, run all tests label Sep 24, 2025

TimothySeah marked this pull request as ready for review September 24, 2025 18:34

TimothySeah requested review from a team as code owners September 24, 2025 18:34

ray-gardener bot added docs An issue or change related to documentation train Ray Train Related Issue labels Sep 24, 2025

justinvyu reviewed Sep 24, 2025

View reviewed changes

matthewdeng reviewed Sep 25, 2025

View reviewed changes

doc/source/train/user-guides/checkpoints.rst Show resolved Hide resolved

doc/source/train/user-guides/checkpoints.rst Outdated Show resolved Hide resolved

doc/source/train/user-guides/checkpoints.rst Outdated Show resolved Hide resolved

TimothySeah requested review from justinvyu and matthewdeng September 26, 2025 02:01

matthewdeng reviewed Sep 26, 2025

View reviewed changes

TimothySeah requested a review from a team as a code owner September 26, 2025 18:58

TimothySeah requested a review from matthewdeng September 26, 2025 20:03

aslonnie requested review from elliot-barn and removed request for a team September 26, 2025 22:49

matthewdeng reviewed Sep 29, 2025

View reviewed changes

doc/source/train/user-guides/checkpoints.rst Outdated Show resolved Hide resolved

doc/source/train/user-guides/checkpoints.rst Show resolved Hide resolved

angelinalg approved these changes Sep 29, 2025

View reviewed changes

JasonLi1909 reviewed Sep 29, 2025

View reviewed changes

TimothySeah requested review from JasonLi1909 and matthewdeng September 30, 2025 23:59

justinvyu reviewed Oct 1, 2025

View reviewed changes

doc/source/train/images/sync_vs_async_checkpointing.png Show resolved Hide resolved

doc/source/train/user-guides/checkpoints.rst Outdated Show resolved Hide resolved

TimothySeah added 9 commits October 2, 2025 11:01

[train][doc] Add checkpoint_upload_mode to checkpoint docs

9f4e67b

Signed-off-by: Timothy Seah <tseah@anyscale.com>

remove empty lines

66a4f8b

Signed-off-by: Timothy Seah <tseah@anyscale.com>

add spaces back according to lint

1081cde

Signed-off-by: Timothy Seah <tseah@anyscale.com>

fix rendering issue

99441dd

Signed-off-by: Timothy Seah <tseah@anyscale.com>

address comments

ed17230

Signed-off-by: Timothy Seah <tseah@anyscale.com>

change 1 word

b2ff587

Signed-off-by: Timothy Seah <tseah@anyscale.com>

ci fixes

fa76c29

Signed-off-by: Timothy Seah <tseah@anyscale.com>

try adding s3torchconnector to doc baezl target

522130b

Signed-off-by: Timothy Seah <tseah@anyscale.com>

address comments

bb2db5f

Signed-off-by: Timothy Seah <tseah@anyscale.com>

TimothySeah and others added 10 commits October 2, 2025 11:01

fix lint + tweak wording

b751400

Signed-off-by: Timothy Seah <tseah@anyscale.com>

Revert "try adding s3torchconnector to doc baezl target"

2fa930c

This reverts commit f073cef. Signed-off-by: Timothy Seah <tseah@anyscale.com>

adding dep

c3b3020

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

lint fix

9e2a3bd

Signed-off-by: Timothy Seah <tseah@anyscale.com>

add CheckpointUploadMode to api.rst

e6b5f30

Signed-off-by: Timothy Seah <tseah@anyscale.com>

retrigger doc build

b69f436

Signed-off-by: Timothy Seah <tseah@anyscale.com>

retrigger doc build

78d8321

Signed-off-by: Timothy Seah <tseah@anyscale.com>

address comments

09c3763

Signed-off-by: Timothy Seah <tseah@anyscale.com>

Update doc/source/train/user-guides/checkpoints.rst

aa73d94

Co-authored-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: Timothy Seah <tseah@anyscale.com>

update figure

b5b05fc

Signed-off-by: Timothy Seah <tseah@anyscale.com>

TimothySeah force-pushed the tseah/doc-checkpoint-upload-mode branch from 9dcf683 to b5b05fc Compare October 2, 2025 18:01

justinvyu approved these changes Oct 2, 2025

View reviewed changes

justinvyu merged commit 77b1dcf into ray-project:master Oct 2, 2025
6 checks passed

Conversation

TimothySeah commented Sep 24, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

justinvyu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JasonLi1909 Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

TimothySeah Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

justinvyu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

TimothySeah commented Sep 24, 2025 •

edited by cursor bot

Loading