[Train] [release test] Release tests for ray train local mode by xinyuangui2 · Pull Request #56862 · ray-project/ray

xinyuangui2 · 2025-09-24T00:27:30Z

Why are these changes needed?

Release test for ray train local mode.

Setup:

4 nodes, each with 4 gpus.
Each node runs torchrun.

Test job: https://buildkite.com/ray-project/release/builds/61135

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Note

Adds a nightly Train release test that launches torchrun across 2 GPU nodes to run a Ray Train TorchTrainer on FashionMNIST and validates loss.

Train tests:
- Add torch_local_mode to release/release_tests.yaml (nightly, BYOD GPU) running python torch_local_mode_launcher.py with cluster config train_tests/local_mode/compute_gpu_2x4_aws.yaml.
New local mode assets:
- release/train_tests/local_mode/compute_gpu_2x4_aws.yaml: 1 head m5.4xlarge, 2 workers g4dn.12xlarge (non-spot) in us-west-2.
- release/train_tests/local_mode/torch_local_mode_launcher.py: Starts a Ray cluster task that uploads torch_local_mode_test.py to 2 nodes and launches torchrun (--nnodes=2, --nproc-per-node=4) with RAY_TRAIN_V2_ENABLED=1.
- release/train_tests/local_mode/torch_local_mode_test.py: Defines a TorchTrainer training resnet18 on FashionMNIST, reports checkpoints/metrics over 20 epochs, and asserts loss decreases.

^{Written by Cursor Bugbot for commit 1157068. This will update automatically on new commits. Configure here.}

Signed-off-by: xgui <xgui@anyscale.com>

release/train_tests/local_mode/torch_local_mode_launcher.py

release/train_tests/local_mode/compute_gpu_4x4_aws.yaml

Co-authored-by: Jason Li <57246540+JasonLi1909@users.noreply.github.com> Signed-off-by: Xinyuan <43737116+xinyuangui2@users.noreply.github.com>

Resolve comments Signed-off-by: Xinyuan <43737116+xinyuangui2@users.noreply.github.com>

Signed-off-by: xgui <xgui@anyscale.com>

release/train_tests/local_mode/torch_local_mode_test.py

matthewdeng · 2025-09-29T21:27:49Z

release/train_tests/local_mode/torch_local_mode_test.py

+def fit_func():
+    # Define configurations.
+    train_loop_config = {"num_epochs": 20, "lr": 0.01, "batch_size": 32}
+    scaling_config = ScalingConfig(num_workers=0, use_gpu=True)
+    run_config = RunConfig(checkpoint_config=CheckpointConfig(num_to_keep=1))
+
+    # Initialize the Trainer.
+    trainer = TorchTrainer(
+        train_loop_per_worker=train_func,
+        train_loop_config=train_loop_config,
+        scaling_config=scaling_config,
+        run_config=run_config,
+    )
+
+    # Train the model.
+    result = trainer.fit()
+
+    # Inspect the results.
+    final_loss = result.metrics["loss"]
+    logger.info(f"final_loss: {final_loss}")
+
+
+if __name__ == "__main__":
+    fit_func()


Is this run anywhere?

Would it be helpful to verify that the runtimes are the same? And maybe set the seeds and compare the results?

Added loss convergence check.

release/train_tests/local_mode/torch_local_mode_launcher.py

Signed-off-by: xgui <xgui@anyscale.com>

release/train_tests/local_mode/torch_local_mode_launcher.py

Release test for ray train local mode. Setup: - 2 nodes, each with 4 gpus. - Each node runs `torchrun`. Test job: https://buildkite.com/ray-project/release/builds/61135 --------- Signed-off-by: xgui <xgui@anyscale.com> Signed-off-by: Xinyuan <43737116+xinyuangui2@users.noreply.github.com> Co-authored-by: Jason Li <57246540+JasonLi1909@users.noreply.github.com> Signed-off-by: Douglas Strodtman <douglas@anyscale.com>

…oject#56862) Release test for ray train local mode. Setup: - 2 nodes, each with 4 gpus. - Each node runs `torchrun`. Test job: https://buildkite.com/ray-project/release/builds/61135 --------- Signed-off-by: xgui <xgui@anyscale.com> Signed-off-by: Xinyuan <43737116+xinyuangui2@users.noreply.github.com> Co-authored-by: Jason Li <57246540+JasonLi1909@users.noreply.github.com> Signed-off-by: Seiji Eicher <seiji@anyscale.com>

…oject#56862) Release test for ray train local mode. Setup: - 2 nodes, each with 4 gpus. - Each node runs `torchrun`. Test job: https://buildkite.com/ray-project/release/builds/61135 --------- Signed-off-by: xgui <xgui@anyscale.com> Signed-off-by: Xinyuan <43737116+xinyuangui2@users.noreply.github.com> Co-authored-by: Jason Li <57246540+JasonLi1909@users.noreply.github.com>

…oject#56862) Release test for ray train local mode. Setup: - 2 nodes, each with 4 gpus. - Each node runs `torchrun`. Test job: https://buildkite.com/ray-project/release/builds/61135 --------- Signed-off-by: xgui <xgui@anyscale.com> Signed-off-by: Xinyuan <43737116+xinyuangui2@users.noreply.github.com> Co-authored-by: Jason Li <57246540+JasonLi1909@users.noreply.github.com> Signed-off-by: Josh Kodi <joshkodi@gmail.com>

…oject#56862) Release test for ray train local mode. Setup: - 2 nodes, each with 4 gpus. - Each node runs `torchrun`. Test job: https://buildkite.com/ray-project/release/builds/61135 --------- Signed-off-by: xgui <xgui@anyscale.com> Signed-off-by: Xinyuan <43737116+xinyuangui2@users.noreply.github.com> Co-authored-by: Jason Li <57246540+JasonLi1909@users.noreply.github.com>

…oject#56862) Release test for ray train local mode. Setup: - 2 nodes, each with 4 gpus. - Each node runs `torchrun`. Test job: https://buildkite.com/ray-project/release/builds/61135 --------- Signed-off-by: xgui <xgui@anyscale.com> Signed-off-by: Xinyuan <43737116+xinyuangui2@users.noreply.github.com> Co-authored-by: Jason Li <57246540+JasonLi1909@users.noreply.github.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…oject#56862) Release test for ray train local mode. Setup: - 2 nodes, each with 4 gpus. - Each node runs `torchrun`. Test job: https://buildkite.com/ray-project/release/builds/61135 --------- Signed-off-by: xgui <xgui@anyscale.com> Signed-off-by: Xinyuan <43737116+xinyuangui2@users.noreply.github.com> Co-authored-by: Jason Li <57246540+JasonLi1909@users.noreply.github.com> Signed-off-by: Future-Outlier <eric901201@gmail.com>

xinyuangui2 added 5 commits September 24, 2025 00:27

adding release tests for local mode

5ccbe29

Signed-off-by: xgui <xgui@anyscale.com>

fix the func calling bug

27674a5

Signed-off-by: xgui <xgui@anyscale.com>

avoid overwriting the visible cuda for actors

c3bd2a4

Signed-off-by: xgui <xgui@anyscale.com>

update name

bfc0ae2

Signed-off-by: xgui <xgui@anyscale.com>

upload file to all nodes

54e5224

Signed-off-by: xgui <xgui@anyscale.com>

xinyuangui2 changed the title ~~adding release tests for local mode~~ [Train] [release test] Release tests for ray train local mode Sep 24, 2025

xinyuangui2 marked this pull request as ready for review September 24, 2025 22:38

xinyuangui2 requested review from justinvyu and matthewdeng September 24, 2025 22:38

This comment was marked as outdated.

Sign in to view

xinyuangui2 and others added 2 commits September 24, 2025 22:51

use config inside train func

6cb0028

Signed-off-by: xgui <xgui@anyscale.com>

Merge branch 'master' into local-run-test

06dcbd0

This comment was marked as outdated.

Sign in to view

ray-gardener bot added train Ray Train Related Issue release-test release test labels Sep 25, 2025

matthewdeng requested a review from JasonLi1909 September 26, 2025 22:10

JasonLi1909 approved these changes Sep 27, 2025

View reviewed changes

release/train_tests/local_mode/torch_local_mode_launcher.py Outdated Show resolved Hide resolved

release/train_tests/local_mode/torch_local_mode_launcher.py Outdated Show resolved Hide resolved

release/train_tests/local_mode/compute_gpu_4x4_aws.yaml Outdated Show resolved Hide resolved

xinyuangui2 and others added 4 commits September 29, 2025 11:11

Update release/train_tests/local_mode/torch_local_mode_launcher.py

3597e6b

Co-authored-by: Jason Li <57246540+JasonLi1909@users.noreply.github.com> Signed-off-by: Xinyuan <43737116+xinyuangui2@users.noreply.github.com>

Update torch_local_mode_launcher.py

a61fd8a

Resolve comments Signed-off-by: Xinyuan <43737116+xinyuangui2@users.noreply.github.com>

Merge branch 'master' into local-run-test

8d4ce13

change to 2 nodes

8f0e813

Signed-off-by: xgui <xgui@anyscale.com>

matthewdeng reviewed Sep 29, 2025

View reviewed changes

xinyuangui2 and others added 2 commits October 1, 2025 14:44

Merge branch 'master' into local-run-test

584e584

try to add validations for local mode

1157068

Signed-off-by: xgui <xgui@anyscale.com>

xinyuangui2 requested a review from matthewdeng October 2, 2025 00:00

matthewdeng approved these changes Oct 3, 2025

View reviewed changes

matthewdeng enabled auto-merge (squash) October 3, 2025 00:19

github-actions bot added the go add ONLY when ready to merge, run all tests label Oct 3, 2025

Merge branch 'master' into local-run-test

27d892f

github-actions bot disabled auto-merge October 3, 2025 17:51

cursor bot reviewed Oct 3, 2025

View reviewed changes

release/train_tests/local_mode/torch_local_mode_launcher.py Show resolved Hide resolved

matthewdeng merged commit 637aaf3 into ray-project:master Oct 3, 2025
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Train] [release test] Release tests for ray train local mode#56862

[Train] [release test] Release tests for ray train local mode#56862
matthewdeng merged 14 commits intoray-project:masterfrom
xinyuangui2:local-run-test

xinyuangui2 commented Sep 24, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

matthewdeng Sep 29, 2025

Uh oh!

xinyuangui2 Oct 1, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

xinyuangui2 commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

matthewdeng Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

xinyuangui2 Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xinyuangui2 commented Sep 24, 2025 •

edited

Loading