Add NCCL support for `pytorch_distributed_simple.py`#150

Closed

toshihikoyanase wants to merge 6 commits intooptuna:mainfrom

toshihikoyanase:add-distributed-support-for-pytorch-distributed-simple

Member

toshihikoyanase commented Nov 29, 2022

Motivation

Alternative approach for #145.

Description of the changes

Use device argument of optuna.integration.TorchDistributedTrial
Check local rank to specify CUDA device in a node
Use MASTER_ADDR and MASTER_PORT in environment variables if exist

I confirmed that this script worked with 2 nodes.

$ mpiexec -x OMP_NUM_THREADS=1 --bind-to none \
    -x MASTER_ADDR="your master address" \
    -x MASTER_PORT="your master port" \
    python pytorch/pytorch_distributed_simple.py

See
https://pytorch.org/docs/stable/distributed.html

toshihikoyanase added 5 commits

November 29, 2022 15:24


          Add support for GPU training.

1cb11e0


          Read MASTER_ADDR and MASTER_PORT if exist

555c61b


          minor fix

86271f2


          Fix data download

bcf468b


          cast local rank

435867d

toshihikoyanase mentioned this pull request

TorchDistributedTrial uses group as parameter instead of device optuna/optuna#4106

Merged


          Apply black

bc5002e

HideakiImamura assigned not522

Member

HideakiImamura commented Nov 30, 2022

@not522 Could you review this PR?

Member Author

toshihikoyanase commented Dec 1, 2022 •

edited

Loading

I found that this code did not work on the PC which had NVIDIA GPUs without NCCL support.
If you have any idea to check NCCL availability, please let me know.

github-actions bot commented Dec 8, 2022

This pull request has not seen any recent activity.

github-actions bot added the stale label

not522 removed the stale label

github-actions bot commented Dec 22, 2022

This pull request has not seen any recent activity.

github-actions bot added the stale label

not522 removed the stale label

github-actions bot commented Jan 1, 2023

This pull request has not seen any recent activity.

github-actions bot added the stale label

not522 removed the stale label

nzw0301 mentioned this pull request

An example of using pytorch distributed data parallel on 1 machine with arbitrary multiple GPUs. #155

Merged

github-actions bot commented Jan 17, 2023

This pull request has not seen any recent activity.

github-actions bot added the stale label

not522 removed the stale label

github-actions bot commented Feb 7, 2023

This pull request has not seen any recent activity.

github-actions bot added the stale label

Member

not522 commented Feb 8, 2023

Sorry for my late response. I have been busy these days and don't have time to review this PR, so could you please reassign the reviewer?

github-actions bot removed the stale label

github-actions bot commented Feb 19, 2023

This pull request has not seen any recent activity.

github-actions bot added the stale label

github-actions bot commented Mar 6, 2023

This pull request was closed automatically because it had not seen any recent activity. If you want to discuss it, you can reopen it freely.

github-actions bot closed this

c-bata reopened this

c-bata removed the stale label

c-bata assigned c-bata and unassigned not522

Member

c-bata commented Mar 13, 2023

I will review this PR along with optuna/optuna#4268. @toshihikoyanase Could you fix CI errors?

Member Author

toshihikoyanase commented Mar 13, 2023

Let me close this PR since it uses the old API. I'll create a new one.

toshihikoyanase closed this

toshihikoyanase deleted the add-distributed-support-for-pytorch-distributed-simple branch

March 13, 2023 04:41

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet