Skip to content

Fix hang in DDP HPC accelerators#5157

Merged
tchaton merged 2 commits intoLightning-AI:masterfrom
ananthsub:fix-ddp-hpc-accelerator
Dec 16, 2020
Merged

Fix hang in DDP HPC accelerators#5157
tchaton merged 2 commits intoLightning-AI:masterfrom
ananthsub:fix-ddp-hpc-accelerator

Conversation

@ananthsub
Copy link
Copy Markdown
Contributor

@ananthsub ananthsub commented Dec 16, 2020 •
edited
Loading

What does this PR do?

init_device was not called, so the root_gpu is 0 throughout. This leads to a hang later on in ddp_train when configure_ddp was called, as the LightningDistributedDataParallel call never completes with the redundant device ids set.

This is a recent change from #5016 and there aren't tests in OSS for this particular accelerator

2 main questions:

  • Here I'm calling init_device immediately after set_world_ranks - is that correct?
  • What should be done in the DDP HPC CPU accelerator which extends this? Setting the root gpu there doesn't make sense so for now I've set init_device as a no-op, but I'm not sure if that's the most robust.

Before submitting

  • Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?
  • Did you verify new and existing tests pass locally with your changes?
  • If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified; Bugfixes should be including in bug-fix release milestones (m.f.X) and features should be included in (m.X.b) releases.

Did you have fun?

Make sure you had fun coding 🙃

@ananthsub
Fix hang in DDP HPC accelerators
cc569f4 
init_device was never called
@codecov
Copy link
Copy Markdown

codecov bot commented Dec 16, 2020 •
edited
Loading

Codecov Report

Merging #5157 (67c2c60) into master (b4d926b) will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #5157   +/-   ##
======================================
  Coverage      93%     93%           
======================================
  Files         134     134           
  Lines        9905    9905           
======================================
  Hits         9204    9204           
  Misses        701     701

@SeanNaren SeanNaren added bug Something isn't working distributed Generic distributed-related topic priority: 0 High priority task labels Dec 16, 2020
@Borda Borda added this to the 1.1.x milestone Dec 16, 2020
Borda
Borda approved these changes Dec 16, 2020
View reviewed changes
Copy link
Copy Markdown
Collaborator

@Borda Borda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, mind add chlog?

@SeanNaren
Copy link
Copy Markdown
Contributor

SeanNaren commented Dec 16, 2020

I think the changes are fine apologies for the mistake. This is correct, devices just now have to be initialised separately from the model due to changes for the RPC plugin, and before initializing any model/connections.

@tchaton tchaton merged commit 94838d3 into Lightning-AI:master Dec 16, 2020
awaelchli pushed a commit that referenced this pull request Dec 18, 2020
@ananthsub @awaelchli
Fix hang in DDP HPC accelerators (#5157)
a50a1e4 
* Fix hang in DDP HPC accelerators

init_device was never called

* Update CHANGELOG.md
Borda pushed a commit that referenced this pull request Dec 23, 2020
@ananthsub @Borda
Fix hang in DDP HPC accelerators (#5157)
571195c 
* Fix hang in DDP HPC accelerators

init_device was never called

* Update CHANGELOG.md
Borda pushed a commit that referenced this pull request Dec 29, 2020
@ananthsub @Borda
Fix hang in DDP HPC accelerators (#5157)
96adf7f 
* Fix hang in DDP HPC accelerators

init_device was never called

* Update CHANGELOG.md
LoicGrobol added a commit to LoicGrobol/zeldarose that referenced this pull request Dec 30, 2020
@LoicGrobol
remove support for py37, bump pl version for [5157][1]
05a4031 
[1]: Lightning-AI/pytorch-lightning#5157
Borda pushed a commit that referenced this pull request Jan 4, 2021
@ananthsub @Borda
Fix hang in DDP HPC accelerators (#5157)
785ea1e 
* Fix hang in DDP HPC accelerators

init_device was never called

* Update CHANGELOG.md
Borda pushed a commit that referenced this pull request Jan 4, 2021
@ananthsub @Borda
Fix hang in DDP HPC accelerators (#5157)
744f537 
* Fix hang in DDP HPC accelerators

init_device was never called

* Update CHANGELOG.md
Borda pushed a commit that referenced this pull request Jan 5, 2021
@ananthsub @Borda
Fix hang in DDP HPC accelerators (#5157)
a7fe24e 
* Fix hang in DDP HPC accelerators

init_device was never called

* Update CHANGELOG.md
@ananthsub ananthsub mentioned this pull request Jan 13, 2021
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Borda Borda Borda approved these changes

@tchaton tchaton tchaton approved these changes

@awaelchli awaelchli Awaiting requested review from awaelchli

@justusschock justusschock Awaiting requested review from justusschock justusschock is a code owner

@williamFalcon williamFalcon Awaiting requested review from williamFalcon

+1 more reviewer

@SeanNaren SeanNaren SeanNaren approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

bug Something isn't working distributed Generic distributed-related topic priority: 0 High priority task

Projects

None yet

Milestone

pl:1.1.x

Development

Successfully merging this pull request may close these issues.

4 participants

@ananthsub @SeanNaren @Borda @tchaton