[WIP][Grad Compression] Unittest to verify allreduce_hook parity#50851
Closed
rohan-varma wants to merge 4 commits intogh/rohan-varma/218/basefrom
Closed
[WIP][Grad Compression] Unittest to verify allreduce_hook parity#50851rohan-varma wants to merge 4 commits intogh/rohan-varma/218/basefrom
rohan-varma wants to merge 4 commits intogh/rohan-varma/218/basefrom
Conversation
Improves upon the previous unittest to ensure allreduce_hook results in the same gradients as vanilla allreduce in DDP. Note that this diff removes the powerSGD hook test, will add that in the next PR as it needs to be slightly different (gradients will no longer be equal) Differential Revision: [D25963654](https://our.internmc.facebook.com/intern/diff/D25963654/) [ghstack-poisoned]
…parity" Improves upon the previous unittest to ensure allreduce_hook results in the same gradients as vanilla allreduce in DDP. Differential Revision: [D25963654](https://our.internmc.facebook.com/intern/diff/D25963654/) [ghstack-poisoned]
rohan-varma
added a commit
that referenced
this pull request
Jan 21, 2021
Pull Request resolved: #50851 Improves upon the previous unittest to ensure allreduce_hook results in the same gradients as vanilla allreduce in DDP. ghstack-source-id: 120082598 Differential Revision: [D25963654](https://our.internmc.facebook.com/intern/diff/D25963654/)
Contributor
|
Since this test is only for allreduce_hook, it will be better to add it rather than replace the test for PowerSGD_hook. |
Codecov Report
@@ Coverage Diff @@
## gh/rohan-varma/218/base #50851 +/- ##
===========================================================
- Coverage 81.00% 80.65% -0.35%
===========================================================
Files 1916 1916
Lines 209532 209560 +28
===========================================================
- Hits 169728 169026 -702
- Misses 39804 40534 +730 |
…parity" Improves upon the previous unittest to ensure allreduce_hook results in the same gradients as vanilla allreduce in DDP. Differential Revision: [D25963654](https://our.internmc.facebook.com/intern/diff/D25963654/) [ghstack-poisoned]
rohan-varma
added a commit
that referenced
this pull request
Jan 21, 2021
Pull Request resolved: #50851 Improves upon the previous unittest to ensure allreduce_hook results in the same gradients as vanilla allreduce in DDP. ghstack-source-id: 120108788 Differential Revision: [D25963654](https://our.internmc.facebook.com/intern/diff/D25963654/)
wayi1
approved these changes
Jan 21, 2021
…parity" Improves upon the previous unittest to ensure allreduce_hook results in the same gradients as vanilla allreduce in DDP. Differential Revision: [D25963654](https://our.internmc.facebook.com/intern/diff/D25963654/) [ghstack-poisoned]
rohan-varma
added a commit
that referenced
this pull request
Jan 22, 2021
Pull Request resolved: #50851 Improves upon the previous unittest to ensure allreduce_hook results in the same gradients as vanilla allreduce in DDP. ghstack-source-id: 120229103 Differential Revision: [D25963654](https://our.internmc.facebook.com/intern/diff/D25963654/)
Contributor
|
This pull request has been merged in 5a661e0. |
laurentdupin
pushed a commit
to laurentdupin/pytorch
that referenced
this pull request
Apr 24, 2026
…orch#50851) Summary: Pull Request resolved: pytorch#50851 Improves upon the previous unittest to ensure allreduce_hook results in the same gradients as vanilla allreduce in DDP. ghstack-source-id: 120229103 Test Plan: buck build mode/dev-nosan //caffe2/test/distributed:distributed_nccl_fork --keep-going BACKEND=nccl WORLD_SIZE=2 ~/fbcode/buck-out/dev/gen/caffe2/test/distributed/distributed_nccl_fork#binary.par -r test_ddp_hook_parity Reviewed By: SciPioneer Differential Revision: D25963654 fbshipit-source-id: d55eee0aee9cf1da52aa0c4ba1066718aa8fd9a4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
Improves upon the previous unittest to ensure allreduce_hook results in the same gradients as vanilla allreduce in DDP.
Differential Revision: D25963654