add stats that can only be collected at runtime#51386
Closed
zhaojuanmao wants to merge 14 commits intogh/zhaojuanmao/59/basefrom
Closed
add stats that can only be collected at runtime#51386zhaojuanmao wants to merge 14 commits intogh/zhaojuanmao/59/basefrom
zhaojuanmao wants to merge 14 commits intogh/zhaojuanmao/59/basefrom
Conversation
add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/) [ghstack-poisoned]
Contributor
💊 CI failures summary and remediationsAs of commit f429313 (more details on the Dr. CI page): 💚 💚 Looks good so far! There are no failures yet. 💚 💚 This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions to the (internal) Dr. CI Users group. |
This was referenced Jan 29, 2021
zhaojuanmao
added a commit
that referenced
this pull request
Jan 29, 2021
add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/) ghstack-source-id: 120695335 Pull Request resolved: #51386
wayi1
reviewed
Jan 30, 2021
wayi1
reviewed
Jan 30, 2021
add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/) [ghstack-poisoned]
zhaojuanmao
added a commit
that referenced
this pull request
Feb 2, 2021
Pull Request resolved: #51386 add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data ghstack-source-id: 120818571 Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/)
Contributor
Author
|
right now all the timers work for CPU only, will add GPU timers |
wayi1
approved these changes
Feb 2, 2021
add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/) [ghstack-poisoned]
add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/) [ghstack-poisoned]
zhaojuanmao
added a commit
that referenced
this pull request
Feb 4, 2021
Pull Request resolved: #51386 add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data ghstack-source-id: 121022194 Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/)
add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/) [ghstack-poisoned]
zhaojuanmao
added a commit
that referenced
this pull request
Feb 5, 2021
Pull Request resolved: #51386 add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data ghstack-source-id: 121100800 Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/)
rohan-varma
reviewed
Feb 5, 2021
rohan-varma
reviewed
Feb 5, 2021
rohan-varma
reviewed
Feb 5, 2021
rohan-varma
reviewed
Feb 5, 2021
rohan-varma
reviewed
Feb 5, 2021
rohan-varma
reviewed
Feb 12, 2021
rohan-varma
reviewed
Feb 12, 2021
rohan-varma
reviewed
Feb 12, 2021
rohan-varma
reviewed
Feb 12, 2021
Contributor
Author
|
although current diff did not result in perf regression in our benchmarks, but for better control performance just in case some application needs it in the future, I will add sampling for event recording in this diff. |
Contributor
Author
|
@ilia-cher and @ngimel, would you please kindly help reviewing whether CUDA time measurement parts are written properly (any pitfall I did not realize)? thanks a lot! |
rohan-varma
approved these changes
Feb 13, 2021
Contributor
rohan-varma
left a comment
There was a problem hiding this comment.
Looks good from my side! Thank you for working through all of the different timing issues!
add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data 1. gpu time stats are not collected for single process multiple devices and multi device modules in this diff, as that requires events are created and recorded on multiple devices 2. use at::cuda::event API for safer calls 3. events may not be created in autograd hook if hook is not triggered in user's codes, e.g., users runs in non-sync mode in some iterations. So we checked events are created or not before synchronizing, also skipped invalid results. 4. users may not set device upfront, so explicitly set proper device before creating events in our prepare_forward() and prepare_backward() calls Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/) [ghstack-poisoned]
Contributor
Author
|
added sampling control for event recording |
rohan-varma
reviewed
Feb 17, 2021
add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data 1. gpu time stats are not collected for single process multiple devices and multi device modules in this diff, as that requires events are created and recorded on multiple devices 2. use at::cuda::event API for safer calls 3. events may not be created in autograd hook if hook is not triggered in user's codes, e.g., users runs in non-sync mode in some iterations. So we checked events are created or not before synchronizing, also skipped invalid results. 4. users may not set device upfront, so explicitly set proper device before creating events in our prepare_forward() and prepare_backward() calls Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/) [ghstack-poisoned]
zhaojuanmao
added a commit
that referenced
this pull request
Feb 17, 2021
Pull Request resolved: #51386 add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data 1. gpu time stats are not collected for single process multiple devices in this diff, as that requires events are created and recorded on multiple devices 2. use at::cuda::event API for safer calls 3. events may not be created in autograd hook if hook is not triggered in user's codes, e.g., users runs in non-sync mode in some iterations. So we checked events are created or not before synchronizing, also skipped invalid results. 4. users may not set device upfront, so explicitly set proper device before creating events in our prepare_forward() and prepare_backward() calls ghstack-source-id: 121829631 Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D26158645/)!
mrshenli
reviewed
Feb 17, 2021
add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data 1. gpu time stats are not collected for single process multiple devices and multi device modules in this diff, as that requires events are created and recorded on multiple devices 2. use at::cuda::event API for safer calls 3. events may not be created in autograd hook if hook is not triggered in user's codes, e.g., users runs in non-sync mode in some iterations. So we checked events are created or not before synchronizing, also skipped invalid results. 4. users may not set device upfront, so explicitly set proper device before creating events in our prepare_forward() and prepare_backward() calls Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/) [ghstack-poisoned]
add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data 1. gpu time stats are not collected for single process multiple devices and multi device modules in this diff, as that requires events are created and recorded on multiple devices 2. use at::cuda::event API for safer calls 3. events may not be created in autograd hook if hook is not triggered in user's codes, e.g., users runs in non-sync mode in some iterations. So we checked events are created or not before synchronizing, also skipped invalid results. 4. users may not set device upfront, so explicitly set proper device before creating events in our prepare_forward() and prepare_backward() calls Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/) [ghstack-poisoned]
zhaojuanmao
added a commit
that referenced
this pull request
Feb 18, 2021
Pull Request resolved: #51386 add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data 1. gpu time stats are not collected for single process multiple devices in this diff, as that requires events are created and recorded on multiple devices 2. use at::cuda::event API for safer calls 3. events may not be created in autograd hook if hook is not triggered in user's codes, e.g., users runs in non-sync mode in some iterations. So we checked events are created or not before synchronizing, also skipped invalid results. 4. users may not set device upfront, so explicitly set proper device before creating events in our prepare_forward() and prepare_backward() calls ghstack-source-id: 121933566 Differential Revision: [D26158645](https://our.internmc.facebook.com/intern/diff/D26158645/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D26158645/)!
Contributor
|
This pull request has been merged in c75fa39. |
aocsa
pushed a commit
to Quansight/pytorch
that referenced
this pull request
Mar 15, 2021
Summary: Pull Request resolved: pytorch#51386 add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data 1. gpu time stats are not collected for single process multiple devices in this diff, as that requires events are created and recorded on multiple devices 2. use at::cuda::event API for safer calls 3. events may not be created in autograd hook if hook is not triggered in user's codes, e.g., users runs in non-sync mode in some iterations. So we checked events are created or not before synchronizing, also skipped invalid results. 4. users may not set device upfront, so explicitly set proper device before creating events in our prepare_forward() and prepare_backward() calls ghstack-source-id: 121933566 Test Plan: unit tests Reviewed By: SciPioneer Differential Revision: D26158645 fbshipit-source-id: ce5f15187802eba76accb980449be68902c10178
xsacha
pushed a commit
to xsacha/pytorch
that referenced
this pull request
Mar 31, 2021
Summary: Pull Request resolved: pytorch#51386 add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data 1. gpu time stats are not collected for single process multiple devices in this diff, as that requires events are created and recorded on multiple devices 2. use at::cuda::event API for safer calls 3. events may not be created in autograd hook if hook is not triggered in user's codes, e.g., users runs in non-sync mode in some iterations. So we checked events are created or not before synchronizing, also skipped invalid results. 4. users may not set device upfront, so explicitly set proper device before creating events in our prepare_forward() and prepare_backward() calls ghstack-source-id: 121933566 Test Plan: unit tests Reviewed By: SciPioneer Differential Revision: D26158645 fbshipit-source-id: ce5f15187802eba76accb980449be68902c10178
laurentdupin
pushed a commit
to laurentdupin/pytorch
that referenced
this pull request
Apr 24, 2026
Summary: Pull Request resolved: pytorch#51386 add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data 1. gpu time stats are not collected for single process multiple devices in this diff, as that requires events are created and recorded on multiple devices 2. use at::cuda::event API for safer calls 3. events may not be created in autograd hook if hook is not triggered in user's codes, e.g., users runs in non-sync mode in some iterations. So we checked events are created or not before synchronizing, also skipped invalid results. 4. users may not set device upfront, so explicitly set proper device before creating events in our prepare_forward() and prepare_backward() calls ghstack-source-id: 121933566 Test Plan: unit tests Reviewed By: SciPioneer Differential Revision: D26158645 fbshipit-source-id: ce5f15187802eba76accb980449be68902c10178
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
add stats such as rebuilt bucket stats, unused parameter stats and performance stats to ddp logging data
Differential Revision: D26158645