Quantization aware training: Freeze batch norm support#26624
Closed
raghuramank100 wants to merge 11 commits intogh/raghuramank100/35/basefrom
Closed
Quantization aware training: Freeze batch norm support#26624raghuramank100 wants to merge 11 commits intogh/raghuramank100/35/basefrom
raghuramank100 wants to merge 11 commits intogh/raghuramank100/35/basefrom
Conversation
For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training Differential Revision: [D17512199](https://our.internmc.facebook.com/intern/diff/D17512199/) [ghstack-poisoned]
This was referenced Sep 22, 2019
Closed
For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training Differential Revision: [D17512199](https://our.internmc.facebook.com/intern/diff/D17512199/) [ghstack-poisoned]
For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training Differential Revision: [D17512199](https://our.internmc.facebook.com/intern/diff/D17512199/) [ghstack-poisoned]
For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training Differential Revision: [D17512199](https://our.internmc.facebook.com/intern/diff/D17512199/) [ghstack-poisoned]
jerryzh168
reviewed
Sep 23, 2019
jerryzh168
reviewed
Sep 23, 2019
| return super(ConvReLU2d, cls).from_float(mod, qconfig) | ||
|
|
||
| def update_bn_stats(mod): | ||
| if type(mod) in set([ConvBnReLU2d,ConvBn2d]): |
For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training Differential Revision: [D17512199](https://our.internmc.facebook.com/intern/diff/D17512199/) [ghstack-poisoned]
dzhulgakov
reviewed
Sep 24, 2019
Collaborator
dzhulgakov
left a comment
There was a problem hiding this comment.
General question, probably not for this diff - do we need freeze_bn support also in non-fused modules?
For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training Differential Revision: [D17512199](https://our.internmc.facebook.com/intern/diff/D17512199/) [ghstack-poisoned]
Contributor
Author
Setting bn to eval() would do that and allow for frozen statistics to be used during training, but this is ugly as we want the rest of the modules to be in train(), with bn alone being in eval. |
For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training Differential Revision: [D17512199](https://our.internmc.facebook.com/intern/diff/D17512199/) [ghstack-poisoned]
For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training Differential Revision: [D17512199](https://our.internmc.facebook.com/intern/diff/D17512199/) [ghstack-poisoned]
dzhulgakov
approved these changes
Sep 28, 2019
| return super(ConvReLU2d, cls).from_float(mod, qconfig) | ||
|
|
||
| def update_bn_stats(mod): | ||
| if type(mod) in set([ConvBnReLU2d, ConvBn2d]): |
Collaborator
There was a problem hiding this comment.
you can also just do 'hasattr' (in case more modules appear in the future)
Contributor
Author
There was a problem hiding this comment.
Is this safer? We want to modify this field only for specific modules. Tagging based on a specific attribute could be theoretically more risky as we would set freeze_bn to false for any module that has this field. Leaving this as is so that we have explicit control over which modules we apply this modification to.
For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training Differential Revision: [D17512199](https://our.internmc.facebook.com/intern/diff/D17512199/) [ghstack-poisoned]
For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training Differential Revision: [D17512199](https://our.internmc.facebook.com/intern/diff/D17512199/) [ghstack-poisoned]
For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training Differential Revision: [D17512199](https://our.internmc.facebook.com/intern/diff/D17512199/) [ghstack-poisoned]
Contributor
|
This pull request has been merged in 84ee8ac. |
jamesr66a
pushed a commit
that referenced
this pull request
Oct 3, 2019
Summary: Pull Request resolved: #26624 For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training ghstack-source-id: 91008297 Test Plan: buck test caffe2/test:quantization -- --print-passing-details Differential Revision: D17512199 fbshipit-source-id: f7b981e2b1966ab01c4dbb161030177274a998b6
jamesr66a
pushed a commit
that referenced
this pull request
Oct 3, 2019
Summary: Pull Request resolved: #26624 For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training ghstack-source-id: 91008297 Test Plan: buck test caffe2/test:quantization -- --print-passing-details Differential Revision: D17512199 fbshipit-source-id: f7b981e2b1966ab01c4dbb161030177274a998b6
jamesr66a
pushed a commit
that referenced
this pull request
Oct 4, 2019
Summary: Pull Request resolved: #26624 For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training ghstack-source-id: 91008297 Test Plan: buck test caffe2/test:quantization -- --print-passing-details Differential Revision: D17512199 fbshipit-source-id: f7b981e2b1966ab01c4dbb161030177274a998b6
jamesr66a
pushed a commit
that referenced
this pull request
Oct 4, 2019
Summary: Pull Request resolved: #26624 For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training ghstack-source-id: 91008297 Test Plan: buck test caffe2/test:quantization -- --print-passing-details Differential Revision: D17512199 fbshipit-source-id: f7b981e2b1966ab01c4dbb161030177274a998b6
soumith
pushed a commit
that referenced
this pull request
Oct 7, 2019
Summary: Pull Request resolved: #26624 For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training ghstack-source-id: 91008297 Test Plan: buck test caffe2/test:quantization -- --print-passing-details Differential Revision: D17512199 fbshipit-source-id: f7b981e2b1966ab01c4dbb161030177274a998b6
pdlive215
pushed a commit
to pdlive215/pytorch
that referenced
this pull request
Nov 27, 2019
Summary: Pull Request resolved: pytorch#26624 For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training ghstack-source-id: 91008297 Test Plan: buck test caffe2/test:quantization -- --print-passing-details Differential Revision: D17512199 fbshipit-source-id: f7b981e2b1966ab01c4dbb161030177274a998b6
xxtEchjovs44
pushed a commit
to xxtEchjovs44/pytorch
that referenced
this pull request
Jan 29, 2020
Pull Request resolved: pytorch/pytorch#26624 For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training ghstack-source-id: 90704775 Differential Revision: [D17512199](https://our.internmc.facebook.com/intern/diff/D17512199/)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
For QAT we need to be able to control batch norm for all modules from the top. Adding helper functions to enable/disable batch norm freezing during training
Differential Revision: D17512199