[quant][graphmode][fx] Add support for one value being quantized with different qconfigs by jerryzh168 · Pull Request #53586 · pytorch/pytorch

jerryzh168 · 2021-03-09T04:27:50Z

Stack from ghstack:

[quant][graphmode][fx][refactor] Quantize by Use of a Tensor instead of Tensor #54928 [quant][graphmode][fx][refactor] Quantize by Use of a Tensor instead of Tensor
[quant][graphmode][fx] Produce torch.cat instead of torch.ops.quantized.cat #54924 [quant][graphmode][fx] Produce torch.cat instead of torch.ops.quantized.cat
[quant][graphmode][fx] Optimize cat #54813 [quant][graphmode][fx] Optimize cat
[nn] Add remove_duplicate option for named_modules #54812 [nn] Add allow_duplicate option for named_modules
[quant][graphmode][fx][refactor] Factor out insert_observers_for_model to a separate function #54733 [quant][graphmode][fx][refactor] Factor out insert_observers_for_model to a separate function
[quant][graphmode][fx] Separate handling Copy operator to a helper function #54644 [quant][graphmode][fx] Separate handling Copy operator to a helper function
[quant][fx][graphmode][refactor] Change activation_post_process_map to track the observer name instead #54643 [quant][fx][graphmode][refactor] Change activation_post_process_map to track the observer name instead
[quant][graphmode][refactor] Remove reduandent code #54073 [quant][graphmode][refactor] Remove reduandent code
[quant][graphmode][fx] Add support for one value being quantized with different qconfigs #53586 [quant][graphmode][fx] Add support for one value being quantized with different qconfigs

Summary:
Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value
in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16

might do some followup PRs to clean up the hacks and refactor the code.

Test Plan:
python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: D26912676

… different qconfigs Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

… different qconfigs Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 3340917 Pull Request resolved: #53586

facebook-github-bot · 2021-03-09T04:28:09Z

💊 CI failures summary and remediations

As of commit ff09ea5 (more details on the Dr. CI page):

1/1 failures possibly* introduced in this PR
- 1/1 non-scanned failure(s)

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

…ntized with different qconfigs" Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

… different qconfigs Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 8d09c22 Pull Request resolved: #53586

…ntized with different qconfigs" Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D26912676](https://our.internmc.facebook.com/intern/diff/D26912676) [ghstack-poisoned]

… different qconfigs Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 25f2e03 Pull Request resolved: #53586

…ntized with different qconfigs" Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D26912676](https://our.internmc.facebook.com/intern/diff/D26912676) [ghstack-poisoned]

… different qconfigs Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 63e0623 Pull Request resolved: #53586

…ntized with different qconfigs" Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D26912676](https://our.internmc.facebook.com/intern/diff/D26912676) [ghstack-poisoned]

… different qconfigs Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: f1c4408 Pull Request resolved: #53586

…ntized with different qconfigs" Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D26912676](https://our.internmc.facebook.com/intern/diff/D26912676) [ghstack-poisoned]

… different qconfigs Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 05c05b6 Pull Request resolved: #53586

…ntized with different qconfigs" Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D26912676](https://our.internmc.facebook.com/intern/diff/D26912676) [ghstack-poisoned]

… different qconfigs Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: a738021 Pull Request resolved: #53586

…ntized with different qconfigs" Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D26912676](https://our.internmc.facebook.com/intern/diff/D26912676) [ghstack-poisoned]

… different qconfigs Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 6e99e50 Pull Request resolved: #53586

…ntized with different qconfigs" Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D26912676](https://our.internmc.facebook.com/intern/diff/D26912676) [ghstack-poisoned]

vkuzo

makes sense, accepting to unblock, would be great to undo the additional state in the future. One idea I had when reading this PR...thoughts about this?

# map from (node.name, qconfig) to observer
self.activation_post_process_map: Dict[Tuple(str, QConfig), ObserverBase]

If the only use case for multiple observers for a node is multiple qconfigs, perhaps that can be used to deduplicate, and we wouldn't need to add an additional indices variable? I haven't thought through it deeply, more asking for your early opinion.

vkuzo · 2021-03-17T04:37:36Z

+    if not model.training and isinstance(observer, torch.quantization.FixedQParamsFakeQuantize):
+        return


just curious, what's the context on this one? Maybe we can add a comment?

Sure I can add a comment. reason is fixedqparams fake quant only needs to be there for training, we can either do this check here or in _find_quant.

…ntized with different qconfigs" Summary: Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D26912676](https://our.internmc.facebook.com/intern/diff/D26912676) [ghstack-poisoned]

facebook-github-bot · 2021-04-01T00:50:46Z

This pull request has been merged in 55544cb.

… different qconfigs (pytorch#53586) Summary: Pull Request resolved: pytorch#53586 Previously one value can only be quantized to one dtype, this PR adds the support for quantizing one value in the fx graph with multiple dtypes, e.g. first quantize to int8 and then float16 might do some followup PRs to clean up the hacks and refactor the code. Test Plan: python test/test_quantization.py TestQuantizeFx.test_multiple_qconfigs_single_value Imported from OSS Reviewed By: vkuzo Differential Revision: D26912676 fbshipit-source-id: ae3653fd67f05870a3a9e808f491871826c555d5

This was referenced Mar 9, 2021

[quant][graphmode][fix] Handle the case when observed node has no users #53210

Closed

[quant][fx][graphmode][fix] Only insert observers for fixed qparam ops #53330

Closed

[quant][graphmode][fx] Fix a condition check for CopyNode #53585

Closed

facebook-github-bot added cla signed fx labels Mar 9, 2021

vkuzo reviewed Mar 9, 2021

View reviewed changes

Comment thread torch/quantization/fx/quantize.py

vkuzo reviewed Mar 9, 2021

View reviewed changes

Comment thread torch/quantization/fx/quantization_patterns.py Outdated

vkuzo reviewed Mar 10, 2021

View reviewed changes

Comment thread torch/quantization/fx/quantize.py

vkuzo reviewed Mar 10, 2021

View reviewed changes

Comment thread torch/quantization/fx/quantization_patterns.py Outdated

vkuzo reviewed Mar 10, 2021

View reviewed changes

Comment thread torch/quantization/fx/quantize.py

jerryzh168 requested a review from vkuzo March 12, 2021 19:29

jerryzh168 mentioned this pull request Mar 16, 2021

[quant][graphmode][refactor] Remove reduandent code #54073

Closed

vkuzo approved these changes Mar 17, 2021

View reviewed changes

jerryzh168 added 4 commits March 17, 2021 14:24

This was referenced Mar 25, 2021

[quant][fx][graphmode][refactor] Change activation_post_process_map to track the observer name instead #54643

Closed

[quant][graphmode][fx] Separate handling Copy operator to a helper function #54644

Closed

jerryzh168 mentioned this pull request Mar 25, 2021

[quant][graphmode][fx][refactor] Factor out insert_observers_for_model to a separate function #54733

Closed

This was referenced Mar 27, 2021

[nn] Add remove_duplicate option for named_modules #54812

Closed

[quant][graphmode][fx] Optimize cat #54813

Closed

jerryzh168 added 3 commits March 26, 2021 17:13

jerryzh168 mentioned this pull request Mar 29, 2021

[quant][graphmode][fx] Produce torch.cat instead of torch.ops.quantized.cat #54924

Closed

jerryzh168 added 2 commits March 29, 2021 17:38

jerryzh168 mentioned this pull request Mar 30, 2021

[quant][graphmode][fx][refactor] Quantize by Use of a Tensor instead of Tensor #54928

Closed

jerryzh168 added 3 commits March 30, 2021 12:05

facebook-github-bot closed this in 55544cb Apr 1, 2021

facebook-github-bot added the Merged label Apr 1, 2021

facebook-github-bot deleted the gh/jerryzh168/570/head branch April 4, 2021 14:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[quant][graphmode][fx] Add support for one value being quantized with different qconfigs#53586

[quant][graphmode][fx] Add support for one value being quantized with different qconfigs#53586
jerryzh168 wants to merge 24 commits intogh/jerryzh168/570/basefrom
gh/jerryzh168/570/head

jerryzh168 commented Mar 9, 2021 •

edited

Loading

Uh oh!

facebook-github-bot commented Mar 9, 2021 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vkuzo left a comment

Uh oh!

Uh oh!

vkuzo Mar 17, 2021

Uh oh!

jerryzh168 Mar 17, 2021

Uh oh!

facebook-github-bot commented Apr 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		if not model.training and isinstance(observer, torch.quantization.FixedQParamsFakeQuantize):
		return

Conversation

jerryzh168 commented Mar 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Mar 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vkuzo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vkuzo Mar 17, 2021

Choose a reason for hiding this comment

Uh oh!

jerryzh168 Mar 17, 2021

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Apr 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jerryzh168 commented Mar 9, 2021 •

edited

Loading

facebook-github-bot commented Mar 9, 2021 •

edited

Loading