Per channel fake quant by raghuramank100 · Pull Request #26623 · pytorch/pytorch

raghuramank100 · 2019-09-22T08:45:55Z

Stack from ghstack:

Default observer and fake-quant for backends #26627 Default observer and fake-quant for backends
Emulate weight and activation only quant with fake quant, numerics test #26625 Emulate weight and activation only quant with fake quant, numerics test
Quantization aware training: Freeze batch norm support #26624 Quantization aware training: Freeze batch norm support
Per channel fake quant #26623 Per channel fake quant

Per-channel fake quant cpu and cuda operators,
per-channel support in fake quant module,
tests for per-channel fake-quant and serializability of fake quant modules

Differential Revision: D17439406

Per-channel fake quant cpu and cuda operators, per-channel support in fake quant module, tests for per-channel fake-quant and serializability of fake quant modules ghstack-source-id: 90483681 Differential Revision: [D17439406](https://our.internmc.facebook.com/intern/diff/D17439406/) [ghstack-poisoned]

Per-channel fake quant cpu and cuda operators, per-channel support in fake quant module, tests for per-channel fake-quant and serializability of fake quant modules Differential Revision: [D17439406](https://our.internmc.facebook.com/intern/diff/D17439406/) [ghstack-poisoned]

aten/src/ATen/native/quantized/cpu/fake_quantize_per_channel_affine.cpp

jerryzh168 · 2019-09-23T23:42:44Z

aten/src/ATen/native/quantized/cpu/fake_quantize_per_channel_affine.cpp

+    float inv_scale = 1.0f / sc;
+    int64_t z_point = zero_point[i].item().toLong();
+    auto iter = TensorIterator::unary_op(output_slice, input_slice);
+    cpu_kernel(iter, [&](float self) -> float {


should we extract this into a separate function so that we can be sure that fq_per_channel and fq_per_tensor are using the same thing?

aten/src/ATen/native/quantized/cpu/fake_quantize_per_channel_affine.cpp

aten/src/ATen/native/quantized/cuda/fake_quantize_per_channel_affine.cu

jerryzh168

Could you fix the checks for arguments and formatting in tests?

Per-channel fake quant cpu and cuda operators, per-channel support in fake quant module, tests for per-channel fake-quant and serializability of fake quant modules Differential Revision: [D17439406](https://our.internmc.facebook.com/intern/diff/D17439406/) [ghstack-poisoned]

aten/src/ATen/native/quantized/cpu/fake_quantize_per_channel_affine.cpp

dzhulgakov · 2019-09-24T05:14:33Z

aten/src/ATen/native/quantized/cpu/fake_quantize_per_channel_affine.cpp

+    auto input_slice = self.slice(0,i,i+1);
+    auto output_slice = Y.slice(0,i,i+1);
+
+    float sc = scale[i].item().toFloat();


you could use TensorAccessor (it's faster as indexing creating a tensor). But it matters less because you have slicing above that creates tensors anyway

aten/src/ATen/native/quantized/cpu/fake_quantize_per_channel_affine.cpp

test/common_quantized.py

jerryzh168

approve to unblock

Per-channel fake quant cpu and cuda operators, per-channel support in fake quant module, tests for per-channel fake-quant and serializability of fake quant modules Differential Revision: [D17439406](https://our.internmc.facebook.com/intern/diff/D17439406/) [ghstack-poisoned]

Pull Request resolved: #26623 Per-channel fake quant cpu and cuda operators, per-channel support in fake quant module, tests for per-channel fake-quant and serializability of fake quant modules ghstack-source-id: 90980058 ghstack-source-id: 90980058 Differential Revision: [D17439406](https://our.internmc.facebook.com/intern/diff/D17439406/)

Per-channel fake quant cpu and cuda operators, per-channel support in fake quant module, tests for per-channel fake-quant and serializability of fake quant modules Differential Revision: [D17439406](https://our.internmc.facebook.com/intern/diff/D17439406/) [ghstack-poisoned]

facebook-github-bot · 2019-09-30T08:34:20Z

This pull request has been merged in 7dc7075.

Summary: Pull Request resolved: pytorch/pytorch#26623 Per-channel fake quant cpu and cuda operators, per-channel support in fake quant module, tests for per-channel fake-quant and serializability of fake quant modules ghstack-source-id: 91008299 ghstack-source-id: 91008299 Test Plan: buck test mode/dev caffe2/test:fake_quant -- Started new test run: https://our.intern.facebook.com/intern/testinfra/testrun/1970324848875929 ✓ caffe2/test:fake_quant - test_backward_per_tensor (test_fake_quant.TestFakeQuantizePerTensor) 0.242 1/10 (passed) ✓ caffe2/test:fake_quant - test_numerical_consistency_per_tensor (test_fake_quant.TestFakeQuantizePerTensor) 0.204 2/10 (passed) ✓ caffe2/test:fake_quant - test_fq_serializable (test_fake_quant.TestFakeQuantizePerTensor) 0.174 3/10 (passed) ✓ caffe2/test:fake_quant - test_numerical_consistency_per_channel (test_fake_quant.TestFakeQuantizePerChannel) 0.279 4/10 (passed) ✓ caffe2/test:fake_quant - test_forward_per_tensor (test_fake_quant.TestFakeQuantizePerTensor) 0.241 5/10 (passed) ✓ caffe2/test:fake_quant - test_forward_per_channel (test_fake_quant.TestFakeQuantizePerChannel) 0.353 6/10 (passed) ✓ caffe2/test:fake_quant - test_fq_module (test_fake_quant.TestFakeQuantizePerTensor) 0.354 7/10 (passed) ✓ caffe2/test:fake_quant - test_backward_per_channel (test_fake_quant.TestFakeQuantizePerChannel) 0.334 8/10 (passed) ✓ caffe2/test:fake_quant - test_fq_serializable (test_fake_quant.TestFakeQuantizePerChannel) 0.168 9/10 (passed) ✓ caffe2/test:fake_quant - test_fq_module (test_fake_quant.TestFakeQuantizePerChannel) 0.429 10/10 (passed) ✓ caffe2/test:fake_quant - main 0.000 (passed) Differential Revision: D17439406 fbshipit-source-id: 64bfff5e4f40bc2ab8af2b432c7bc33805418077

Summary: Pull Request resolved: #26623 Per-channel fake quant cpu and cuda operators, per-channel support in fake quant module, tests for per-channel fake-quant and serializability of fake quant modules ghstack-source-id: 91008299 ghstack-source-id: 91008299 Test Plan: buck test mode/dev caffe2/test:fake_quant -- Started new test run: https://our.intern.facebook.com/intern/testinfra/testrun/1970324848875929 ✓ caffe2/test:fake_quant - test_backward_per_tensor (test_fake_quant.TestFakeQuantizePerTensor) 0.242 1/10 (passed) ✓ caffe2/test:fake_quant - test_numerical_consistency_per_tensor (test_fake_quant.TestFakeQuantizePerTensor) 0.204 2/10 (passed) ✓ caffe2/test:fake_quant - test_fq_serializable (test_fake_quant.TestFakeQuantizePerTensor) 0.174 3/10 (passed) ✓ caffe2/test:fake_quant - test_numerical_consistency_per_channel (test_fake_quant.TestFakeQuantizePerChannel) 0.279 4/10 (passed) ✓ caffe2/test:fake_quant - test_forward_per_tensor (test_fake_quant.TestFakeQuantizePerTensor) 0.241 5/10 (passed) ✓ caffe2/test:fake_quant - test_forward_per_channel (test_fake_quant.TestFakeQuantizePerChannel) 0.353 6/10 (passed) ✓ caffe2/test:fake_quant - test_fq_module (test_fake_quant.TestFakeQuantizePerTensor) 0.354 7/10 (passed) ✓ caffe2/test:fake_quant - test_backward_per_channel (test_fake_quant.TestFakeQuantizePerChannel) 0.334 8/10 (passed) ✓ caffe2/test:fake_quant - test_fq_serializable (test_fake_quant.TestFakeQuantizePerChannel) 0.168 9/10 (passed) ✓ caffe2/test:fake_quant - test_fq_module (test_fake_quant.TestFakeQuantizePerChannel) 0.429 10/10 (passed) ✓ caffe2/test:fake_quant - main 0.000 (passed) Differential Revision: D17439406 fbshipit-source-id: 64bfff5e4f40bc2ab8af2b432c7bc33805418077

Summary: Pull Request resolved: pytorch#26623 Per-channel fake quant cpu and cuda operators, per-channel support in fake quant module, tests for per-channel fake-quant and serializability of fake quant modules ghstack-source-id: 91008299 ghstack-source-id: 91008299 Test Plan: buck test mode/dev caffe2/test:fake_quant -- Started new test run: https://our.intern.facebook.com/intern/testinfra/testrun/1970324848875929 ✓ caffe2/test:fake_quant - test_backward_per_tensor (test_fake_quant.TestFakeQuantizePerTensor) 0.242 1/10 (passed) ✓ caffe2/test:fake_quant - test_numerical_consistency_per_tensor (test_fake_quant.TestFakeQuantizePerTensor) 0.204 2/10 (passed) ✓ caffe2/test:fake_quant - test_fq_serializable (test_fake_quant.TestFakeQuantizePerTensor) 0.174 3/10 (passed) ✓ caffe2/test:fake_quant - test_numerical_consistency_per_channel (test_fake_quant.TestFakeQuantizePerChannel) 0.279 4/10 (passed) ✓ caffe2/test:fake_quant - test_forward_per_tensor (test_fake_quant.TestFakeQuantizePerTensor) 0.241 5/10 (passed) ✓ caffe2/test:fake_quant - test_forward_per_channel (test_fake_quant.TestFakeQuantizePerChannel) 0.353 6/10 (passed) ✓ caffe2/test:fake_quant - test_fq_module (test_fake_quant.TestFakeQuantizePerTensor) 0.354 7/10 (passed) ✓ caffe2/test:fake_quant - test_backward_per_channel (test_fake_quant.TestFakeQuantizePerChannel) 0.334 8/10 (passed) ✓ caffe2/test:fake_quant - test_fq_serializable (test_fake_quant.TestFakeQuantizePerChannel) 0.168 9/10 (passed) ✓ caffe2/test:fake_quant - test_fq_module (test_fake_quant.TestFakeQuantizePerChannel) 0.429 10/10 (passed) ✓ caffe2/test:fake_quant - main 0.000 (passed) Differential Revision: D17439406 fbshipit-source-id: 64bfff5e4f40bc2ab8af2b432c7bc33805418077

Pull Request resolved: pytorch/pytorch#26623 Per-channel fake quant cpu and cuda operators, per-channel support in fake quant module, tests for per-channel fake-quant and serializability of fake quant modules ghstack-source-id: 90704802 ghstack-source-id: 90704802 Differential Revision: [D17439406](https://our.internmc.facebook.com/intern/diff/D17439406/)

pytorchbot added module: cuda Related to torch.cuda, and CUDA support in general module: internals Related to internal abstractions in c10 and ATen module: operators oncall: quantization Quantization support in PyTorch labels Sep 22, 2019

raghuramank100 requested a review from jerryzh168 September 23, 2019 21:20

raghuramank10000 added 2 commits September 23, 2019 15:48