[pt1][quant] Add QInt32 ScalarType and qint32 data type by jerryzh168 · Pull Request #19816 · pytorch/pytorch

jerryzh168 · 2019-04-27T00:23:04Z

Stack:
    :white_circle: #20107 [pt1][quant] Add dequantize_linear for JIT pass  💚
    :white_circle: #19984 [pt1][quant] Add qint8 type (int8_t)  💚
    :white_circle: #19932 [pt1][quant] Rename qint8 data type  💚
    :black_circle: #19816 [pt1][quant] Add QInt32 ScalarType and qint32 data type  💚

We need this for quantization for bias
add third argument of ScalarType to quantize_linear

Differential Revision: D15094174

Differential Revision: D15094174 Differential Version: 80838655

jerryzh168 · 2019-04-27T00:24:16Z

@gchanan #6593 says it does not support tracing the dtype version of the function, does this still apply?

Differential Revision: D15094174 Differential Version: 80839141

Differential Revision: D15094174 Differential Version: 80909406

Differential Revision: D15094174 Differential Version: 80918409

Differential Revision: D15094174 Differential Version: 80977784

Differential Revision: D15094174 Differential Version: 80996728

Differential Revision: D15094174 Differential Version: 81007920

Differential Revision: D15094174 Differential Version: 81099685

Differential Revision: D15094174 Differential Version: 81173037

jerryzh168 · 2019-05-04T00:38:36Z

@dzhulgakov looks like we need to define the template function in all the callsite?

May 03 19:22:31 CMakeFiles/quantized_test.dir/__/aten/src/ATen/test/quantized_test.cpp.o: In function `TestQTensor_QuantDequantAPIs_Test::TestBody()':
May 03 19:22:31 quantized_test.cpp:(.text+0x1f56): undefined reference to `c10::qint8 at::quantize_val<c10::qint8>(float, int, float)'

Differential Revision: D15094174 Differential Version: 81239113

Differential Revision: D15094174 Differential Version: 81253515

Differential Revision: D15094174 Differential Version: 81263921

Differential Revision: D15094174 Differential Version: 81319070

Differential Revision: D15094174 Differential Version: 81328044

jerryzh168 · 2019-05-08T02:16:27Z

looks like the CI is passing, @dzhulgakov @gchanan @raghuramank100 please review again

Differential Revision: D15094174 Differential Version: 81380836

c10/util/qint8.h

c10/util/qint32.h

aten/src/ATen/quantized/Quantizer.h

aten/src/ATen/quantized/Quantizer.cpp

gchanan · 2019-05-08T19:30:53Z

aten/src/ATen/quantized/Quantizer.cpp

+  for (auto i = 0; i < qtensor.numel(); ++i) {
+    // We need to convert the qint8 value to float to ensure the subtraction
+    // subexpression returns a float
+    rd[i] = (static_cast<float>(qd[i].val_) - zero_point) * scale;


why force either of these to be contiguous?

Right now I'm just assuming everything is contiguous, we can change it later if there is a need

Not assuming, we have .contiguous call before for the input tensor

ok, but we already have a bunch of code for copying and and for doing type conversions. Did you check what that code does?

Do you mean TensorIterators?

Since fbgemm implementation is contiguous only (and we expect that one to be enabled by default), I think it's ok to force contiguous

It seems fine to make the output contiguous (although you still need to check in the ops themselves, because unlike say, MKLDNN, there is no guarantee you didn't do a view after getting the contiguous tensor).

But this forces the input to be contiguous too.

And I still would like to understand how the existing copy/type conversion mechanisms fit into this.

For example, we almost certainly want memory_order support for quantized tensors, right? If it's possible to use the same mechanism, we should get it "for free", but in this way we certainly won't.

I think this should be a separate discussion. Also I don't have enough context on how we should handle copy/type conversion for qtensor yet.. I do plan to implement permute and view on QTensor though.

aten/src/ATen/quantized/Quantizer.cpp

aten/src/ATen/native/native_functions.yaml

aten/src/ATen/Dispatch.h

Differential Revision: D15094174 Differential Version: 81411153

Differential Revision: D15094174 Differential Version: 81423512

c10/util/typeid.h

Differential Revision: D15094174 Differential Version: 81476388

Differential Revision: D15094174 Differential Version: 81478101

Differential Revision: D15094174 Differential Version: 81496498

aten/src/ATen/native/quantized/cpu/qsumrelu.cpp

Differential Revision: D15094174 Differential Version: 81553389

Differential Revision: D15094174 Differential Version: 81642142

dzhulgakov

Looks good modulo a few comments

dzhulgakov · 2019-05-14T07:15:48Z

aten/src/ATen/quantized/Quantizer.cpp

+  for (auto i = 0; i < qtensor.numel(); ++i) {
+    // We need to convert the qint8 value to float to ensure the subtraction
+    // subexpression returns a float
+    rd[i] = (static_cast<float>(qd[i].val_) - zero_point) * scale;


Since fbgemm implementation is contiguous only (and we expect that one to be enabled by default), I think it's ok to force contiguous

dzhulgakov · 2019-05-14T07:17:38Z

aten/src/ATen/quantized/Quantizer.h

 struct CAFFE2_API Quantizer : public c10::intrusive_ptr_target {
  const QScheme qscheme_;
-  explicit Quantizer(QScheme qscheme) : qscheme_(qscheme) {}
+  const ScalarType scalar_type_;


why do you need to keep scalar type in quantizer if it's already present in the tensor?

quantize function in Quantizer only takes a float Tensor as argument, so we need to have this info in Quantizer as well, scalar type is in the output Tensor not input Tensor.

aten/src/ATen/quantized/Quantizer.h

Differential Revision: D15094174 Differential Version: 81670049

Differential Revision: D15094174 Differential Version: 81728373

Summary: Pull Request resolved: pytorch/pytorch#19816 We need this for quantization for bias add third argument of ScalarType to `quantize_linear` Differential Revision: D15094174 fbshipit-source-id: f19ec8f4716cf5fe0aa21b38d45af6d27c9ab377

facebook-github-bot · 2019-05-16T04:11:30Z

This pull request has been merged in abb3698.

Summary: Close #20642 Possibly broken by #19816 Pull Request resolved: #20853 Differential Revision: D15474620 Pulled By: jerryzh168 fbshipit-source-id: 99b52d92a93bac7cab52537f1ebdbd286d4b2cfe

V2: test passes

98aeb2a

Differential Revision: D15094174 Differential Version: 80838655

pytorchbot added module: internals Related to internal abstractions in c10 and ATen module: operators oncall: quantization Quantization support in PyTorch labels Apr 27, 2019

jerryzh168 requested review from gchanan and li-roy April 27, 2019 00:23

jerryzh168 added 3 commits April 26, 2019 17:29

V3: merge with master

88776d1

Differential Revision: D15094174 Differential Version: 80839141

V5: Merge with parent diff changes

b5172c8

Differential Revision: D15094174 Differential Version: 80909406

V6: add dequant

efd9e2a

Differential Revision: D15094174 Differential Version: 80918409

jerryzh168 mentioned this pull request Apr 29, 2019

[pt1][quant] Rename qint8 data type #19932

Closed

jerryzh168 added 2 commits April 30, 2019 13:06

V8: (no description)

21b8d1e

Differential Revision: D15094174 Differential Version: 80977784

V9: (no description)

545b458

Differential Revision: D15094174 Differential Version: 80996728

jerryzh168 mentioned this pull request Apr 30, 2019

[pt1][quant] Add qint8 type (int8_t) #19984

Closed

jerryzh168 added 2 commits April 30, 2019 19:31

V10: (no description)

2851bd7

Differential Revision: D15094174 Differential Version: 81007920

V12: Merge with parent diff changes

4e1de9d

Differential Revision: D15094174 Differential Version: 81099685

jerryzh168 mentioned this pull request May 3, 2019

[pt1][quant] Add dequantize_linear for JIT pass #20107

Closed

V13: CAFFE2_API

295ce54

Differential Revision: D15094174 Differential Version: 81173037

jerryzh168 requested review from dzhulgakov and z-a-f May 4, 2019 00:38

jerryzh168 added 5 commits May 6, 2019 11:08

V14: Merge with parent diff changes

5c10cbd

Differential Revision: D15094174 Differential Version: 81239113

V15: (no description)

cc878f5

Differential Revision: D15094174 Differential Version: 81253515

V16: Merge with parent diff changes

e88ab0a

Differential Revision: D15094174 Differential Version: 81263921

V17: (no description)

dc043fe

Differential Revision: D15094174 Differential Version: 81319070

V18: (no description)

6bd2ad4

Differential Revision: D15094174 Differential Version: 81328044

V19: test comes in later diffs

9553847

Differential Revision: D15094174 Differential Version: 81380836

gchanan requested changes May 8, 2019

View reviewed changes

V20: (no description)

ef0ec4f

Differential Revision: D15094174 Differential Version: 81411153

V21: Merge with parent diff changes

a57ce95

Differential Revision: D15094174 Differential Version: 81423512

jerryzh168 requested a review from gchanan May 9, 2019 01:11

gchanan reviewed May 9, 2019

View reviewed changes

c10/util/typeid.h Show resolved Hide resolved

jerryzh168 added 2 commits May 9, 2019 15:04

V22: (no description)

0d652d0

Differential Revision: D15094174 Differential Version: 81476388

V23: (no description)

3b955c4

Differential Revision: D15094174 Differential Version: 81478101

jerryzh168 requested a review from gchanan May 9, 2019 22:30

V24: (no description)

7cd3aff

Differential Revision: D15094174 Differential Version: 81496498

jerryzh168 commented May 10, 2019

View reviewed changes

aten/src/ATen/native/quantized/cpu/qsumrelu.cpp Outdated Show resolved Hide resolved

jerryzh168 added 2 commits May 10, 2019 17:22

V25: Merge with parent diff changes

3614dd5

Differential Revision: D15094174 Differential Version: 81553389

V26: Merge with parent diff changes

8ff6e26

Differential Revision: D15094174 Differential Version: 81642142

nishantpdce mentioned this pull request May 14, 2019

Add mandatory ScalarType nodes as input to the quant-dequant nodes. #20468

Closed

This was referenced May 14, 2019

Fixes the tests after introduction of qint8 #20473

Closed

Quantized Max Pool op #20474

Closed

nishantpdce mentioned this pull request May 14, 2019

Add quant-dequant nodes for bias. #20045

Closed

dzhulgakov approved these changes May 14, 2019

View reviewed changes

jerryzh168 added 2 commits May 14, 2019 10:51

V27: (no description)

3e7bdc9

Differential Revision: D15094174 Differential Version: 81670049

V28: (no description)

53c0d1a

Differential Revision: D15094174 Differential Version: 81728373

pytorchbot added the module: nn Related to torch.nn label May 14, 2019

facebook-github-bot closed this in abb3698 May 16, 2019

facebook-github-bot added the merged label May 16, 2019

This was referenced May 17, 2019

Build error with MSVC (aten\src\ATen\native\quantized\Copy.cpp) #20642

Closed

Fix build error with MSVC #20853

Closed

ezyang deleted the export-D15094174 branch May 30, 2019 16:02

Conversation

jerryzh168 commented Apr 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jerryzh168 commented Apr 27, 2019

Uh oh!

jerryzh168 commented May 4, 2019

Uh oh!

jerryzh168 commented May 8, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dzhulgakov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

facebook-github-bot commented May 16, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jerryzh168 commented Apr 27, 2019 •

edited

Loading