[pt1][quant] Add quantized::fbgemm_linear_unpack operator for serialization by jianyuh · Pull Request #20721 · pytorch/pytorch

jianyuh · 2019-05-20T20:40:13Z

Stack:
:black_circle: #20721 [pt1][quant] Add quantized::fbgemm_linear_unpack operator for serialization 💚

Pull Request resolved: pytorch/FBGEMM#97

FBGEMM: Add unpack function for PackBMatrix class: Unpack pmat buffer to the origin_buf (Used for the serialization to recover weight matrix).
PyTorch Quantizer: Add quantized::fbgemm_linear_unpack operator for serialization.

Differential Revision: D15314568

Differential Revision: D15314568 Differential Version: 82426298

jerryzh168 · 2019-05-23T21:18:06Z

aten/src/ATen/native/quantized/cpu/qfc_unpack.cpp

+  at::Tensor operator()(at::Tensor packed_weight) {
+    // Pull out the PackBMatrix instance from the owning tensor.
+    auto& pack_ptr = cpp_custom_type_hack::cast<PackedFCWeight>(packed_weight);
+    auto packB = pack_ptr.w.get();


should we check the packed weight is actually int8 here

Do you mean that we need to add an ASSERT statement here? We can add that to check the type to make sure we have c10::qint8 instead of c10::quint8 for the packed_weight tensor.

Differential Revision: D15314568 Differential Version: 83024055

jerryzh168 · 2019-05-30T01:18:03Z

aten/src/ATen/native/quantized/cpu/qlinear_unpack.cpp

+    int8_t* weight_ptr_int8 =
+        reinterpret_cast<int8_t*>(weight_origin.data<c10::qint8>());
+
+    packB->unpack(weight_ptr_int8);


do you have to unpack weight if we store weight in int8?

Yes: pack means the memory layout has been changed. We have to recover the original memory layout from the packed buffers.

jerryzh168 · 2019-05-30T23:16:30Z

aten/src/ATen/native/quantized/cpu/qlinear_unpack.cpp

+    // We make a strong guarantee that models using these operators will have
+    // the same numerics across different machines. Therefore, we do not provide
+    // a fallback path and rather fail loudly if we cannot run FBGEMM.
+    AT_ASSERTM(


nit: TORCH_INTERNAL_ASSERT

jerryzh168 · 2019-05-31T00:30:39Z

test/test_quantized.py

+        ).astype(np.int8)
+
+        W = torch.from_numpy(_dequantize(W_q0, W_scale, W_zp)).to(dtype=torch.float)
+        W_q = W.quantize_linear(scale=W_scale, zero_point=W_zp, dtype=torch.qint8)


could you use torch.quantize_linear(...) here

jerryzh168

lgtm, please address comments before land.

Differential Revision: D15314568 Differential Version: 84027566

…ch#97) Summary: Pull Request resolved: pytorch#97 Pull Request resolved: pytorch/pytorch#20721 - FBGEMM: Add unpack function for PackBMatrix class: Unpack pmat buffer to the origin_buf (Used for the serialization to recover weight matrix). - PyTorch Quantizer: Add quantized::fbgemm_linear_unpack operator for serialization. Reviewed By: zafartahirov Differential Revision: D15314568 fbshipit-source-id: 506506df13457ce1fe6c487bc3c0eae6972bc54a

Differential Revision: D15314568 Differential Version: 84127135

Differential Revision: D15314568 Differential Version: 84137538

…ch#97) Summary: Pull Request resolved: pytorch/FBGEMM#97 Pull Request resolved: pytorch#20721 - FBGEMM: Add unpack function for PackBMatrix class: Unpack pmat buffer to the origin_buf (Used for the serialization to recover weight matrix). - PyTorch Quantizer: Add quantized::fbgemm_linear_unpack operator for serialization. Reviewed By: zafartahirov Differential Revision: D15314568 fbshipit-source-id: 12080c8887ce31dc849d23e132ae1766ac319407

Summary: Pull Request resolved: pytorch/FBGEMM#97 Pull Request resolved: pytorch/pytorch#20721 - FBGEMM: Add unpack function for PackBMatrix class: Unpack pmat buffer to the origin_buf (Used for the serialization to recover weight matrix). - PyTorch Quantizer: Add quantized::fbgemm_linear_unpack operator for serialization. Reviewed By: zafartahirov Differential Revision: D15314568 fbshipit-source-id: 12080c8887ce31dc849d23e132ae1766ac319407

ezyang · 2019-06-04T12:08:21Z

This diff broke CUDA builds:

Jun 04 03:46:33 [ 51%] Building CXX object caffe2/CMakeFiles/caffe2.dir/__/aten/src/ATen/CPUType.cpp.o
Jun 04 03:46:33 /var/lib/jenkins/workspace/aten/src/ATen/native/quantized/cpu/qlinear_unpack.cpp: In member function 'at::Tensor at::native::{anonymous}::QLinearUnpackWeightInt8::operator()(at::Tensor)':
Jun 04 03:46:33 /var/lib/jenkins/workspace/aten/src/ATen/native/quantized/cpu/qlinear_unpack.cpp:37:12: error: 'class fbgemm::PackBMatrix<signed char>' has no member named 'unpack'
Jun 04 03:46:33      packB->unpack(weight_ptr_int8);
Jun 04 03:46:33             ^

jianyuh · 2019-06-04T16:30:08Z

Thanks @ezyang for pointing this out! @bddppq fixed this with #21328. As pointed out by @bddppq, this Diff has Pytorch part and FBGEMM part (https://github.com/pytorch/fbgemm) and the Pytorch part depends on a new api added in the FBGEMM part. In such cases the two parts should be split into two diffs, first land the fbgemm diff first, and then land the pytorch part together with a submodule update. I will pay attention to this next time.

Summary: Pull Request resolved: #97 Pull Request resolved: pytorch/pytorch#20721 - FBGEMM: Add unpack function for PackBMatrix class: Unpack pmat buffer to the origin_buf (Used for the serialization to recover weight matrix). - PyTorch Quantizer: Add quantized::fbgemm_linear_unpack operator for serialization. Reviewed By: zafartahirov Differential Revision: D15314568 fbshipit-source-id: 12080c8887ce31dc849d23e132ae1766ac319407

V5: Initial commit

a2adc68

Differential Revision: D15314568 Differential Version: 82426298

pytorchbot added module: operators oncall: quantization Quantization support in PyTorch labels May 20, 2019

jerryzh168 reviewed May 23, 2019

View reviewed changes

V7: Merge with parent diff changes

4976e05

Differential Revision: D15314568 Differential Version: 83024055

jerryzh168 reviewed May 30, 2019

View reviewed changes

jerryzh168 reviewed May 31, 2019

View reviewed changes

jerryzh168 approved these changes May 31, 2019

View reviewed changes

V10: Merge with parent diff changes

948aa79

Differential Revision: D15314568 Differential Version: 84027566

jianyuh mentioned this pull request Jun 3, 2019

Add quantized::fbgemm_linear_unpack operator for serialization (#20721) pytorch/FBGEMM#97

Closed

jianyuh added 2 commits June 3, 2019 14:22

V16: Merge with parent diff changes

b8a8def

Differential Revision: D15314568 Differential Version: 84127135

V17: (no description)

bdcdb58

Differential Revision: D15314568 Differential Version: 84137538

facebook-github-bot closed this in pytorch/FBGEMM@7786841 Jun 4, 2019

facebook-github-bot added the merged label Jun 4, 2019

ezyang deleted the export-D15314568 branch July 19, 2019 15:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pt1][quant] Add quantized::fbgemm_linear_unpack operator for serialization#20721

[pt1][quant] Add quantized::fbgemm_linear_unpack operator for serialization#20721
jianyuh wants to merge 5 commits intomasterfrom
export-D15314568

jianyuh commented May 20, 2019 •

edited

Loading

Uh oh!

jerryzh168 May 23, 2019

Uh oh!

jianyuh May 30, 2019

Uh oh!

jerryzh168 May 30, 2019

Uh oh!

jianyuh May 30, 2019

Uh oh!

jerryzh168 May 30, 2019

Uh oh!

jianyuh Jun 3, 2019

Uh oh!

jerryzh168 May 31, 2019

Uh oh!

jianyuh Jun 3, 2019

Uh oh!

jerryzh168 left a comment

Uh oh!

ezyang commented Jun 4, 2019

Uh oh!

jianyuh commented Jun 4, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

jianyuh commented May 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jerryzh168 left a comment

Choose a reason for hiding this comment

Uh oh!

ezyang commented Jun 4, 2019

Uh oh!

jianyuh commented Jun 4, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jianyuh commented May 20, 2019 •

edited

Loading