Skip to content

[quant] Add backward compatiblity test#36771

Closed
jerryzh168 wants to merge 1 commit intopytorch:masterfrom
jerryzh168:nbc
Closed

[quant] Add backward compatiblity test#36771
jerryzh168 wants to merge 1 commit intopytorch:masterfrom
jerryzh168:nbc

Conversation

@jerryzh168
Copy link
Copy Markdown
Contributor

@jerryzh168 jerryzh168 commented Apr 16, 2020

Summary:
re-created the same PR: #36639
because ghimport does not support importing binary files right now

Test Plan:
python test/quantization/test_backward_compatibility.py

Reviewers:
.

Subscribers:

Tasks:

Tags:

@jerryzh168
[quant] Add backward compatiblity test
b856ead 
Summary:
re-created the same PR: pytorch#36639
because ghimport does not support importing binary files right now

Test Plan:
python test/quantization/test_backward_compatibility.py

Reviewers:
.

Subscribers:

Tasks:

Tags:
facebook-github-bot
facebook-github-bot reviewed Apr 16, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jerryzh168 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

This was referenced Apr 17, 2020
Closed
Closed
@dr-ci
Copy link
Copy Markdown

dr-ci Bot commented Apr 17, 2020 •
edited
Loading

💊 Build failures summary and remediations

As of commit b856ead (more details on the Dr. CI page):

  • 19/19 failures possibly* introduced in this PR
    • 1/19 non-CircleCI failure(s)

🕵️ 5 new failures recognized by patterns

The following build failures do not appear to be due to upstream breakages:

See CircleCI build pytorch_ios_11_2_1_arm64_build (1/5)

Step: "Build" (full log | pattern match details | 🔁 rerun)

ERROR: test_Embedding_sparse (__main__.TestNN) 
    test_case.assertEqual(test_case._get_parameters(module)[1], d_param) 
  File "/var/lib/jenkins/workspace/test/common.py", line 374, in assertEqual 
    self.assertEqual(x_, y_, prec, message) 
  File "/var/lib/jenkins/workspace/test/common.py", line 354, in assertEqual 
    x = self.safeCoalesce(x) 
  File "/var/lib/jenkins/workspace/test/common.py", line 307, in safeCoalesce 
    new_values = [value_map[idx] for idx in new_indices] 
UnboundLocalError: local variable 'new_indices' referenced before assignment 
 
====================================================================== 
ERROR: test_Embedding_sparse (__main__.TestNN) 
---------------------------------------------------------------------- 
Traceback (most recent call last): 
  File "test_nn.py", line 6547, in <lambda> 
    add(test_name, lambda self, test=test: test(self)) 
  File "/var/lib/jenkins/workspace/test/common_nn.py", line 1002, in __call__ 
    self.test_noncontig(test_case, module, input) 
  File "/var/lib/jenkins/workspace/test/common_nn.py", line 1063, in test_noncontig 
    test_case.assertEqual(test_case._get_parameters(module)[1], d_param) 
  File "/var/lib/jenkins/workspace/test/common.py", line 374, in assertEqual 
    self.assertEqual(x_, y_, prec, message)

See CircleCI build pytorch_bazel_build (2/5)

Step: "Build" (full log | pattern match details | 🔁 rerun)

ERROR: test_Embedding_sparse (__main__.TestNN) 
    test_case.assertEqual(test_case._get_parameters(module)[1], d_param) 
  File "/var/lib/jenkins/workspace/test/common.py", line 374, in assertEqual 
    self.assertEqual(x_, y_, prec, message) 
  File "/var/lib/jenkins/workspace/test/common.py", line 354, in assertEqual 
    x = self.safeCoalesce(x) 
  File "/var/lib/jenkins/workspace/test/common.py", line 307, in safeCoalesce 
    new_values = [value_map[idx] for idx in new_indices] 
UnboundLocalError: local variable 'new_indices' referenced before assignment 
 
====================================================================== 
ERROR: test_Embedding_sparse (__main__.TestNN) 
---------------------------------------------------------------------- 
Traceback (most recent call last): 
  File "test_nn.py", line 6547, in <lambda> 
    add(test_name, lambda self, test=test: test(self)) 
  File "/var/lib/jenkins/workspace/test/common_nn.py", line 1002, in __call__ 
    self.test_noncontig(test_case, module, input) 
  File "/var/lib/jenkins/workspace/test/common_nn.py", line 1063, in test_noncontig 
    test_case.assertEqual(test_case._get_parameters(module)[1], d_param) 
  File "/var/lib/jenkins/workspace/test/common.py", line 374, in assertEqual 
    self.assertEqual(x_, y_, prec, message)

See CircleCI build pytorch_linux_xenial_py3_clang5_android_ndk_r19c_x86_32_build (3/5)

Step: "Build" (full log | pattern match details | 🔁 rerun)

ERROR: test_Embedding_sparse (__main__.TestNN) 
    test_case.assertEqual(test_case._get_parameters(module)[1], d_param) 
  File "/var/lib/jenkins/workspace/test/common.py", line 374, in assertEqual 
    self.assertEqual(x_, y_, prec, message) 
  File "/var/lib/jenkins/workspace/test/common.py", line 354, in assertEqual 
    x = self.safeCoalesce(x) 
  File "/var/lib/jenkins/workspace/test/common.py", line 307, in safeCoalesce 
    new_values = [value_map[idx] for idx in new_indices] 
UnboundLocalError: local variable 'new_indices' referenced before assignment 
 
====================================================================== 
ERROR: test_Embedding_sparse (__main__.TestNN) 
---------------------------------------------------------------------- 
Traceback (most recent call last): 
  File "test_nn.py", line 6547, in <lambda> 
    add(test_name, lambda self, test=test: test(self)) 
  File "/var/lib/jenkins/workspace/test/common_nn.py", line 1002, in __call__ 
    self.test_noncontig(test_case, module, input) 
  File "/var/lib/jenkins/workspace/test/common_nn.py", line 1063, in test_noncontig 
    test_case.assertEqual(test_case._get_parameters(module)[1], d_param) 
  File "/var/lib/jenkins/workspace/test/common.py", line 374, in assertEqual 
    self.assertEqual(x_, y_, prec, message)

See CircleCI build pytorch_linux_xenial_py3_clang5_mobile_custom_build_static (4/5)

Step: "Build" (full log | pattern match details | 🔁 rerun)

ERROR: test_Embedding_sparse (__main__.TestNN) 
    test_case.assertEqual(test_case._get_parameters(module)[1], d_param) 
  File "/var/lib/jenkins/workspace/test/common.py", line 374, in assertEqual 
    self.assertEqual(x_, y_, prec, message) 
  File "/var/lib/jenkins/workspace/test/common.py", line 354, in assertEqual 
    x = self.safeCoalesce(x) 
  File "/var/lib/jenkins/workspace/test/common.py", line 307, in safeCoalesce 
    new_values = [value_map[idx] for idx in new_indices] 
UnboundLocalError: local variable 'new_indices' referenced before assignment 
 
====================================================================== 
ERROR: test_Embedding_sparse (__main__.TestNN) 
---------------------------------------------------------------------- 
Traceback (most recent call last): 
  File "test_nn.py", line 6547, in <lambda> 
    add(test_name, lambda self, test=test: test(self)) 
  File "/var/lib/jenkins/workspace/test/common_nn.py", line 1002, in __call__ 
    self.test_noncontig(test_case, module, input) 
  File "/var/lib/jenkins/workspace/test/common_nn.py", line 1063, in test_noncontig 
    test_case.assertEqual(test_case._get_parameters(module)[1], d_param) 
  File "/var/lib/jenkins/workspace/test/common.py", line 374, in assertEqual 
    self.assertEqual(x_, y_, prec, message)

See CircleCI build pytorch_linux_xenial_py3_clang5_asan_build (5/5)

Step: "Build" (full log | pattern match details | 🔁 rerun)

../../../../usr/local/caffe2/lib/python2.7/dist-packages/caffe2/python/data_parallel_model_test.py::RecurrentNetworkParallelTest::test_equiv_recurrent FAILED [ 6%] 
../../../../usr/local/caffe2/lib/python2.7/dist-packages/caffe2/python/core_test.py::TestInferDevice::test_inject_copy_multi_use SKIPPED [  5%] 
../../../../usr/local/caffe2/lib/python2.7/dist-packages/caffe2/python/core_test.py::TestInferDevice::test_inject_copy_placeholder_ops SKIPPED [  6%] 
../../../../usr/local/caffe2/lib/python2.7/dist-packages/caffe2/python/core_test.py::TestRerouteTensor::test_reroute_tensor PASSED [  6%] 
../../../../usr/local/caffe2/lib/python2.7/dist-packages/caffe2/python/data_parallel_model_test.py::DataParallelModelTest::test_checkpoint_params PASSED [  6%] 
../../../../usr/local/caffe2/lib/python2.7/dist-packages/caffe2/python/data_parallel_model_test.py::DataParallelModelTest::test_device_scope_check PASSED [  6%] 
../../../../usr/local/caffe2/lib/python2.7/dist-packages/caffe2/python/data_parallel_model_test.py::DataParallelModelTest::test_equiv PASSED [  6%] 
../../../../usr/local/caffe2/lib/python2.7/dist-packages/caffe2/python/data_parallel_model_test.py::DataParallelModelTest::test_net_conversion_and_append_net PASSED [  6%] 
../../../../usr/local/caffe2/lib/python2.7/dist-packages/caffe2/python/data_parallel_model_test.py::DataParallelModelTest::test_net_transformer_function PASSED [  6%] 
../../../../usr/local/caffe2/lib/python2.7/dist-packages/caffe2/python/data_parallel_model_test.py::DataParallelModelTest::test_pre_train_synchronization_barrier PASSED [  6%] 
../../../../usr/local/caffe2/lib/python2.7/dist-packages/caffe2/python/data_parallel_model_test.py::DataParallelModelTest::test_synchronization_barrier PASSED [  6%] 
../../../../usr/local/caffe2/lib/python2.7/dist-packages/caffe2/python/data_parallel_model_test.py::RecurrentNetworkParallelTest::test_equiv_recurrent FAILED [  6%] 
 
=================================== FAILURES =================================== 
______________ RecurrentNetworkParallelTest.test_equiv_recurrent _______________ 
 
self = <caffe2.python.data_parallel_model_test.RecurrentNetworkParallelTest testMethod=test_equiv_recurrent> 
 
    def test_equiv_recurrent(self): 
        ''' 
            Test that the model produces exactly same results given 
            total batchsize, independent of number of GPUs/CPUs.

Extra GitHub checks

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

See how this bot performed.

This comment has been revised 1 time.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Apr 17, 2020

@jerryzh168 merged this pull request in 484a00b.

@mruberry
Copy link
Copy Markdown
Collaborator

mruberry commented Apr 17, 2020

Unlanding. This appears to have broken pytorch_macos_10_13_py3_test. Sample log:

Apr 16 20:58:18 Test results will be stored in test-reports/dist-gloo
Apr 16 20:58:18 
Apr 16 20:58:18 Running tests...
Apr 16 20:58:18 ----------------------------------------------------------------------
Apr 16 20:58:18   test_conv2d (__main__.TestSerialization) ... ERROR (0.110s)
Apr 16 20:58:18   test_conv2d_relu (__main__.TestSerialization) ... ERROR (0.007s)
Apr 16 20:58:18   test_conv3d (__main__.TestSerialization) ... ERROR (0.009s)
Apr 16 20:58:18   test_conv3d_relu (__main__.TestSerialization) ... ERROR (0.009s)
Apr 16 20:58:18   test_linear (__main__.TestSerialization) ... ERROR (0.007s)
Apr 16 20:58:18   test_linear_dynamic (__main__.TestSerialization) ... ERROR (0.008s)
Apr 16 20:58:18   test_linear_relu (__main__.TestSerialization) ... ERROR (0.007s)
Apr 16 20:58:18 
Apr 16 20:58:18 ======================================================================
Apr 16 20:58:18 ERROR [0.110s]: test_conv2d (__main__.TestSerialization)
Apr 16 20:58:18 ----------------------------------------------------------------------
Apr 16 20:58:18 Traceback (most recent call last):
Apr 16 20:58:18   File "quantization/test_backward_compatibility.py", line 115, in test_conv2d
Apr 16 20:58:18     groups=1, bias=True, padding_mode="zeros")
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/conv.py", line 316, in __init__
Apr 16 20:58:18     False, _pair(0), groups, bias, padding_mode)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/conv.py", line 51, in __init__
Apr 16 20:58:18     self.set_weight_bias(qweight, bias_float)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/conv.py", line 324, in set_weight_bias
Apr 16 20:58:18     w, b, self.stride, self.padding, self.dilation, self.groups)
Apr 16 20:58:18 RuntimeError: Didn't find engine for operation quantized::conv2d_prepack NoQEngine
Apr 16 20:58:18 
Apr 16 20:58:18 ======================================================================
Apr 16 20:58:18 ERROR [0.007s]: test_conv2d_relu (__main__.TestSerialization)
Apr 16 20:58:18 ----------------------------------------------------------------------
Apr 16 20:58:18 Traceback (most recent call last):
Apr 16 20:58:18   File "quantization/test_backward_compatibility.py", line 127, in test_conv2d_relu
Apr 16 20:58:18     groups=1, bias=True, padding_mode="zeros")
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/intrinsic/quantized/modules/conv_relu.py", line 29, in __init__
Apr 16 20:58:18     padding_mode=padding_mode)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/conv.py", line 316, in __init__
Apr 16 20:58:18     False, _pair(0), groups, bias, padding_mode)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/conv.py", line 51, in __init__
Apr 16 20:58:18     self.set_weight_bias(qweight, bias_float)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/conv.py", line 324, in set_weight_bias
Apr 16 20:58:18     w, b, self.stride, self.padding, self.dilation, self.groups)
Apr 16 20:58:18 RuntimeError: Didn't find engine for operation quantized::conv2d_prepack NoQEngine
Apr 16 20:58:18 
Apr 16 20:58:18 ======================================================================
Apr 16 20:58:18 ERROR [0.009s]: test_conv3d (__main__.TestSerialization)
Apr 16 20:58:18 ----------------------------------------------------------------------
Apr 16 20:58:18 Traceback (most recent call last):
Apr 16 20:58:18   File "quantization/test_backward_compatibility.py", line 139, in test_conv3d
Apr 16 20:58:18     groups=1, bias=True, padding_mode="zeros")
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/conv.py", line 442, in __init__
Apr 16 20:58:18     False, _triple(0), groups, bias, padding_mode)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/conv.py", line 51, in __init__
Apr 16 20:58:18     self.set_weight_bias(qweight, bias_float)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/conv.py", line 450, in set_weight_bias
Apr 16 20:58:18     w, b, self.stride, self.padding, self.dilation, self.groups)
Apr 16 20:58:18 RuntimeError: Didn't find engine for operation quantized::conv2d_prepack NoQEngine
Apr 16 20:58:18 
Apr 16 20:58:18 ======================================================================
Apr 16 20:58:18 ERROR [0.009s]: test_conv3d_relu (__main__.TestSerialization)
Apr 16 20:58:18 ----------------------------------------------------------------------
Apr 16 20:58:18 Traceback (most recent call last):
Apr 16 20:58:18   File "quantization/test_backward_compatibility.py", line 151, in test_conv3d_relu
Apr 16 20:58:18     groups=1, bias=True, padding_mode="zeros")
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/intrinsic/quantized/modules/conv_relu.py", line 70, in __init__
Apr 16 20:58:18     padding_mode=padding_mode)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/conv.py", line 442, in __init__
Apr 16 20:58:18     False, _triple(0), groups, bias, padding_mode)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/conv.py", line 51, in __init__
Apr 16 20:58:18     self.set_weight_bias(qweight, bias_float)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/conv.py", line 450, in set_weight_bias
Apr 16 20:58:18     w, b, self.stride, self.padding, self.dilation, self.groups)
Apr 16 20:58:18 RuntimeError: Didn't find engine for operation quantized::conv2d_prepack NoQEngine
Apr 16 20:58:18 
Apr 16 20:58:18 ======================================================================
Apr 16 20:58:18 ERROR [0.007s]: test_linear (__main__.TestSerialization)
Apr 16 20:58:18 ----------------------------------------------------------------------
Apr 16 20:58:18 Traceback (most recent call last):
Apr 16 20:58:18   File "quantization/test_backward_compatibility.py", line 82, in test_linear
Apr 16 20:58:18     module = nnq.Linear(3, 1, bias_=True, dtype=torch.qint8)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/linear.py", line 136, in __init__
Apr 16 20:58:18     self._packed_params = LinearPackedParams(dtype)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/linear.py", line 20, in __init__
Apr 16 20:58:18     self.set_weight_bias(wq, None)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/linear.py", line 26, in set_weight_bias
Apr 16 20:58:18     self._packed_params = torch.ops.quantized.linear_prepack(weight, bias)
Apr 16 20:58:18 RuntimeError: Didn't find engine for operation quantized::linear_prepack NoQEngine
Apr 16 20:58:18 
Apr 16 20:58:18 ======================================================================
Apr 16 20:58:18 ERROR [0.008s]: test_linear_dynamic (__main__.TestSerialization)
Apr 16 20:58:18 ----------------------------------------------------------------------
Apr 16 20:58:18 Traceback (most recent call last):
Apr 16 20:58:18   File "quantization/test_backward_compatibility.py", line 102, in test_linear_dynamic
Apr 16 20:58:18     module_qint8 = nnqd.Linear(3, 1, bias_=True, dtype=torch.qint8)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/dynamic/modules/linear.py", line 32, in __init__
Apr 16 20:58:18     super(Linear, self).__init__(in_features, out_features, bias_, dtype=dtype)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/linear.py", line 136, in __init__
Apr 16 20:58:18     self._packed_params = LinearPackedParams(dtype)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/linear.py", line 20, in __init__
Apr 16 20:58:18     self.set_weight_bias(wq, None)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/linear.py", line 26, in set_weight_bias
Apr 16 20:58:18     self._packed_params = torch.ops.quantized.linear_prepack(weight, bias)
Apr 16 20:58:18 RuntimeError: Didn't find engine for operation quantized::linear_prepack NoQEngine
Apr 16 20:58:18 
Apr 16 20:58:18 ======================================================================
Apr 16 20:58:18 ERROR [0.007s]: test_linear_relu (__main__.TestSerialization)
Apr 16 20:58:18 ----------------------------------------------------------------------
Apr 16 20:58:18 Traceback (most recent call last):
Apr 16 20:58:18   File "quantization/test_backward_compatibility.py", line 92, in test_linear_relu
Apr 16 20:58:18     module = nniq.LinearReLU(3, 1, bias=True, dtype=torch.qint8)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/intrinsic/quantized/modules/linear_relu.py", line 26, in __init__
Apr 16 20:58:18     super(LinearReLU, self).__init__(in_features, out_features, bias, dtype)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/linear.py", line 136, in __init__
Apr 16 20:58:18     self._packed_params = LinearPackedParams(dtype)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/linear.py", line 20, in __init__
Apr 16 20:58:18     self.set_weight_bias(wq, None)
Apr 16 20:58:18   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/nn/quantized/modules/linear.py", line 26, in set_weight_bias
Apr 16 20:58:18     self._packed_params = torch.ops.quantized.linear_prepack(weight, bias)
Apr 16 20:58:18 RuntimeError: Didn't find engine for operation quantized::linear_prepack NoQEngine
Apr 16 20:58:18 
Apr 16 20:58:18 ----------------------------------------------------------------------
Apr 16 20:58:18 Ran 7 tests in 0.158s
Apr 16 20:58:18 
Apr 16 20:58:18 FAILED (errors=7)

laurentdupin pushed a commit to laurentdupin/pytorch that referenced this pull request Apr 24, 2026
@jerryzh168 @facebook-github-bot
[quant] Add backward compatiblity test (pytorch#36771)
0785684 
Summary:
re-created the same PR: pytorch#36639
because ghimport does not support importing binary files right now
Pull Request resolved: pytorch#36771

Test Plan: python test/quantization/test_backward_compatibility.py

Differential Revision: D21080503

Pulled By: jerryzh168

fbshipit-source-id: 1dca08208bccead60bba03e5fb5d39e1a1d7c20d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@supriyar supriyar supriyar approved these changes

+1 more reviewer

@facebook-github-bot facebook-github-bot facebook-github-bot left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jerryzh168 @facebook-github-bot @mruberry @supriyar