speed up hardswish and hardsigmoid tests by vkuzo · Pull Request #38256 · pytorch/pytorch

vkuzo · 2020-05-11T18:37:11Z

Stack from ghstack:

speed up hardswish and hardsigmoid tests #38256 speed up hardswish and hardsigmoid tests

Summary:

Removes hypothesis to speed these tests up, as these tests were flagged as top slow
tests in CI. At the same time, combines the fbgemm and qnnpack test
cases together for better reuse.

Test Plan:

python test/test_quantization.py TestQuantizedOps.test_hardswish
python test/test_quantization.py TestQuantizedOps.test_qhardsigmoid

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: D21506831

Summary: Removes hypothesis to speed these tests up, as these tests were flagged as top slow tests in CI. At the same time, combines the fbgemm and qnnpack test cases together for better reuse. Test Plan: ``` python test/test_quantization.py TestQuantizedOps.test_hardswish python test/test_quantization.py TestQuantizedOps.test_qhardsigmoid ``` Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

Summary: Removes hypothesis to speed these tests up, as these tests were flagged as top slow tests in CI. At the same time, combines the fbgemm and qnnpack test cases together for better reuse. Test Plan: ``` python test/test_quantization.py TestQuantizedOps.test_hardswish python test/test_quantization.py TestQuantizedOps.test_qhardsigmoid ``` Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 94bf814 Pull Request resolved: #38256

supriyar · 2020-05-11T18:39:55Z

+        max_sides = (3, 5)
+        side_lens = (1, 7, 8)
+        torch_types = (torch.quint8, torch.qint8)
+        engines = ('fbgemm', 'qnnpack')


Maybe use @override_qengines instead?

supriyar · 2020-05-11T18:41:30Z

+
+                qY = torch.nn.quantized.functional.hardsigmoid(qX)
+                self.assertEqual(qY, qY_hat,
+                                 message="Hardsigmoid failed: {} vs. {}".format(qY, qY_hat))


Might help for debugging if we specify which backend this failed on.

dr-ci · 2020-05-11T19:09:58Z

💊 CI failures summary and remediations

As of commit b467758 (more details on the Dr. CI page):

1/1 failures possibly* introduced in this PR
- 1/1 non-CircleCI failure(s)

ci.pytorch.org: 1 failed

Failed: pr/py3.6-clang7-rocmdeb-ubuntu16.04

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

See how this bot performed.

This comment has been revised 14 times.

malfet · 2020-05-11T19:22:43Z

Change looks good to me, but outside of CI environment test already finish in a reasonable amount of time:

$ python test/test_quantization.py TestQuantizedOps.test_hardswish TestQuantizedOps.test_qhardsigmoid -v
test_hardswish (quantization.test_quantized_op.TestQuantizedOps) ... /home/nshulga/git/pytorch/test/quantization/test_quantized_op.py:101: UserWarning: 0quantize_tensor_per_tensor_affine current rounding mode is not set to round-to-nearest-ties-to-even (FE_TONEAREST). This will cause accuracy issues in quantized models. (Triggered internally at  /pytorch/aten/src/ATen/native/quantized/affine_quantizer.cpp:25.)
  dtype=torch_type)
ok
test_qhardsigmoid (quantization.test_quantized_op.TestQuantizedOps) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.619s

OK

malfet · 2020-05-11T19:33:09Z

I've tried selecting different profiles, as defined in https://github.com/pytorch/pytorch/blob/master/torch/testing/_internal/common_utils.py#L659 but still can not get to 120 sec times reported during CircleCI runs

Summary: Removes hypothesis to speed these tests up, as these tests were flagged as top slow tests in CI. At the same time, combines the fbgemm and qnnpack test cases together for better reuse. Test Plan: ``` python test/test_quantization.py TestQuantizedOps.test_hardswish python test/test_quantization.py TestQuantizedOps.test_qhardsigmoid ``` Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D21506831](https://our.internmc.facebook.com/intern/diff/D21506831) [ghstack-poisoned]

Summary: Removes hypothesis to speed these tests up, as these tests were flagged as top slow tests in CI. At the same time, combines the fbgemm and qnnpack test cases together for better reuse. Test Plan: ``` python test/test_quantization.py TestQuantizedOps.test_hardswish python test/test_quantization.py TestQuantizedOps.test_qhardsigmoid ``` Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: c510f9f Pull Request resolved: #38256

supriyar · 2020-05-12T03:20:09Z

+            if torch.backends.quantized.engine == 'qnnpack' and torch_type != torch.quint8:
+                continue
+
+            with override_quantized_engine(engine):


This can be removed as well. The override will take care of it at the test level.

malfet · 2020-05-12T03:26:41Z

I know why tests can be slow when run by CI, but fast during single runs:
hu.array_shapes(1, 8, 1, 8) returns randomized tensor shape which can be as large as 16 million elements (i.e. 8**8)
During individual test runs those values are always the same(since we fix the seed), and by chance quite small.
But since the seed is not reset between the runs it can generate a very different (and apparently quite large) tensor shapes when run in sequence with other tests.

malfet · 2020-05-12T18:00:36Z

@vkuzo you might need to rebase your change against latest master as I've landed a PR that adds limit to max number of element for hypothesis generated tensors (and it touches the same files are your diff)

vkuzo · 2020-05-12T22:49:47Z

I know why tests can be slow when run by CI, but fast during single runs:
hu.array_shapes(1, 8, 1, 8) returns randomized tensor shape which can be as large as 16 million elements (i.e. 8**8)
During individual test runs those values are always the same(since we fix the seed), and by chance quite small.
But since the seed is not reset between the runs it can generate a very different (and apparently quite large) tensor shapes when run in sequence with other tests.

thanks for the find, makes sense!

Summary: Removes hypothesis to speed these tests up, as these tests were flagged as top slow tests in CI. At the same time, combines the fbgemm and qnnpack test cases together for better reuse. Test Plan: ``` python test/test_quantization.py TestQuantizedOps.test_hardswish python test/test_quantization.py TestQuantizedOps.test_qhardsigmoid ``` Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D21506831](https://our.internmc.facebook.com/intern/diff/D21506831) [ghstack-poisoned]

Summary: Removes hypothesis to speed these tests up, as these tests were flagged as top slow tests in CI. At the same time, combines the fbgemm and qnnpack test cases together for better reuse. Test Plan: ``` python test/test_quantization.py TestQuantizedOps.test_hardswish python test/test_quantization.py TestQuantizedOps.test_qhardsigmoid ``` Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 85abe95 Pull Request resolved: #38256

facebook-github-bot · 2020-05-14T00:12:32Z

This pull request has been merged in ff76de8.

Summary: Pull Request resolved: pytorch#38256 Removes hypothesis to speed these tests up, as these tests were flagged as top slow tests in CI. At the same time, combines the fbgemm and qnnpack test cases together for better reuse. Test Plan: ``` python test/test_quantization.py TestQuantizedOps.test_hardswish python test/test_quantization.py TestQuantizedOps.test_qhardsigmoid ``` Imported from OSS Differential Revision: D21506831 fbshipit-source-id: 9ff70e4ec7ae30b6948fe808878f0187e631f4d8

vkuzo requested review from malfet and supriyar May 11, 2020 18:37

supriyar reviewed May 11, 2020

View reviewed changes

malfet approved these changes May 12, 2020

View reviewed changes

supriyar reviewed May 12, 2020

View reviewed changes

facebook-github-bot closed this in ff76de8 May 13, 2020

facebook-github-bot added the merged label May 14, 2020

facebook-github-bot deleted the gh/vkuzo/63/head branch May 17, 2020 14:18

mruberry added the Merged label Oct 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

speed up hardswish and hardsigmoid tests#38256

speed up hardswish and hardsigmoid tests#38256
vkuzo wants to merge 3 commits intogh/vkuzo/63/basefrom
gh/vkuzo/63/head

vkuzo commented May 11, 2020 •

edited

Loading

Uh oh!

supriyar May 11, 2020

Uh oh!

supriyar May 11, 2020

Uh oh!

dr-ci Bot commented May 11, 2020 •

edited

Loading

Uh oh!

malfet commented May 11, 2020

Uh oh!

malfet commented May 11, 2020

Uh oh!

supriyar May 12, 2020

Uh oh!

malfet commented May 12, 2020

Uh oh!

malfet commented May 12, 2020

Uh oh!

vkuzo commented May 12, 2020

Uh oh!

facebook-github-bot commented May 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

vkuzo commented May 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

supriyar May 11, 2020

Choose a reason for hiding this comment

Uh oh!

supriyar May 11, 2020

Choose a reason for hiding this comment

Uh oh!

dr-ci Bot commented May 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

ci.pytorch.org: 1 failed

Uh oh!

malfet commented May 11, 2020

Uh oh!

malfet commented May 11, 2020

Uh oh!

supriyar May 12, 2020

Choose a reason for hiding this comment

Uh oh!

malfet commented May 12, 2020

Uh oh!

malfet commented May 12, 2020

Uh oh!

vkuzo commented May 12, 2020

Uh oh!

facebook-github-bot commented May 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vkuzo commented May 11, 2020 •

edited

Loading

dr-ci Bot commented May 11, 2020 •

edited

Loading