Migrated hardshrink() to ATen and deprecated nn.Hardshrink() by weiyangfb · Pull Request #8117 · pytorch/pytorch

weiyangfb · 2018-06-04T16:33:52Z

Summary:

Added hardshrink() to ATen (CPU + GPU)
Removed nn.Hardshrink(), but keeping file "aten/src/THNN/generic/HardShrink.c"
Reusing nn.Hardshrink() tests in test_nn and including CUDA tests. Not sure test_nn is a good place for hardshrink() since it is no longer under nn
Added tests in test_torch

This replaces #7695, fixes #4154

Future work:

not supporting default lambd=0.5 in torch.hardshrink() due to a Scalar bug float default values for Scalar in native_functions.yaml don't work #8484

Performance:

CPU forward:

Previous impl:

large_data = torch.zeros(1000, 1000).fill_(0.3).requires_grad_()
def origfn(data):
  f = torch.nn.Hardshrink(0.3)
  large_out = f(data)
%timeit origfn(large_data) 
####################
100 loops, best of 3: 2.15 ms per loop

Current impl:

large_data = torch.zeros(1000, 1000).fill_(0.3).requires_grad_()
def newfn(data):
    large_out = data.hardshrink(0.3)
%timeit newfn(large_data)
####################
100 loops, best of 3: 2.62 ms per loop

CPU backward:

Previous impl:

large_data = torch.zeros(1000, 1000).fill_(0.3).requires_grad_()
def origfn_backward(data):
    f = torch.nn.Hardshrink(0.3)
    large_out = f(data)
    large_out.sum().backward()
%timeit origfn_backward(large_data)
####################
100 loops, best of 3: 5.2 ms per loop

Current impl:

large_data = torch.zeros(1000, 1000).fill_(0.3).requires_grad_()
def newfn_backward(data):
    large_out = data.hardshrink(0.3)
    large_out.sum().backward()
%timeit newfn_backward(large_data)
####################
100 loops, best of 3: 6.47 ms per loop

CUDA:

Current impl:

large_data = torch.zeros(1000, 1000, device=cuda).fill_(0.3)
%timeit data.hardshrink(0.3)
####################
10000 loops, best of 3: 97.1 µs per loop

Benchmark:

large_data = torch.zeros(1000, 1000, device=cuda).fill_(0.3)
%timeit data.mul_(2)
####################
10000 loops, best of 3: 51.6 µs per loop

…; 3. reusing previous tests for nn.Hardshrink() and included CUDA tests at test_nn; 4. default parameter lambda=0.5 is not working yet

aten/src/ATen/native/cuda/Activation.cu

+
+Tensor hardshrink_cuda(const Tensor & self, Scalar lambd) {
+  auto lambd_tensor = at::zeros_like(self).fill_(lambd);
+  auto out_tensor = self.clone();


…hardshrink at test_legacy_nn

test/test_nn.py

                module.half().cuda()
                module(input)
-                for o in module.parameters():
+                for p in module.parameters():


weiyangfb · 2018-06-06T20:12:50Z

The default parameter lambda=0.5 is still not working. Currently I declared it at native_functions.yaml, maybe more work is needed? Also, where should I add warning messages for the deprecation of nn.Hardshrink? @ezyang

ezyang · 2018-06-07T02:45:51Z

Default should work. Can you push with the test uncommented so we can see the failure?

Re deprecation, cc @soumith and @colesbury, but it won't be easy to "deprecate" the legacy_nn codepath in a reasonable way. Maybe best to drop it.

weiyangfb · 2018-06-08T00:30:44Z

Now the failure should appears:

00:21:26 ======================================================================
00:21:26 FAIL: test_hardshrink (test_torch.TestTorch)
00:21:26 ----------------------------------------------------------------------
00:21:26 Traceback (most recent call last):
00:21:26   File "/var/lib/jenkins/workspace/test/test_torch.py", line 5592, in test_hardshrink
00:21:26     self.assertEqual(torch.tensor([1, 0, 0, 0.6]).view(2, 2), data.hardshrink())
00:21:26   File "/var/lib/jenkins/workspace/test/common.py", line 307, in assertEqual
00:21:26     assertTensorsEqual(x, y)
00:21:26   File "/var/lib/jenkins/workspace/test/common.py", line 299, in assertTensorsEqual
00:21:26     self.assertLessEqual(max_err, prec, message)
00:21:26 AssertionError: tensor(0.5000) not less than or equal to 1e-05
00:21:26 
00:21:26 ----------------------------------------------------------------------
00:21:26 lambd = 0.3
00:21:26 lambd = 0.5
00:21:26 lambd = 0
00:21:26 Ran 508 tests in 29.749s
00:21:26 
00:21:26 FAILED (failures=1, skipped=123)
00:21:26 Traceback (most recent call last):
00:21:26   File "test/run_test.py", line 344, in <module>
00:21:26     main()
00:21:26   File "test/run_test.py", line 336, in main
00:21:26     raise RuntimeError(message)
00:21:26 RuntimeError: test_sparse failed!

The 3rd lambd equals 0, where it should be default value 0.5

@ezyang

weiyangfb · 2018-06-12T18:21:10Z

For reasons I don't understand, this PR also allows nn.Hardshrink() to work in both of CPU and CUDA. For instance:

cuda = torch.device('cuda')
data = torch.zeros(2,2, device=cuda).fill_(0.3)
f = torch.nn.Hardshrink(0.3)
f(data)
-------------
tensor([[ 0.,  0.],
        [ 0.,  0.]], device='cuda:0')

Is nn.Hardshrink() borrowing the implementation from torch.hardshrink()? Did I miss something?

ezyang · 2018-06-13T20:31:16Z

Is nn.Hardshrink() borrowing the implementation from torch.hardshrink()? Did I miss something?

That wouldn't surprise me! Have you checked the code?

ngimel · 2018-06-13T20:51:47Z

nn.Hardshrink is using def from functional.py, and you've told functional.py to call torch.hardshrink. All is good, I think.

weiyangfb · 2018-06-14T00:05:52Z

@ezyang @ngimel You got it! Correcting what I said before, I think all looks good now :D

…tive_functions.yaml, and declare it at nn/functional.py

weiyangfb · 2018-06-14T17:19:52Z

@pytorchbot retest this please

weiyangfb · 2018-06-14T18:48:35Z

@pytorchbot retest this please

weiyangfb · 2018-06-14T20:31:31Z

caffe CI test failures look not related, I will merge this PR

aten/src/ATen/native/Activation.cpp

+}
+
+Tensor hardshrink_backward_cpu(const Tensor & grad, const Tensor & self, Scalar lambd) {
+  auto lambd_tensor = lambd.toTensor().toType(self.type().scalarType()).toBackend(self.is_cuda() ? Backend::CUDA : Backend::CPU);


1. added hardshrink() to ATen (CPU + GPU); 2. removed nn.Hardshrink()…

712d886

…; 3. reusing previous tests for nn.Hardshrink() and included CUDA tests at test_nn; 4. default parameter lambda=0.5 is not working yet

weiyangfb requested review from apaszke, colesbury, ezyang, gchanan, soumith and zdevito as code owners June 4, 2018 16:33

ngimel reviewed Jun 4, 2018

View reviewed changes

weiyangfb added 4 commits June 4, 2018 23:35

optimized memory read/write

b3bb062

1. pass in lambd as scalar for CPU/CUDA_apply*; 2. removed tests for …

8a306d2

…hardshrink at test_legacy_nn

fixes test_utils

db233a0

1. replace zeros_like with empty_like; 2. use scalar_cast in cuda

399685b

ezyang reviewed Jun 6, 2018

View reviewed changes

test/test_nn.py

module.half().cuda()

module(input)

for o in module.parameters():

for p in module.parameters():

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

ezyang approved these changes Jun 6, 2018

View reviewed changes

1. printing lambd value; 2. default lambd=0.5 is still failing

f89b36e

getting around Scalar bug buy removing default value of lambd from na…

7306cf8

…tive_functions.yaml, and declare it at nn/functional.py

cleaned up debug printf

00e3f6e

soumith approved these changes Jun 14, 2018

View reviewed changes

soumith merged commit ae55865 into pytorch:master Jun 14, 2018

weiyangfb deleted the migrate_hardshrink_to_ATen branch June 22, 2018 18:12

ssnl reviewed Jul 26, 2018

View reviewed changes

weiyangfb restored the migrate_hardshrink_to_ATen branch July 27, 2018 02:45

weiyangfb deleted the migrate_hardshrink_to_ATen branch July 27, 2018 03:00

Conversation

weiyangfb commented Jun 4, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary:

Future work:

Performance:

CPU forward:

CPU backward:

CUDA:

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

weiyangfb commented Jun 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ezyang commented Jun 7, 2018

Uh oh!

weiyangfb commented Jun 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weiyangfb commented Jun 12, 2018

Uh oh!

ezyang commented Jun 13, 2018

Uh oh!

ngimel commented Jun 13, 2018

Uh oh!

weiyangfb commented Jun 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weiyangfb commented Jun 14, 2018

Uh oh!

weiyangfb commented Jun 14, 2018

Uh oh!

weiyangfb commented Jun 14, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

weiyangfb commented Jun 4, 2018 •

edited

Loading

weiyangfb commented Jun 6, 2018 •

edited

Loading

weiyangfb commented Jun 8, 2018 •

edited

Loading

weiyangfb commented Jun 14, 2018 •

edited

Loading