Skip to content

[WIP] Implemented hardshrink in ATen with cuda#7695

Closed
weiyangfb wants to merge 3 commits intopytorch:masterfrom
weiyangfb:migrate_hardshrink_to_ATen_with_CUDA
Closed

[WIP] Implemented hardshrink in ATen with cuda#7695
weiyangfb wants to merge 3 commits intopytorch:masterfrom
weiyangfb:migrate_hardshrink_to_ATen_with_CUDA

Conversation

@weiyangfb
Copy link
Contributor

@weiyangfb weiyangfb commented May 19, 2018 •
edited
Loading

Summary:

  1. fix Implement CUDA Hardshrink #4154, implemented hard_shrink in ATen (CPU + CUDA)
  2. performance test results:
large_data = torch.zeros(1000, 1000, 1000).fill_(0.3).requires_grad_()
def origfn(data):
  f = torch.nn.Hardshrink(0.3)
  large_out = f(data)
%timeit origfn(large_data)

=> 1 loop, best of 3: 3.99 s per loop

large_data = torch.zeros(1000, 1000, 1000).fill_(0.3).requires_grad_()
def newfn(data):
    large_out = data.hard_shrink(0.3)
%timeit newfn(large_data)

=> 1 loop, best of 3: 4.04 s per loop

large_data = torch.zeros(1000, 1000, 1000).fill_(0.3).requires_grad_()
def origfn_backward(data):
    f = torch.nn.Hardshrink(0.3)
    large_out = f(data)
    large_out.sum().backward()
%timeit origfn_backward(large_data)

=> 1 loop, best of 3: 10 s per loop

large_data = torch.zeros(1000, 1000, 1000).fill_(0.3).requires_grad_()
def newfn_backward(data):
    large_out = data.hard_shrink(0.3)
    large_out.sum().backward()
%timeit newfn_backward(large_data)

=> 1 loop, best of 3: 7.78 s per loop

TODO:

  1. fix bug in using default value of Scalar lambda=0.5, currently 0.5 doesn't get picked up as default value. Failing in testcase of test_torch.py

weiyangfb added 2 commits May 18, 2018 17:42
@weiyangfb
hard_shrink is working now
20f2881 
implement hardshrink with CPU/CUDA_tensor_apply*, similar performance as nn.Hardshrink
@weiyangfb
implemented hard_shrink (CPU+CUDA); TODO: 1) fix bug in default param…
585e102 
…eter for Scalar lambda
zou3519
zou3519 reviewed May 21, 2018
View reviewed changes
aten/src/ATen/native/Activation.cpp
const scalar_t& lambda_t_val,
const scalar_t& zero_t_val) {
if (out_t_val >= -lambda_t_val && out_t_val <= lambda_t_val) {
out_t_val = zero_t_val;

This comment was marked as off-topic.

Sign in to view

zou3519
zou3519 reviewed May 21, 2018
View reviewed changes
test/test_torch.py
res = torch.LongTensor((-bignumber,))
self.assertGreater(res.abs()[0], 0)

def test_hardshrink(self):

This comment was marked as off-topic.

Sign in to view

@ssnl
Copy link
Collaborator

ssnl commented Jun 1, 2018

Any updates on this?

@weiyangfb
Copy link
Contributor Author

weiyangfb commented Jun 1, 2018

For some reasons the compilation of this branch is not working correctly for me. One thing might be related to this was at some point I tried to build using a different version of python. I will close this and create a new PR.

@weiyangfb
Copy link
Contributor Author

weiyangfb commented Jun 1, 2018 •
edited
Loading

Do we need to support HalfTensor type for hardshrink? @ssnl @zou3519

@weiyangfb weiyangfb closed this Jun 4, 2018
@ssnl
Copy link
Collaborator

ssnl commented Jun 4, 2018

@weiyangfb Not necessary, but it shouldn't be too hard, right?

@weiyangfb weiyangfb mentioned this pull request Jun 4, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zou3519 zou3519 zou3519 left review comments

@apaszke apaszke Awaiting requested review from apaszke

@colesbury colesbury Awaiting requested review from colesbury

@ezyang ezyang Awaiting requested review from ezyang

@gchanan gchanan Awaiting requested review from gchanan

@soumith soumith Awaiting requested review from soumith

@zdevito zdevito Awaiting requested review from zdevito

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Implement CUDA Hardshrink

3 participants

@weiyangfb @ssnl @zou3519