Add mish activation function #58648

Adnios · 2021-05-20T07:57:07Z

See issus: #58375

Signed-off-by: Adnios <2780199647@qq.com>

facebook-github-bot · 2021-05-20T07:57:11Z

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

facebook-github-bot · 2021-05-20T07:57:13Z

💊 CI failures summary and remediations

As of commit a5635db (more details on the Dr. CI page):

2/2 failures introduced in this PR

🕵️ 2 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_xla_linux_bionic_py3_6_clang9_build (1/2)

Step: "(Optional) Merge target branch" (full log | diagnosis details | 🔁 rerun)

Automatic merge failed; fix conflicts and then commit the result.

CONFLICT (add/add): Merge conflict in .github/scripts/generate_ci_workflows.py
Auto-merging .github/scripts/generate_ci_workflows.py
CONFLICT (add/add): Merge conflict in .github/scale-config.yml
Auto-merging .github/scale-config.yml
CONFLICT (add/add): Merge conflict in .circleci/scripts/binary_linux_test.sh
Auto-merging .circleci/scripts/binary_linux_test.sh
CONFLICT (add/add): Merge conflict in .circleci/config.yml
Auto-merging .circleci/config.yml
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/windows_build_definitions.py
Auto-merging .circleci/cimodel/data/windows_build_definitions.py
Automatic merge failed; fix conflicts and then commit the result.


Exited with code exit status 1

pytorch_linux_xenial_py3_6_gcc5_4_build (2/2)

Step: "(Optional) Merge target branch" (full log | diagnosis details | 🔁 rerun)

Automatic merge failed; fix conflicts and then commit the result.

CONFLICT (add/add): Merge conflict in .github/scripts/generate_ci_workflows.py
Auto-merging .github/scripts/generate_ci_workflows.py
CONFLICT (add/add): Merge conflict in .github/scale-config.yml
Auto-merging .github/scale-config.yml
CONFLICT (add/add): Merge conflict in .circleci/scripts/binary_linux_test.sh
Auto-merging .circleci/scripts/binary_linux_test.sh
CONFLICT (add/add): Merge conflict in .circleci/config.yml
Auto-merging .circleci/config.yml
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/windows_build_definitions.py
Auto-merging .circleci/cimodel/data/windows_build_definitions.py
Automatic merge failed; fix conflicts and then commit the result.


Exited with code exit status 1

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

facebook-github-bot · 2021-05-20T09:52:53Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

facebook-github-bot · 2021-05-20T10:09:26Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

Signed-off-by: Adnios <2780199647@qq.com>

ezyang · 2021-05-20T14:34:50Z

i simple man, i see structured, i likey (letting @jbschlosser do the actual review)

jbschlosser

Wow, thank you for the beautiful and thorough work! Couple small things and this will be good to go :)

Please add tests for the C++ module / functional forms to test/cpp/api/modules.cpp and test/cpp/api/functional.cpp. Once built, they can be run with build/bin/test_api.
Optionally, adding an OpInfo entry for mish() to the op_db list in torch/testing/_internal/common_methods_invocations.py will add a bunch more generic testing.
I don't think it's worth doing for this PR, but our softplus implementation accepts a beta parameter; in the future, it might be worth exposing this as configurable for mish as well.

aten/src/ATen/test/math_kernel_test.cpp

torch/testing/_internal/common_nn.py

aten/src/ATen/native/cpu/Activation.cpp

test/test_unary_ufuncs.py

docs/source/scripts/build_activation_images.py

jbschlosser

Looks great! One tiny lint-related issue and we're good to go :)

aten/src/ATen/native/Activation.h

facebook-github-bot · 2021-05-22T17:47:21Z

@jbschlosser has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

JackCaoG · 2021-05-22T22:03:53Z

@Adnios Could you mark the test with onlyOnCPUAndCUDA so it won't be run with XLA? Xla usually won't try to follow the same runtime error message throw by pytorch core so we asked developer not to run this kind of these against pt/xla.

Adnios · 2021-05-23T00:43:19Z

@Adnios Could you mark the test with onlyOnCPUAndCUDA so it won't be run with XLA? Xla usually won't try to follow the same runtime error message throw by pytorch core so we asked developer not to run this kind of these against pt/xla.

Hello @JackCaoG, thanks for your help!

I just check the log message of pytorch_xla_linux_bionic_py3_6_clang9_test and find the test_silu_inplace_overlap_xla and test_softplus_inplace_overlap_xla are skiped. The test_mish_inplace_overlap_xla should also be skiped. I think I will try to add @onlyOnCPUAndCUDA.

May 22 19:39:42   test_silu_inplace_overlap_xla (__main__.TestNNDeviceTypeXLA) ... skip (0.002s)
May 22 19:39:42   test_smooth_l1_loss_vs_huber_loss_xla (__main__.TestNNDeviceTypeXLA) ... skip (0.003s)
May 22 19:39:42   test_softmax_64bit_indexing_xla (__main__.TestNNDeviceTypeXLA) ... skip (0.002s)
May 22 19:39:42   test_softmax_bfloat16_xla (__main__.TestNNDeviceTypeXLA) ... skip (0.001s)
May 22 19:43:27   test_softmax_results_xla_float32 (__main__.TestNNDeviceTypeXLA) ... ok (224.887s)
May 22 19:43:27   test_softmax_xla_float16 (__main__.TestNNDeviceTypeXLA) ... skip (0.005s)
May 22 19:43:27   test_softmax_xla_float32 (__main__.TestNNDeviceTypeXLA) ... skip (0.003s)
May 22 19:43:27   test_softplus_inplace_overlap_xla (__main__.TestNNDeviceTypeXLA) ... skip (0.001s)

aten/src/ATen/native/cpu/Activation.cpp

codecov · 2021-05-23T08:37:32Z

Codecov Report

Merging #58648 (09d65c0) into master (8e4fc00) will increase coverage by 0.00%.
The diff coverage is 70.90%.

❗ Current head 09d65c0 differs from pull request most recent head 5191a47. Consider uploading reports for the commit 5191a47 to get more accurate results

@@           Coverage Diff           @@
##           master   #58648   +/-   ##
=======================================
  Coverage   76.44%   76.45%           
=======================================
  Files        1999     1999           
  Lines      200570   200625   +55     
=======================================
+ Hits       153335   153385   +50     
- Misses      47235    47240    +5

facebook-github-bot · 2021-05-24T12:50:06Z

@jbschlosser has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

torch/nn/modules/activation.py

Signed-off-by: Adnios <2780199647@qq.com>

facebook-github-bot · 2021-05-25T13:11:01Z

@jbschlosser has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2021-05-25T17:37:43Z

@jbschlosser merged this pull request in 09a8f22.

Summary: See issus: pytorch#58375 Pull Request resolved: pytorch#58648 Reviewed By: gchanan Differential Revision: D28625390 Pulled By: jbschlosser fbshipit-source-id: 23ea2eb7d5b3dc89c6809ff6581b90ee742149f4

vadimkantorov · 2021-05-25T20:29:34Z

aten/src/ATen/native/Activation.cpp

+    const Tensor& input) {
+  auto input_tanh_softplus = at::tanh(at::softplus(input));
+  auto input_sigmoid = at::sigmoid(input);
+  return grad_output * (input_tanh_softplus + (input * input_sigmoid * (1 - input_tanh_softplus * input_tanh_softplus)));


will this create many intermediate copies? could some of these be done inplace?

Summary: See issus: #58375 Pull Request resolved: #58648 Reviewed By: gchanan Differential Revision: D28625390 Pulled By: jbschlosser fbshipit-source-id: 23ea2eb7d5b3dc89c6809ff6581b90ee742149f4 Co-authored-by: Adnios <2780199647@qq.com>

Summary: See issus: pytorch#58375 Pull Request resolved: pytorch#58648 Reviewed By: gchanan Differential Revision: D28625390 Pulled By: jbschlosser fbshipit-source-id: 23ea2eb7d5b3dc89c6809ff6581b90ee742149f4

Adnios added 3 commits May 20, 2021 15:10

add mish activation function

76e733c

update

f833754

Signed-off-by: Adnios <2780199647@qq.com>

update

779e786

Signed-off-by: Adnios <2780199647@qq.com>

Adnios requested review from BowenBao, albanD, ezyang, jbschlosser, neginraoof, soulitzer and spandantiwari as code owners May 20, 2021 07:57

facebook-github-bot added oncall: jit Add this issue/PR to JIT oncall triage queue fx labels May 20, 2021

pytorchbot added the open source label May 20, 2021

facebook-github-bot added the cla signed label May 20, 2021

Adnios added 7 commits May 20, 2021 18:17

Merge branch 'master' of https://github.com/adnios/pytorch

7df85a0

Merge branch 'master' of https://github.com/adnios/pytorch

bc78916

Merge branch 'master' of https://github.com/adnios/pytorch

8030742

update

663d7a8

Signed-off-by: Adnios <2780199647@qq.com>

update

87a6f6b

Signed-off-by: Adnios <2780199647@qq.com>

update

553bb66

update

c5899d4

Signed-off-by: Adnios <2780199647@qq.com>

jbschlosser mentioned this pull request May 20, 2021

Implementation of Mish: Self Regularized Non-Monotonic Activation Function #58640

Closed

ezyang removed their request for review May 20, 2021 14:34

jbschlosser reviewed May 20, 2021

View reviewed changes

aten/src/ATen/test/math_kernel_test.cpp Outdated Show resolved Hide resolved

torch/testing/_internal/common_nn.py Outdated Show resolved Hide resolved

aten/src/ATen/native/cpu/Activation.cpp Outdated Show resolved Hide resolved

test/test_unary_ufuncs.py Outdated Show resolved Hide resolved

jbschlosser reviewed May 20, 2021

View reviewed changes

docs/source/scripts/build_activation_images.py Outdated Show resolved Hide resolved

jbschlosser reviewed May 21, 2021

View reviewed changes

aten/src/ATen/native/Activation.h Outdated Show resolved Hide resolved

Adnios changed the title ~~[WIP] Add mish activation function~~ Add mish activation function May 21, 2021

fix linting issus

d9e0bf2

Adnios mentioned this pull request May 22, 2021

[PT_BREAK] pytorch_xla_linux_bionic_py3_6_clang9_test failing pytorch/xla#2892

Closed

fix typo

2098f98

add @onlyOnCPUAndCUDA to skip xla

09d65c0

venkateshtata reviewed May 23, 2021

View reviewed changes

aten/src/ATen/native/cpu/Activation.cpp Show resolved Hide resolved

Adnios requested a review from jbschlosser May 24, 2021 06:52

jbschlosser reviewed May 24, 2021

View reviewed changes

torch/nn/modules/activation.py Outdated Show resolved Hide resolved

fix paper link

a5635db

Signed-off-by: Adnios <2780199647@qq.com>

jbschlosser self-requested a review May 25, 2021 13:11

jbschlosser approved these changes May 25, 2021

View reviewed changes

facebook-github-bot closed this in 09a8f22 May 25, 2021

facebook-github-bot added the Merged label May 25, 2021

jbschlosser mentioned this pull request May 25, 2021

Support Mish() activation function. #58375

Closed

This was referenced May 25, 2021

Add mish activation function (#58648) #58940

Merged

[v1.9.0] Release Tracker #58518

Closed

vadimkantorov reviewed May 25, 2021

View reviewed changes

digantamisra98 mentioned this pull request Aug 4, 2021

[Feature Request] mish activation function jax-ml/jax#4523

Closed

Add mish activation function #58648

Add mish activation function #58648

Uh oh!

Conversation

Adnios commented May 20, 2021

Uh oh!

facebook-github-bot commented May 20, 2021

Action Required

Process

Uh oh!

facebook-github-bot commented May 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

🕵️ 2 new failures recognized by patterns

pytorch_xla_linux_bionic_py3_6_clang9_build (1/2)

pytorch_linux_xenial_py3_6_gcc5_4_build (2/2)

Uh oh!

facebook-github-bot commented May 20, 2021

Uh oh!

facebook-github-bot commented May 20, 2021

Uh oh!

ezyang commented May 20, 2021

Uh oh!

jbschlosser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbschlosser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

facebook-github-bot commented May 22, 2021

Uh oh!

JackCaoG commented May 22, 2021

Uh oh!

Adnios commented May 23, 2021

Uh oh!

Uh oh!

codecov bot commented May 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

facebook-github-bot commented May 24, 2021

Uh oh!

Uh oh!

facebook-github-bot commented May 25, 2021

Uh oh!

facebook-github-bot commented May 25, 2021

Uh oh!

vadimkantorov May 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

facebook-github-bot commented May 20, 2021 •

edited

Loading

codecov bot commented May 23, 2021 •

edited

Loading

vadimkantorov May 25, 2021 •

edited

Loading