Add scalar.conj() and update backward formulas for add and sub by anjali411 · Pull Request #46596 · pytorch/pytorch

anjali411 · 2020-10-20T17:17:15Z

Added conj method for scalar similar to numpy.
Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex.
Enabled complex backward for nonzero (no formula update needed).

Stack from ghstack:

Add scalar.conj() and update backward formulas for add and sub #46596 Add scalar.conj() and update backward formulas for add and sub

Differential Revision: D24529227

[ghstack-poisoned]

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). [ghstack-poisoned]

dr-ci · 2020-10-20T17:37:25Z

💊 CI failures summary and remediations

As of commit 5b6d3a7 (more details on the Dr. CI page):

✅ None of the CI failures appear to be your fault 💚

1/1 tentatively recognized as flaky ❄️
- Click here to rerun these jobs

❄️ 1 failure tentatively classified as flaky

but reruns have not yet been triggered to confirm:

pytorch_linux_xenial_py3_clang5_android_ndk_r19c_vulkan_x86_32_build (1/1)

Step: "Build" (full log | diagnosis details | 🔁 rerun) ❄️

Error response from daemon: received unexpected HTTP status: 500 Internal Server Error

Parallel backend flags:  
DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c:4d22616f4e3faf830d069137acd2208e7a023625 
Error response from daemon: received unexpected HTTP status: 500 Internal Server Error

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 19 times.

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). [ghstack-poisoned]

ghstack-source-id: 422aeb2 Pull Request resolved: #46596

ezyang · 2020-10-21T19:40:19Z

+  if (isComplex()) {
+    return Scalar(std::conj(v.z));
+  } else if (isFloatingPoint()) {
+    return Scalar(v.d);


can't you just return *this here?

ezyang

At some point when you're not in a rush, it would be good to use @robieta's tools for instruction counts to assess what the non-complex overheads of the new complex idioms and conjugations are. It will be a little tricky to setup since you have to go through autograd, but we might be able to setup a microbenchmark where we just call the microbenchmark directly. That will help us understand how urgent extra optimizing here is (which I know @ngimel is nervous about the conj additions here.)

mruberry

Test changes look good. Test failure doesn't appear to be related.

gchanan · 2020-10-21T22:25:28Z

 - name: sub.Tensor(Tensor self, Tensor other, *, Scalar alpha=1) -> Tensor
-  self: grad
-  other: -grad * alpha
+  self: handle_r_to_c(self.scalar_type(), grad)


why do we need this? This seems pretty similar to

pytorch/torch/csrc/autograd/engine.cpp

Line 611 in e02a3e1

grad = grad.to(c10::typeMetaToScalarType(metadata.options().dtype()));

-- is it because to doesn't work c->r?

.to works for C->R. currently handle_r_to_c returns a (real) view of the final gradient value in case the gradient is complex and the input is not.

anjali411 · 2020-10-22T02:03:30Z

At some point when you're not in a rush, it would be good to use @robieta's tools for instruction counts to assess what the non-complex overheads of the new complex idioms and conjugations are. It will be a little tricky to setup since you have to go through autograd, but we might be able to setup a microbenchmark where we just call the microbenchmark directly. That will help us understand how urgent extra optimizing here is (which I know @ngimel is nervous about the conj additions here.)

I think that's a great idea! is there a doc to help with the setup?

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). [ghstack-poisoned]

ghstack-source-id: 3fb369b Pull Request resolved: #46596

ezyang · 2020-10-23T02:26:19Z

Hopefully Taylor's talk today should help? He's also been working on written docs as well.

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). Differential Revision: [D24529227](https://our.internmc.facebook.com/intern/diff/D24529227) [ghstack-poisoned]

ghstack-source-id: a94571f Pull Request resolved: #46596

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). Differential Revision: [D24529227](https://our.internmc.facebook.com/intern/diff/D24529227) [ghstack-poisoned]

ghstack-source-id: 1ddd262 Pull Request resolved: #46596

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). Differential Revision: [D24529227](https://our.internmc.facebook.com/intern/diff/D24529227) [ghstack-poisoned]

ghstack-source-id: 459ce18 Pull Request resolved: #46596

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). Differential Revision: [D24529227](https://our.internmc.facebook.com/intern/diff/D24529227) [ghstack-poisoned]

ghstack-source-id: 9a8402e Pull Request resolved: #46596

facebook-github-bot · 2020-11-03T01:13:36Z

@anjali411 merged this pull request in cedeee2.

albanD · 2021-03-02T21:16:50Z

        ('add', (), ((S, S, S),), 'scalar_broadcast_lhs', (True,)),
        ('add', (S, S, S), (3.14,), 'constant', (True,)),
        ('add', (), (3.14,), 'scalar_constant', (True,)),
+        ('add', (S, S, S), (3.14j,), 'complex_scalar_constant', (True,)),


Does this actually calls into the scalar overload? I think that the python binding is converting these to Tensors all the time.

You can double check this by doing:

import torch a = torch.rand(10, requires_grad=True) print((a + a).grad_fn) print((a + 1.).grad_fn)

That will print

<AddBackward0 object at 0x7fc1c1446150> <AddBackward0 object at 0x7fc1c2f14f50>

The important part is the 0 here that means that both are actually using the 0th overload (the one for Tensor inputs).

…ch#46596) Summary: Pull Request resolved: pytorch#46596 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). Test Plan: Imported from OSS Reviewed By: glaringlee Differential Revision: D24529227 Pulled By: anjali411 fbshipit-source-id: da871309a6decf5a4ab5c561d5ab35fc66b5273d

Add scalar.conj() and update backward formulas for add and sub

b6be40c

[ghstack-poisoned]

anjali411 requested review from albanD and gchanan October 20, 2020 17:17

anjali411 added complex_autograd module: complex Related to complex number support in PyTorch labels Oct 20, 2020

anjali411 requested a review from ezyang October 20, 2020 17:22

anjali411 added a commit that referenced this pull request Oct 20, 2020

Add scalar.conj() and update backward formulas for add and sub

79cb7e2

ghstack-source-id: 422aeb2 Pull Request resolved: #46596

anjali411 mentioned this pull request Oct 20, 2020

Make torch.det() support complex input. #45980

Closed

anjali411 requested review from mruberry and zou3519 October 21, 2020 00:45

ezyang reviewed Oct 21, 2020

View reviewed changes

ezyang approved these changes Oct 21, 2020

View reviewed changes

mruberry approved these changes Oct 21, 2020

View reviewed changes

gchanan reviewed Oct 21, 2020

View reviewed changes

Comment thread aten/src/ATen/test/scalar_test.cpp Outdated

gchanan reviewed Oct 21, 2020

View reviewed changes

Comment thread tools/autograd/gen_variable_type.py Outdated

gchanan reviewed Oct 21, 2020

View reviewed changes

anjali411 added a commit that referenced this pull request Oct 22, 2020

Add scalar.conj() and update backward formulas for add and sub

758e59a

ghstack-source-id: 3fb369b Pull Request resolved: #46596

anjali411 added a commit that referenced this pull request Oct 26, 2020

Add scalar.conj() and update backward formulas for add and sub

30359fc

ghstack-source-id: a94571f Pull Request resolved: #46596

anjali411 added a commit that referenced this pull request Oct 27, 2020

Add scalar.conj() and update backward formulas for add and sub

f00ab65

ghstack-source-id: 1ddd262 Pull Request resolved: #46596

anjali411 added a commit that referenced this pull request Oct 28, 2020

Add scalar.conj() and update backward formulas for add and sub

633c8a6

ghstack-source-id: 459ce18 Pull Request resolved: #46596

facebook-github-bot added the cla signed label Oct 30, 2020

anjali411 added a commit that referenced this pull request Nov 2, 2020

Add scalar.conj() and update backward formulas for add and sub

f73ffc9

ghstack-source-id: 9a8402e Pull Request resolved: #46596

facebook-github-bot closed this in cedeee2 Nov 3, 2020

facebook-github-bot added the Merged label Nov 3, 2020

facebook-github-bot deleted the gh/anjali411/66/head branch November 6, 2020 15:19

albanD reviewed Mar 2, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add scalar.conj() and update backward formulas for add and sub#46596

Add scalar.conj() and update backward formulas for add and sub#46596
anjali411 wants to merge 9 commits intogh/anjali411/66/basefrom
gh/anjali411/66/head

anjali411 commented Oct 20, 2020 •

edited

Loading

Uh oh!

dr-ci Bot commented Oct 20, 2020 •

edited

Loading

Uh oh!

ezyang Oct 21, 2020

Uh oh!

ezyang left a comment

Uh oh!

mruberry left a comment

Uh oh!

Uh oh!

Uh oh!

gchanan Oct 21, 2020

Uh oh!

anjali411 Oct 22, 2020

Uh oh!

anjali411 commented Oct 22, 2020 •

edited

Loading

Uh oh!

ezyang commented Oct 23, 2020

Uh oh!

facebook-github-bot commented Nov 3, 2020

Uh oh!

albanD Mar 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

anjali411 commented Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr-ci Bot commented Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

❄️ 1 failure tentatively classified as flaky

pytorch_linux_xenial_py3_clang5_android_ndk_r19c_vulkan_x86_32_build (1/1)

Uh oh!

ezyang Oct 21, 2020

Choose a reason for hiding this comment

Uh oh!

ezyang left a comment

Choose a reason for hiding this comment

Uh oh!

mruberry left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gchanan Oct 21, 2020

Choose a reason for hiding this comment

Uh oh!

anjali411 Oct 22, 2020

Choose a reason for hiding this comment

Uh oh!

anjali411 commented Oct 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ezyang commented Oct 23, 2020

Uh oh!

facebook-github-bot commented Nov 3, 2020

Uh oh!

albanD Mar 2, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

anjali411 commented Oct 20, 2020 •

edited

Loading

dr-ci Bot commented Oct 20, 2020 •

edited

Loading

anjali411 commented Oct 22, 2020 •

edited

Loading