Add forward AD gradcheck by albanD · Pull Request #57633 · pytorch/pytorch

albanD · 2021-05-05T14:37:32Z

Stack from ghstack:

Forward AD formulas batch 3 #58094 Forward AD formulas batch 3
Forward AD formulas batch 2 #57863 Forward AD formulas batch 2
Forward AD formulas batch 1 #57768 Forward AD formulas batch 1
Codegen inplace forward AD formula from out of place one if needed #57767 Codegen inplace forward AD formula from out of place one if needed
Add forward AD test for op info #57701 Add forward AD test for op info
Add forward AD gradcheck #57633 Add forward AD gradcheck

Differential Revision: D28387765

[ghstack-poisoned]

facebook-github-bot · 2021-05-05T14:37:38Z

💊 CI failures summary and remediations

As of commit 085640e (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

ghstack-source-id: 5c5d749 Pull Request resolved: #57633

[ghstack-poisoned]

ghstack-source-id: 16ed433 Pull Request resolved: #57633

soulitzer

Looks pretty good overall! Added some minor comments.

torch/autograd/gradcheck.py

soulitzer · 2021-05-05T18:00:14Z

In terms of further testing, what if you had a wrapper function that unpacked the returned dual tensor from the original function, perturbs the tangent by some eps, and then does make_dual again before returning?

test/test_autograd.py

torch/autograd/gradcheck.py

[ghstack-poisoned]

ghstack-source-id: 45471ba Pull Request resolved: #57633

[ghstack-poisoned]

albanD · 2021-05-06T01:53:44Z

Sorry for the important update but implementing the OpInfo showed some limitations in the implementation that was here and that it wasn't working for most complex workloads.
This updated version works unchanged when OpInfos are added and won't get any other major modification.
If some edge cases I don't know about are not supported here, they will be added in a follow up PR.

ghstack-source-id: 9ed86ab Pull Request resolved: pytorch#57633

[ghstack-poisoned]

albanD · 2021-05-07T15:21:58Z

This is ready for final review

[ghstack-poisoned]

soulitzer

Added a couple more comments. Might be good to make a quick PR testing that CI passes with slow_mode=False if you haven't already.

test/test_autograd.py

torch/autograd/gradcheck.py

[ghstack-poisoned]

albanD · 2021-05-07T21:20:13Z

I tested locally with slow gradcheck flag. How do you do a PR to test with slow? Modify the global setting to False by default and push that? Or there is another way?

[ghstack-poisoned]

ghstack-source-id: ad301a5 Pull Request resolved: pytorch#57633

soulitzer · 2021-05-08T00:47:35Z

Yeah, that was what I was thinking.
Also @gchanan says that we should allow people to test slow_mode in ci-all

Alternatively though, we can have a new branch prefix specifically for new gradcheck
See #57880

albanD · 2021-05-10T15:56:27Z

@soulitzer Running the slow gradcheck version in #57952

ghstack-source-id: ad301a5 Pull Request resolved: #57633

[ghstack-poisoned]

ghstack-source-id: 68f15a0 Pull Request resolved: pytorch#57633

[ghstack-poisoned]

albanD · 2021-05-12T18:17:46Z

@albanD has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

soulitzer

LGTM! Thanks for the addition!

facebook-github-bot · 2021-05-13T01:49:32Z

@albanD merged this pull request in 647282c.

mruberry · 2021-05-13T03:39:56Z

Unlanding this stack as it appears to have hit a logical merge conflict and is breaking multiple builds on the HUD. Sample tests broken:

test_forward_mode_AD_narrow_cpu_complex128
test_forward_mode_AD_narrow_cpu_float64
test_forward_mode_AD_acos_cuda_complex128
test_forward_mode_AD_acosh_cuda_complex128
test_forward_mode_AD_narrow_cuda_complex128
test_forward_mode_AD_narrow_cuda_float64

facebook-github-bot · 2021-05-13T03:42:43Z

This pull request has been reverted by 2d7d692.

Summary: Pull Request resolved: pytorch#57633 Test Plan: Imported from OSS Reviewed By: agolynski Differential Revision: D28387765 Pulled By: albanD fbshipit-source-id: ed15049b5bdacca54f775b50ef166d540ba0b847

Add forward AD gradcheck

e3b229f

[ghstack-poisoned]

albanD requested a review from soulitzer as a code owner May 5, 2021 14:37

facebook-github-bot added the cla signed label May 5, 2021

albanD added a commit that referenced this pull request May 5, 2021

Add forward AD gradcheck

c791a46

ghstack-source-id: 5c5d749 Pull Request resolved: #57633

albanD requested a review from zou3519 May 5, 2021 14:37

Update on "Add forward AD gradcheck"

9f6ddce

[ghstack-poisoned]

Update on "Add forward AD gradcheck"

caa01a5

[ghstack-poisoned]

albanD added a commit that referenced this pull request May 5, 2021

Add forward AD gradcheck

b56b606

ghstack-source-id: 16ed433 Pull Request resolved: #57633

soulitzer reviewed May 5, 2021

View reviewed changes

zou3519 reviewed May 5, 2021

View reviewed changes

test/test_autograd.py Outdated Show resolved Hide resolved

zou3519 reviewed May 5, 2021

View reviewed changes

test/test_autograd.py Show resolved Hide resolved

torch/autograd/gradcheck.py Outdated Show resolved Hide resolved

torch/autograd/gradcheck.py Outdated Show resolved Hide resolved

Update on "Add forward AD gradcheck"

a7d7083

[ghstack-poisoned]

Update on "Add forward AD gradcheck"

f38065c

[ghstack-poisoned]

Update on "Add forward AD gradcheck"

e92e0b7

[ghstack-poisoned]

Update on "Add forward AD gradcheck"

d3dd4f9

[ghstack-poisoned]

Update on "Add forward AD gradcheck"

cfe0bee

[ghstack-poisoned]

albanD added a commit that referenced this pull request May 5, 2021

Add forward AD gradcheck

133072b

ghstack-source-id: 45471ba Pull Request resolved: #57633

Update on "Add forward AD gradcheck"

5241751

[ghstack-poisoned]

albanD mentioned this pull request May 6, 2021

Add forward AD test for op info #57701

Closed

dgl-intel pushed a commit to dgl-intel/pytorch that referenced this pull request May 6, 2021

Add forward AD gradcheck

fa37051

ghstack-source-id: 9ed86ab Pull Request resolved: pytorch#57633

Update on "Add forward AD gradcheck"

e254cad

[ghstack-poisoned]

albanD mentioned this pull request May 6, 2021

Add gradcheck for forward AD by default and functional API #49099

Closed

This was referenced May 6, 2021

Codegen inplace forward AD formula from out of place one if needed #57767

Closed

Forward AD formulas batch 1 #57768

Closed

Update on "Add forward AD gradcheck"

cb72e75

[ghstack-poisoned]

soulitzer reviewed May 7, 2021

View reviewed changes

test/test_autograd.py Outdated Show resolved Hide resolved

torch/autograd/gradcheck.py Show resolved Hide resolved

Update on "Add forward AD gradcheck"

1ac7fb2

[ghstack-poisoned]

albanD mentioned this pull request May 7, 2021

Forward AD formulas batch 2 #57863

Closed

Update on "Add forward AD gradcheck"

cda5805

[ghstack-poisoned]

dgl-intel pushed a commit to dgl-intel/pytorch that referenced this pull request May 7, 2021

Add forward AD gradcheck

59fc15e

ghstack-source-id: ad301a5 Pull Request resolved: pytorch#57633

albanD added a commit that referenced this pull request May 10, 2021

Add forward AD gradcheck

ad0b6e7

ghstack-source-id: ad301a5 Pull Request resolved: #57633

Update on "Add forward AD gradcheck"

9def531

[ghstack-poisoned]

albanD mentioned this pull request May 11, 2021

Forward AD formulas batch 3 #58094

Closed

albanD added a commit to albanD/pytorch that referenced this pull request May 11, 2021

Add forward AD gradcheck

826fa3f

ghstack-source-id: 68f15a0 Pull Request resolved: pytorch#57633

Update on "Add forward AD gradcheck"

085640e

[ghstack-poisoned]

soulitzer self-requested a review May 12, 2021 19:45

soulitzer approved these changes May 12, 2021

View reviewed changes

facebook-github-bot closed this in 647282c May 13, 2021

facebook-github-bot added the Merged label May 13, 2021

facebook-github-bot added the Reverted label May 13, 2021

facebook-github-bot deleted the gh/albanD/87/head branch May 16, 2021 14:15

Conversation

albanD commented May 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented May 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

Uh oh!

soulitzer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

soulitzer commented May 5, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

albanD commented May 6, 2021

Uh oh!

albanD commented May 7, 2021

Uh oh!

soulitzer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

albanD commented May 7, 2021

Uh oh!

soulitzer commented May 8, 2021

Uh oh!

albanD commented May 10, 2021

Uh oh!

albanD commented May 12, 2021

Uh oh!

soulitzer left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented May 13, 2021

Uh oh!

mruberry commented May 13, 2021

Uh oh!

facebook-github-bot commented May 13, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

albanD commented May 5, 2021 •

edited

Loading

facebook-github-bot commented May 5, 2021 •

edited

Loading