[ONNX] Add onnx::LayerNorm support for version 17 by titaiwangms · Pull Request #84293 · pytorch/pytorch

titaiwangms · 2022-08-30T18:39:20Z

Stack from ghstack (oldest at bottom):

-> [ONNX] Add onnx::LayerNorm support for version 17 #84293

[ghstack-poisoned]

facebook-github-bot · 2022-08-30T18:39:27Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/84293
✖️ Python docs build was skipped
✖️ C++ docs build was skipped
❓Need help or want to give feedback on the CI? Visit our office hours

✅ No Failures (0 Pending)

As of commit 7aa8ac9 (more details on the Dr. CI page):

Expand to see more

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[ghstack-poisoned]

justinchuby · 2022-08-30T19:01:14Z

I would add a test

titaiwangms · 2022-08-30T19:05:51Z

I would add a test

We already have it.

pytorch/test/onnx/test_pytorch_onnx_onnxruntime.py

Lines 3676 to 3679 in 641c395

    
           def test_layer_norm(self): 
        
               model = torch.nn.LayerNorm([10, 10]) 
        
               x = torch.randn(20, 5, 10, 10) 
        
               self.run_test(model, x)

And this PR should pass it.

justinchuby · 2022-08-30T19:33:22Z

torch/onnx/symbolic_opset17.py

+
+
+@symbolic_helper.parse_args("v", "is", "v", "v", "f", "i")
+def layer_norm(g, input, normalized_shape, weight, bias, eps, cudnn_enable):


one more note: Please annotate the type and decorate the function with _beartype.beartype

offline meeting: more _beartype.beartype use case will be introduced in another PR from Justin. I have done annotation and beartype on shared layer norm function.

justinchuby · 2022-08-30T19:34:33Z

torch/onnx/symbolic_opset17.py

+
+
+@symbolic_helper.parse_args("v", "is", "v", "v", "f", "i")
+def layer_norm(g, input, normalized_shape, weight, bias, eps, cudnn_enable):


Suggested change

def layer_norm(g, input, normalized_shape, weight, bias, eps, cudnn_enable):

def layer_norm(g, input, normalized_shape, weight, bias, eps, cudnn_enable):

cudnn_enable is a boolean. Since it is not used, we can annotate as "none" in parse args

Should we use None or keep it as no conversion(use v) in this case?

If "none" is a common usage, I will add it into doc of parse args.

justinchuby · 2022-08-30T19:35:20Z

torch/onnx/symbolic_opset17.py

+        weight,
+        bias,
+        epsilon_f=eps,
+        axis_i=len(normalized_shape),


Can you write a brief comment inline explaining what axis is and why len(normalized_shape) is the same thing?

Good question, I think particularly in this test case, the value happens to match...

I was referencing on this:

pytorch/torch/onnx/symbolic_opset9.py

Line 2309 in 14093b5

axes = [-i for i in range(len(normalized_shape), 0, -1)]

And according to https://github.com/onnx/onnx/blob/main/docs/Changelog.md#LayerNormalization-17

axis : int (default is -1)
The first normalization dimension. If rank(X) is r, axis' allowed range is [-r, r]. Negative value means counting dimensions from the back.

I tried different params for layer_norm and input, but it works so far. @BowenBao is there a test case you have on mind that might fail?

What are the combinations you tried?
I suspect it should be axis_i = -len(normalized_shape). However for this current test case, both axis_i = 2 and axis_i = -2 maps to the same dimension. So I'd recommend trying alternate the rank

model = torch.nn.LayerNorm([10, 10, 10]) x = torch.randn(20, 5, 10, 10, 10) self.run_test(model, x)

[ghstack-poisoned]

ghstack-source-id: 7d6cd16 Pull Request resolved: #84293

[ghstack-poisoned]

ghstack-source-id: b1630e9 Pull Request resolved: #84293

justinchuby · 2022-09-01T00:37:47Z

test/onnx/test_pytorch_onnx_onnxruntime.py

+        # As layer_norm works on the last D dimension, please keep
+        # this test case at least three dimension to prevent the
+        # situation of axis=2 mapping to the same axis as axis=-2
+        model2 = torch.nn.LayerNorm([10, 10, 10])


Any reason this is named model2? For clarity we can just do layer_norm_model = ... or just model

Ahh, I was going to put two tests, and then only left one. I will clean it, and figure out the CI failure with beartype.

justinchuby · 2022-09-01T01:04:36Z

torch/onnx/symbolic_opset9.py

        return res


+@_beartype.beartype


You may remove the beartype check for this one here. I can handle it with #84091

[ghstack-poisoned]

BowenBao · 2022-09-01T18:27:38Z

@AllenTiTaiWang you can get rid of the build CI issue by rebasing with latest master.

[ghstack-poisoned]

ghstack-source-id: 2dfc81e Pull Request resolved: #84293

[ghstack-poisoned]

ghstack-source-id: 3f12925 Pull Request resolved: #84293

justinchuby · 2022-09-03T22:56:06Z

torch/onnx/symbolic_opset17.py

+):
+    # normalized_shape: input shape from an expected input of size
+    # axis: The first normalization dimension.
+    # layer_norm normalizes on the last D dimension,


Suggested change

# layer_norm normalizes on the last D dimension,

# layer_norm normalizes on the last D dimensions,

justinchuby · 2022-09-03T22:56:17Z

torch/onnx/symbolic_opset17.py

+    # normalized_shape: input shape from an expected input of size
+    # axis: The first normalization dimension.
+    # layer_norm normalizes on the last D dimension,
+    # which D is the size of normalized_shape


Suggested change

# which D is the size of normalized_shape

# where D is the size of normalized_shape

[ghstack-poisoned]

ghstack-source-id: 4155b51 Pull Request resolved: #84293

titaiwangms · 2022-09-04T00:03:00Z

@pytorchbot merge -g

pytorchmergebot · 2022-09-04T00:04:32Z

@pytorchbot successfully started a merge job. Check the current status here.
The merge job was triggered with the green (-g) flag. This means that your change will be merged once all checks on your PR have passed (ETA: 0-4 Hours). If this is not the intended behavior, feel free to use some of the other merge options in the wiki.
Please reach out to the PyTorch DevX Team with feedback or questions!

github-actions · 2022-09-04T02:21:16Z

Hey @AllenTiTaiWang.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

Summary: Pull Request resolved: #84293 Approved by: https://github.com/justinchuby, https://github.com/BowenBao Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/7c4c7dafbdf2c41ccd9042f1db4f9f9f01a42f00 Reviewed By: mehtanirav Differential Revision: D39277708 fbshipit-source-id: 3101337ef4d3ff564c5df604afcae4fd86e1a76d

[ONNX] Add onnx::LayerNorm support for version 17

ff30ef9

[ghstack-poisoned]

titaiwangms requested review from BowenBao and shubhambhokare1 as code owners August 30, 2022 18:39

pytorch-bot bot added the release notes: onnx torch.onnx related changes that should show up in the release notes label Aug 30, 2022

titaiwangms mentioned this pull request Aug 30, 2022

[ONNX] Disable autocast cache in exporter #84219

Closed

facebook-github-bot added the cla signed label Aug 30, 2022

Update on "[ONNX] Add onnx::LayerNorm support for version 17"

e32dd0d

[ghstack-poisoned]

pytorchbot added the open source label Aug 30, 2022

justinchuby self-assigned this Aug 30, 2022

justinchuby approved these changes Aug 30, 2022

View reviewed changes

justinchuby reviewed Aug 30, 2022

View reviewed changes

Update on "[ONNX] Add onnx::LayerNorm support for version 17"

b774d41

[ghstack-poisoned]

justinchuby mentioned this pull request Aug 31, 2022

[ONNX] Support opset 17 operators #81075

Closed

33 tasks

Update on "[ONNX] Add onnx::LayerNorm support for version 17"

f891350

[ghstack-poisoned]

Update on "[ONNX] Add onnx::LayerNorm support for version 17"

1ac710d

[ghstack-poisoned]

titaiwangms added 2 commits August 31, 2022 04:57

Update on "[ONNX] Add onnx::LayerNorm support for version 17"

85bdffa

[ghstack-poisoned]

Update on "[ONNX] Add onnx::LayerNorm support for version 17"

602e44d

[ghstack-poisoned]

Update on "[ONNX] Add onnx::LayerNorm support for version 17"

ceb76a6

[ghstack-poisoned]

titaiwangms added a commit that referenced this pull request Aug 31, 2022

[ONNX] Add onnx::LayerNorm support for version 17

77557bd

ghstack-source-id: 7d6cd16 Pull Request resolved: #84293

BowenBao approved these changes Aug 31, 2022

View reviewed changes

Update on "[ONNX] Add onnx::LayerNorm support for version 17"

e16588b

[ghstack-poisoned]

titaiwangms added a commit that referenced this pull request Aug 31, 2022

[ONNX] Add onnx::LayerNorm support for version 17

42c813b

ghstack-source-id: b1630e9 Pull Request resolved: #84293

justinchuby reviewed Sep 1, 2022

View reviewed changes

Update on "[ONNX] Add onnx::LayerNorm support for version 17"

9f91c9f

[ghstack-poisoned]

Update on "[ONNX] Add onnx::LayerNorm support for version 17"

85f28eb

[ghstack-poisoned]

titaiwangms added a commit that referenced this pull request Sep 1, 2022

[ONNX] Add onnx::LayerNorm support for version 17

c907e3f

ghstack-source-id: 2dfc81e Pull Request resolved: #84293

Update on "[ONNX] Add onnx::LayerNorm support for version 17"

b2011be

[ghstack-poisoned]

titaiwangms added a commit that referenced this pull request Sep 3, 2022

[ONNX] Add onnx::LayerNorm support for version 17

0419d26

ghstack-source-id: 3f12925 Pull Request resolved: #84293

justinchuby reviewed Sep 3, 2022

View reviewed changes

Update on "[ONNX] Add onnx::LayerNorm support for version 17"

7aa8ac9

[ghstack-poisoned]

titaiwangms added a commit that referenced this pull request Sep 4, 2022

[ONNX] Add onnx::LayerNorm support for version 17

593871e

ghstack-source-id: 4155b51 Pull Request resolved: #84293

titaiwangms added the module: onnx Related to torch.onnx label Sep 4, 2022

pytorchmergebot added the Merged label Sep 4, 2022

pytorchmergebot closed this in 7c4c7da Sep 4, 2022

facebook-github-bot deleted the gh/AllenTiTaiWang/10/head branch September 7, 2022 14:20

jcwchen mentioned this pull request Sep 29, 2022

LayerNorm Op missing? onnx/tutorials#267

Open



		@symbolic_helper.parse_args("v", "is", "v", "v", "f", "i")
		def layer_norm(g, input, normalized_shape, weight, bias, eps, cudnn_enable):

	# layer_norm normalizes on the last D dimension,
	# layer_norm normalizes on the last D dimensions,

	# which D is the size of normalized_shape
	# where D is the size of normalized_shape

Conversation

titaiwangms commented Aug 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Aug 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

✅ No Failures (0 Pending)

Uh oh!

justinchuby commented Aug 30, 2022

Uh oh!

titaiwangms commented Aug 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

titaiwangms Aug 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BowenBao commented Sep 1, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

titaiwangms commented Sep 4, 2022

Uh oh!

pytorchmergebot commented Sep 4, 2022

Uh oh!

github-actions bot commented Sep 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

titaiwangms commented Aug 30, 2022 •

edited

Loading

facebook-github-bot commented Aug 30, 2022 •

edited

Loading

titaiwangms commented Aug 30, 2022 •

edited

Loading

titaiwangms Aug 31, 2022 •

edited

Loading