dnn test: move layer norm tests into conformance tests by fengyuentau · Pull Request #24544 · opencv/opencv

fengyuentau · 2023-11-15T09:09:53Z

Motivation

Some ONNX operators, such as LayerNormalization, BatchNormalization and so on, produce outputs for training (mean, stdev). So they have reference outputs of conformance tests for those training outputs as well. However, when it comes to inference, we do not need and produce those outputs for training here in dnn. Hence, output size does not match if we use dnn to infer those conformance models. This has become the barrier if we want to test these operators using their conformance tests.

I checked all ONNX operators with optional outputs. Turns out there are only BatchNormalization, Dropout, LayerNormalization and MaxPool has optional outputs for training. All except LayerNormalization have models set for training mode and eval mode. Blame ONNX for that.

Solution

In this pull request, we remove graph outputs if the graph looks like the following:

    [X]   [Scale]  [Bias]                      [X]   [Scale]  [Bias]
      \      |      /         this patch         \      |      /
     LayerNormalization      ----------->       LayerNormalization
      /      |      \                                   |
    [Y]    [Mean]  [Stdev]                             [Y]

We can update conformance tests and turn on some cases as well if extending to more layers.

Notes:

This workaround does not solve expanded function operators if they are fused into a single operator, such as $onnx/onnx/backend/test/data/node/test_layer_normalization_2d_axis1_expanded, but they can be run without fusion. Note that either dnn or onnxruntime does not fuse those expanded function operators.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

…ance tests

dkurt · 2023-11-15T10:40:06Z

modules/dnn/src/onnx/onnx_graph_simplifier.cpp

+        if (op_type == "LayerNormalization") {
+            net.mutable_output()->DeleteSubrange(1, 2);
+        }
+    }


Isn't the problem in the test ONNX models? Shall they be exported in eval() mode?

https://github.com/opencv/opencv_extra/blob/0c9ec227253c60165b125c7bce313fc2bb4cb592/testdata/dnn/onnx/generate_onnx_models.py#L2615

No. They are generated by the ONNX official. You can find most of them in https://github.com/onnx/onnx/tree/main/onnx/backend/test/data/node.

These models and tests are added via #21088.

Also related issue to check ONNX operator coverage: #21078.

I checked all ONNX operators with optional outputs. Turns out there are only BatchNormalization, Dropout, LayerNormalization and MaxPool which have optional outputs for training. All except LayerNormalization have conformance models set for training mode and eval mode. Some of them have training outputs, some dont, so we can still use those without training outputs for conformance tests. But this is not the case with LayerNormalization. All LayerNormalization conformance models have training outputs. Blame ONNX for that.

Seems like this patch is meaningful only for LayerNormalization. What do you think?

As tests passed and layers' outputs do not depend in train/eval phase on which they were exported to ONNX, I'm fine.

The only concern is that changes made in the unrelated graph fusion method. Can this be moved to parseLayerNorm from onnx_importer.cpp?

Can this be moved to parseLayerNorm from onnx_importer.cpp?

AFAIK, it is not feasible because for conformance tests like I present above their ouputs are attached to graph outputs, and we cannot modify graph proto inside parseLayerNorm.

The only concern is that changes made in the unrelated graph fusion method.

I put conditions to ensure node_size == 1, op_type == "LayerNormalization" and node_output_size > 1. I think it should be safe for basically all models, since they have more than one node even after fusion.

@fengyuentau, please try this:

--- a/modules/dnn/src/onnx/onnx_graph_simplifier.cpp +++ b/modules/dnn/src/onnx/onnx_graph_simplifier.cpp @@ -1184,18 +1184,6 @@ void simplifySubgraphs(opencv_onnx::GraphProto& net) subgraphs.push_back(makePtr<NormalizeSubgraph5>()); simplifySubgraphs(Ptr<ImportGraphWrapper>(new ONNXGraphWrapper(net)), subgraphs); - - // remove training outputs for conformance tests - if (net.node().size() == 1) { - const auto &node = net.node(0); - const auto &op_type = node.op_type(); - if (op_type == "LayerNormalization") { - if (net.output_size() > 1) { - int num = net.output_size() - 1; - net.mutable_output()->DeleteSubrange(1, num); - } - } - } } Mat getMatFromTensor(const opencv_onnx::TensorProto& tensor_proto) diff --git a/modules/dnn/src/onnx/onnx_importer.cpp b/modules/dnn/src/onnx/onnx_importer.cpp index 28dd4d9b77..80f57c4156 100644 --- a/modules/dnn/src/onnx/onnx_importer.cpp +++ b/modules/dnn/src/onnx/onnx_importer.cpp @@ -3187,6 +3187,17 @@ void ONNXImporter::parseLayerNorm(LayerParams& layerParams, const opencv_onnx::N // Remove additional outputs (Mean, InvStdDev) if (node_proto.output_size() > 1) { + for (size_t i = 1; i < node_proto.output_size(); ++i) + { + for (int j = graph_proto.output_size() - 1; j >= 0; --j) + { + if (graph_proto.output(j).name() == node_proto.output(i)) + { + graph_proto.mutable_output()->DeleteSubrange(j, 1); + break; + } + } + } auto outputName = node_proto.output(0); opencv_onnx::NodeProto node_proto_ = node_proto; node_proto_.clear_output();

I did not know graph_proto is available inside each parseXXX. Thanks, it works.

dnn: add openvino, opencl and cuda backends for layer normalization layer #24552 Merge after #24544. Todo: - [x] openvino - [x] opencl - [x] cuda ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

dnn test: move layer norm tests into conformance tests opencv#24544 Merge with opencv/opencv_extra#1122 ## Motivation Some ONNX operators, such as `LayerNormalization`, `BatchNormalization` and so on, produce outputs for training (mean, stdev). So they have reference outputs of conformance tests for those training outputs as well. However, when it comes to inference, we do not need and produce those outputs for training here in dnn. Hence, output size does not match if we use dnn to infer those conformance models. This has become the barrier if we want to test these operators using their conformance tests.  **I checked all ONNX operators with optional outputs. Turns out there are only `BatchNormalization`, `Dropout`, `LayerNormalization` and `MaxPool` has optional outputs for training. All except `LayerNormalization` have models set for training mode and eval mode. Blame ONNX for that.** ## Solution In this pull request, we remove graph outputs if the graph looks like the following: ``` [X] [Scale] [Bias] [X] [Scale] [Bias] \ | / this patch \ | / LayerNormalization -----------> LayerNormalization / | \ | [Y] [Mean] [Stdev] [Y] ``` We can update conformance tests and turn on some cases as well if extending to more layers. Notes: 1. This workaround does not solve expanded function operators if they are fused into a single operator, such as `$onnx/onnx/backend/test/data/node/test_layer_normalization_2d_axis1_expanded`, but they can be run without fusion. Note that either dnn or onnxruntime does not fuse those expanded function operators. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

dnn: add openvino, opencl and cuda backends for layer normalization layer opencv#24552 Merge after opencv#24544. Todo: - [x] openvino - [x] opencl - [x] cuda ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

dnn test: move layer norm tests into conformance tests opencv#24544 Merge with opencv/opencv_extra#1122 ## Motivation Some ONNX operators, such as `LayerNormalization`, `BatchNormalization` and so on, produce outputs for training (mean, stdev). So they have reference outputs of conformance tests for those training outputs as well. However, when it comes to inference, we do not need and produce those outputs for training here in dnn. Hence, output size does not match if we use dnn to infer those conformance models. This has become the barrier if we want to test these operators using their conformance tests.  **I checked all ONNX operators with optional outputs. Turns out there are only `BatchNormalization`, `Dropout`, `LayerNormalization` and `MaxPool` has optional outputs for training. All except `LayerNormalization` have models set for training mode and eval mode. Blame ONNX for that.** ## Solution In this pull request, we remove graph outputs if the graph looks like the following: ``` [X] [Scale] [Bias] [X] [Scale] [Bias] \ | / this patch \ | / LayerNormalization -----------> LayerNormalization / | \ | [Y] [Mean] [Stdev] [Y] ``` We can update conformance tests and turn on some cases as well if extending to more layers. Notes: 1. This workaround does not solve expanded function operators if they are fused into a single operator, such as `$onnx/onnx/backend/test/data/node/test_layer_normalization_2d_axis1_expanded`, but they can be run without fusion. Note that either dnn or onnxruntime does not fuse those expanded function operators. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

dnn: add openvino, opencl and cuda backends for layer normalization layer opencv#24552 Merge after opencv#24544. Todo: - [x] openvino - [x] opencl - [x] cuda ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

remove addition outputs for conformance tests; add layer norm conform…

6c0f21d

…ance tests

fengyuentau added category: dnn category: dnn (onnx) ONNX suport issues in DNN module labels Nov 15, 2023

fengyuentau added this to the 4.9.0 milestone Nov 15, 2023

fengyuentau requested review from asmorkalov and dkurt November 15, 2023 09:09

fengyuentau assigned asmorkalov Nov 15, 2023

fengyuentau added 2 commits November 15, 2023 17:39

modify graph outputs in simplifySubgraphs

373d8f2

safer remove

ca120da

dkurt reviewed Nov 15, 2023

View reviewed changes

fengyuentau added 2 commits November 15, 2023 19:02

add openvino filter

d37d822

move to parseLayerNorm

2223e61

fengyuentau mentioned this pull request Nov 20, 2023

dnn: add openvino, opencl and cuda backends for layer normalization layer #24552

Merged

9 tasks

dkurt approved these changes Nov 20, 2023

View reviewed changes

asmorkalov assigned dkurt and unassigned asmorkalov Nov 20, 2023

asmorkalov merged commit a478757 into opencv:4.x Nov 20, 2023

fengyuentau deleted the layernorm_conformance branch November 20, 2023 15:37

asmorkalov mentioned this pull request Jan 19, 2024

5.x merge 4.x #24862

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

dnn test: move layer norm tests into conformance tests#24544

dnn test: move layer norm tests into conformance tests#24544
asmorkalov merged 5 commits intoopencv:4.xfrom
fengyuentau:layernorm_conformance

fengyuentau commented Nov 15, 2023 •

edited

Loading

Uh oh!

dkurt Nov 15, 2023 •

edited

Loading

Uh oh!

fengyuentau Nov 15, 2023

Uh oh!

fengyuentau Nov 15, 2023

Uh oh!

dkurt Nov 15, 2023

Uh oh!

fengyuentau Nov 16, 2023

Uh oh!

dkurt Nov 17, 2023

Uh oh!

fengyuentau Nov 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

fengyuentau commented Nov 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Solution

Pull Request Readiness Checklist

Uh oh!

dkurt Nov 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fengyuentau Nov 15, 2023

Choose a reason for hiding this comment

Uh oh!

fengyuentau Nov 15, 2023

Choose a reason for hiding this comment

Uh oh!

dkurt Nov 15, 2023

Choose a reason for hiding this comment

Uh oh!

fengyuentau Nov 16, 2023

Choose a reason for hiding this comment

Uh oh!

dkurt Nov 17, 2023

Choose a reason for hiding this comment

Uh oh!

fengyuentau Nov 20, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fengyuentau commented Nov 15, 2023 •

edited

Loading

dkurt Nov 15, 2023 •

edited

Loading