Skip to content

support PPSeg model for dnn module#20511

Merged
alalek merged 9 commits intoopencv:3.4from
wjj19950828:add_humanseg_support_0806
Sep 1, 2021
Merged

support PPSeg model for dnn module#20511
alalek merged 9 commits intoopencv:3.4from
wjj19950828:add_humanseg_support_0806

Conversation

@wjj19950828
Copy link
Copy Markdown
Contributor

@wjj19950828 wjj19950828 commented Aug 6, 2021 •
edited
Loading

Merge with extra: opencv/opencv_extra#892

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
  • The PR is proposed to proper branch
  • There is reference to original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

This PR do follow contributes

1、fixed opencv.dnn resize op,support interp_mode == "half_pixel" situation.if not fixed,the onnx and dnn result has diff.
2、add PaddleSeg Portrait Segmentation demo by OpenCV.
3、update readme.

my previous PR:#20290, because previous PR is based on "master" and if target previous PR to 3.4 branch 3000+ files have changed,so i open PR again based on 3.4 branch.All comments in previous PR have done in this PR.

Test data and model in opencv/opencv_extra#892

opencv_extra=add_onnx_resize_test

@wjj19950828 wjj19950828 mentioned this pull request Aug 6, 2021
Merged
@asmorkalov asmorkalov requested a review from JulieBar August 9, 2021 07:17
@kaingwade
Copy link
Copy Markdown
Contributor

kaingwade commented Aug 11, 2021

jenkins cn please retry a build

1 similar comment
@fengyuentau
Copy link
Copy Markdown
Member

fengyuentau commented Aug 11, 2021

jenkins cn please retry a build

@asmorkalov asmorkalov requested review from rogday and removed request for JulieBar August 12, 2021 08:18
rogday
rogday reviewed Aug 13, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@rogday rogday left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution!

samples/dnn/dnn_model_runner/dnn_conversion/paddlepaddle/README.md Outdated

there are 3 part of this execution

* 1. Export PaddlePaddle ResNet50 model to onnx format;
Copy link
Copy Markdown
Member

@rogday rogday Aug 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, choose bullet list or numbered list and change ; to .

Copy link
Copy Markdown
Contributor Author

@wjj19950828 wjj19950828 Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

samples/dnn/dnn_model_runner/dnn_conversion/paddlepaddle/README.md Outdated

### Convert to ONNX Model

#### Step1. get Paddle Inference model
Copy link
Copy Markdown
Member

@rogday rogday Aug 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Step1. get -> 1. Get

Copy link
Copy Markdown
Contributor Author

@wjj19950828 wjj19950828 Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

samples/dnn/dnn_model_runner/dnn_conversion/paddlepaddle/README.md Outdated

there are 3 part of this execution

* 1. Use `cv2.dnn.readNetFromONNX` load model file;
Copy link
Copy Markdown
Member

@rogday rogday Aug 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Use cv2.dnn.readNetFromONNX to load the model file.

Copy link
Copy Markdown
Contributor Author

@wjj19950828 wjj19950828 Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

samples/dnn/dnn_model_runner/dnn_conversion/paddlepaddle/README.md Outdated
@@ -0,0 +1,80 @@
# Run PaddlePaddle model by OpenCV
Copy link
Copy Markdown
Member

@rogday rogday Aug 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

by -> using

Copy link
Copy Markdown
Contributor Author

@wjj19950828 wjj19950828 Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

samples/dnn/dnn_model_runner/dnn_conversion/paddlepaddle/README.md Outdated
@@ -0,0 +1,80 @@
# Run PaddlePaddle model by OpenCV

This tutorial gives 2 demos show how to run PaddlePaddle model by opencv
Copy link
Copy Markdown
Member

@rogday rogday Aug 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two demonstrations show how to inference PaddlePaddle model using OpenCV.

Copy link
Copy Markdown
Contributor Author

@wjj19950828 wjj19950828 Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

samples/dnn/dnn_model_runner/dnn_conversion/paddlepaddle/README.md Outdated
there are 3 part of this execution

* 1. Export PaddlePaddle ResNet50 model to onnx format;
* 2. Use `cv2.dnn.readNetFromONNX` load model file;
Copy link
Copy Markdown
Member

@rogday rogday Aug 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Use cv2.dnn.readNetFromONNX to load the model file.

Copy link
Copy Markdown
Contributor Author

@wjj19950828 wjj19950828 Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

samples/dnn/dnn_model_runner/dnn_conversion/paddlepaddle/README.md Outdated

* 1. Export PaddlePaddle ResNet50 model to onnx format;
* 2. Use `cv2.dnn.readNetFromONNX` load model file;
* 3. Preprocess image file and do inference.
Copy link
Copy Markdown
Member

@rogday rogday Aug 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Preprocess image file and do the inference.

Copy link
Copy Markdown
Contributor Author

@wjj19950828 wjj19950828 Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

samples/dnn/dnn_model_runner/dnn_conversion/paddlepaddle/README.md Outdated
python paddle_humanseg.py
```

there are 3 part of this execution
Copy link
Copy Markdown
Member

@rogday rogday Aug 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are three parts to the process:

Copy link
Copy Markdown
Contributor Author

@wjj19950828 wjj19950828 Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

samples/dnn/dnn_model_runner/dnn_conversion/paddlepaddle/README.md Outdated

* 1. Use `cv2.dnn.readNetFromONNX` load model file;
* 2. Preprocess image file and do inference.
* 3. Postprocess image file and visualization.
Copy link
Copy Markdown
Member

@rogday rogday Aug 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Postprocess image file and visualize.

Copy link
Copy Markdown
Contributor Author

@wjj19950828 wjj19950828 Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

samples/dnn/dnn_model_runner/dnn_conversion/paddlepaddle/README.md Outdated
* 2. Preprocess image file and do inference.
* 3. Postprocess image file and visualization.

The result is in the path `data/result_test_human.jpg` .
Copy link
Copy Markdown
Member

@rogday rogday Aug 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The resulting file can be found at data/result_test_human.jpg.

Copy link
Copy Markdown
Contributor Author

@wjj19950828 wjj19950828 Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

alalek
alalek requested changes Aug 14, 2021
View reviewed changes
samples/dnn/dnn_model_runner/dnn_conversion/paddlepaddle/README.md Outdated
@@ -0,0 +1,80 @@
# Run PaddlePaddle model using OpenCV
Copy link
Copy Markdown
Member

@alalek alalek Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

samples/dnn/dnn_model_runner/dnn_conversion

Looks like these files are borrowed from the master branch.
There are no such files on "3.4" maintenance branch and no plans to backport this feature.

Copy link
Copy Markdown
Contributor Author

@wjj19950828 wjj19950828 Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because previous PR is based on "master",the main purpose is to add a portrait segmentation sample in the samples/dnn/dnn_model_runner/dnn_conversion/paddlepaddle directory, but because the "3.4" branch does not have a corresponding directory, I created this directory for the future It's easy to merge into the "master" branch. If this is the case, do you have any good suggestions to solve this problem?

Copy link
Copy Markdown
Member

@alalek alalek Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets keep bugfix part for "3.4" branch.
New samples and other features should go to "master" branch (continue PR #20290).

Copy link
Copy Markdown
Contributor Author

@wjj19950828 wjj19950828 Aug 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your suggestions~

@wjj19950828 wjj19950828 mentioned this pull request Aug 22, 2021
Merged
6 tasks
@wjj19950828
Copy link
Copy Markdown
Contributor Author

wjj19950828 commented Aug 22, 2021

@alalek I have kept bugfix part for "3.4" branch,and new samples and other features should go to #20290.
For regression testing, I only kept the resize layer and its input and output data, and added the model and data generation code based on pytorch and paddle.The reason for using paddle is because my portrait segmentation onnx model is generated by paddle2onnx.Looking forward to it your reply~

@alalek
Copy link
Copy Markdown
Member

alalek commented Aug 22, 2021 •
edited
Loading

@wjj19950828 Thank you for update!

Please take a look on tests failures.

Debug dump 
[DEBUG:0] DNN/ONNX: processing ONNX model from file: /home/alalek/projects/opencv/extra/testdata/dnn/onnx/models/resize_humanseg.onnx
[ INFO:0] DNN/ONNX: loading ONNX v7 model produced by 'PaddlePaddle'. Number of nodes = 10, inputs = 1, outputs = 1
[DEBUG:0] DNN/ONNX: graph simplified to 10 nodes
[DEBUG:0] DNN/ONNX: processing node with 0 inputs and 1 outputs: [Constant]:(Constant_0)
[DEBUG:0] DNN/ONNX: processing node with 0 inputs and 1 outputs: [Constant]:(Constant_1)
[DEBUG:0] DNN/ONNX: processing node with 1 inputs and 1 outputs: [Shape]:(Shape_0)
[DEBUG:0] DNN/ONNX: processing node with 0 inputs and 1 outputs: [Constant]:(Constant_2)
[DEBUG:0] DNN/ONNX: processing node with 0 inputs and 1 outputs: [Constant]:(Constant_3)
[DEBUG:0] DNN/ONNX: processing node with 3 inputs and 1 outputs: [Slice]:(Slice_0)
[DEBUG:0] DNN/ONNX: processing node with 0 inputs and 1 outputs: [Constant]:(Constant_4)
[DEBUG:0] DNN/ONNX: processing node with 2 inputs and 1 outputs: [Concat]:(Concat_0)
[DEBUG:0] DNN/ONNX: processing node with 4 inputs and 1 outputs: [Resize]:(bilinear_interp_v2_1.tmp_0)
[DEBUG:0] DNN/ONNX: processing node with 1 inputs and 1 outputs: [Identity]:(save_infer_model/scale_0.tmp_0)
[DEBUG:0] DNN/ONNX: import completed!
[DEBUG:0] DNN/ONNX: processing ONNX model from file: /home/alalek/projects/opencv/extra/testdata/dnn/onnx/models/resize_opset11.onnx
[ INFO:0] DNN/ONNX: loading ONNX v6 model produced by 'pytorch':1.6. Number of nodes = 9, inputs = 2, outputs = 1
[DEBUG:0] DNN/ONNX: graph simplified to 9 nodes
[DEBUG:0] DNN/ONNX: processing node with 0 inputs and 1 outputs: [Constant]:(2)
[DEBUG:0] DNN/ONNX: processing node with 1 inputs and 1 outputs: [Shape]:(3)
[DEBUG:0] DNN/ONNX: processing node with 0 inputs and 1 outputs: [Constant]:(4)
[DEBUG:0] DNN/ONNX: processing node with 0 inputs and 1 outputs: [Constant]:(5)
[DEBUG:0] DNN/ONNX: processing node with 0 inputs and 1 outputs: [Constant]:(6)
[DEBUG:0] DNN/ONNX: processing node with 4 inputs and 1 outputs: [Slice]:(7)
[DEBUG:0] DNN/ONNX: processing node with 2 inputs and 1 outputs: [Concat]:(9)
[ERROR:0] DNN/ONNX: ERROR during processing node with 2 inputs and 1 outputs: [Concat]:(9)
[ INFO:0]     Input[0] = '7'
[ INFO:0]     Input[1] = '12'
[ INFO:0]     Output[0] = '9'
unknown file: Failure
C++ exception with description "OpenCV(3.4.15-dev) /home/alalek/projects/opencv/dev/modules/dnn/src/onnx/onnx_importer.cpp:543: error: (-2:Unspecified error) in function 'handleNode'
> Node [Concat]:(9) parse error: OpenCV(3.4.15-dev) /home/alalek/projects/opencv/dev/modules/dnn/src/layers/concat_layer.cpp:91: error: (-215:Assertion failed) curShape.size() == outputs[0].size() in function 'getMemoryShapes'
> " thrown in the test body.

@alalek
Copy link
Copy Markdown
Member

alalek commented Aug 30, 2021

We still have failed test due to missing test data file (it was renamed in opencv_extra PR).

@wjj19950828
Copy link
Copy Markdown
Contributor Author

wjj19950828 commented Aug 31, 2021

We still have failed test due to missing test data file (it was renamed in opencv_extra PR).

There seems to be no problem with the test cases, but the opencv_extra branch prompts that the file size exceeds the 1M limit, but in fact the relevant test cases do not exceed the 1M size. How should I solve it?

In addition, the status of this PR has not been updated for a long time. Do you have any good suggestions for this PR?

Looking forward to it your reply~

alalek
alalek approved these changes Sep 1, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Thank you for contribution 👍

@alalek alalek merged commit edc442a into opencv:3.4 Sep 1, 2021
@wjj19950828
Copy link
Copy Markdown
Contributor Author

wjj19950828 commented Sep 1, 2021

Looks good to me! Thank you for contribution 👍

Thank you~Do you have any plans or suggestions for this PR:#20290

Looking forward to your reply~

@alalek alalek mentioned this pull request Sep 1, 2021
Merged
@alalek alalek mentioned this pull request Oct 15, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alalek alalek alalek approved these changes

+1 more reviewer

@rogday rogday rogday left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@alalek alalek

Labels

category: dnn feature test

Projects

None yet

Milestone

3.4.16

Development

Successfully merging this pull request may close these issues.

6 participants

@wjj19950828 @kaingwade @fengyuentau @alalek @rogday @asmorkalov