Added PyTorch fcnresnet101 segmentation conversion cases by richard28039 · Pull Request #24397 · opencv/opencv

richard28039 · 2023-10-13T08:54:28Z

We write a sample code about transforming Pytorch fcnresnet101 to ONNX running on OpenCV.

The input source image was shooted by ourself.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

samples/dnn/dnn_model_runner/dnn_conversion/pytorch/segmantation/pytorch_fcnresnet101.py

samples/dnn/dnn_model_runner/dnn_conversion/pytorch/segmantation/data/cat_and_bike.jpg

…on directory

samples/dnn/models.yml

samples/dnn/segmentation.py

samples/dnn/models.yml

dkurt · 2023-10-26T08:14:18Z

samples/dnn/models.yml

+  width: 500
+  height: 500
+  rgb: false
+  classes: "object_detection_classes_pascal_voc.txt"


Please add an image with resulting segmentation from a sample launch. I have a concern that classes might be 1 position shifted (depends on background id).

So please run a sample on an image with more that one segmented object and put to PR description or in this thread,

Hello @dkurt , I think your concern is right, the background classes might have to be add.
Should I add background class to https://github.com/opencv/opencv/blob/4.x/samples/data/dnn/object_detection_classes_pascal_voc.txt ?

without background

with background

Thanks, problem should be solved in a common.py script which will affect other samples so I'd like to merge this PR as is.

I got it, thank you!

asmorkalov

👍

asmorkalov

python3 ./segmentation.py fcnresnet101
[ WARN:0@0.019] global samples.cpp:61 findFile cv::samples::findFile('object_detection_classes_pascal_voc.txt') => ''
[ERROR:0@0.332] global onnx_importer.cpp:2441 parseShape DNN/ONNX(Shape): dynamic 'zero' shapes are not supported, input input [ 0 3 0 0 ]
[ERROR:0@0.332] global onnx_importer.cpp:1033 handleNode DNN/ONNX: ERROR during processing node with 1 inputs and 1 outputs: [Shape]:(onnx_node!Shape_0) from domain='ai.onnx'
Traceback (most recent call last):
  File "./segmentation.py", line 83, in <module>
    net = cv.dnn.readNet(args.model, args.config, args.framework)
cv2.error: OpenCV(4.8.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/onnx/onnx_importer.cpp:1052: error: (-2:Unspecified error) in function 'handleNode'
> Node [Shape@ai.onnx]:(onnx_node!Shape_0) parse error: OpenCV(4.8.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/onnx/onnx_importer.cpp:2442: error: (-215:Assertion failed) !isDynamicShape in function 'parseShape'
>

What do I do wrong?

asmorkalov · 2023-10-31T09:39:41Z

With related patch from @dkurt (DNN graph fusion with shared nodes):

[ WARN:0@0.019] global samples.cpp:61 findFile cv::samples::findFile('object_detection_classes_pascal_voc.txt') => ''
[ERROR:0@0.329] global onnx_importer.cpp:1033 handleNode DNN/ONNX: ERROR during processing node with 1 inputs and 1 outputs: [MaxPool]:(onnx_node!MaxPool_8) from domain='ai.onnx'
Traceback (most recent call last):
  File "./segmentation.py", line 83, in <module>
    net = cv.dnn.readNet(args.model, args.config, args.framework)
cv2.error: OpenCV(4.8.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/onnx/onnx_importer.cpp:1052: error: (-2:Unspecified error) in function 'handleNode'
> Node [MaxPool@ai.onnx]:(onnx_node!MaxPool_8) parse error: OpenCV(4.8.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/pooling_layer.cpp:1338: error: (-215:Assertion failed) inputs.size() != 0 in function 'getMemoryShapes'
>

dkurt · 2023-10-31T09:59:14Z

@asmorkalov, can confirm, let me check.

UPD: solved by cc40308

asmorkalov · 2023-11-01T12:11:08Z

I can confirm that fix from @dkurt works. The sample works properly, if I add "background" to the beggining of object_detection_classes_pascal_voc.txt. Otherwise I get exception

alexander@paradox:~/Projects/OpenCV/opencv-master/samples/dnn$ python3 ./segmentation.py fcnresnet101
Traceback (most recent call last):
  File "./segmentation.py", line 133, in <module>
    showLegend(classes)
  File "./segmentation.py", line 70, in showLegend
    assert(len(classes) == len(colors))

I propose to force add background in the sample code if len(classes) - len(colors) = 1

richard28039 · 2023-11-02T07:56:26Z

@dkurt on the background question about object_detection_classes_pascal_voc.txt , the problem should be solved in a common.py in another pr as you originally said, or should I modify it in the way that @asmorkalov propose, thank you!

asmorkalov · 2023-11-02T10:10:56Z

I propose to add the hack to current PR, because the sample just crashed in default configuration. Python part may be modified in a separate PR.

dkurt · 2023-11-02T10:41:04Z

@richard28039, there are two ways to avoid crash: 1) temporary add "backdround" label to the classes array 2) remove classes: "object_detection_classes_pascal_voc.txt" for fcnresnet101 from models.yml - this way sample just colorize the segmentation without a legend. I prefer second option.

richard28039 · 2023-11-02T13:34:53Z

Hello, I removed the classes in fcnresnet101 from models.yml and the result looks good, thanks for your suggestions.

DNN graph fusion with shared nodes #24463 ### Pull Request Readiness Checklist For now, nodes from matched pattern are removed during the matching process so if nodes are used in similar subgraph, they cannot be found. required for #24397 **Merge with extra**: opencv/opencv_extra#1115 A part from [model_name ](https://github.com/onnx/models/blob/main/vision/object_detection_segmentation/fcn/model/fcn-resnet101-11.onnx) with two Resize subgraphs with shared nodes: ![image](https://github.com/opencv/opencv/assets/25801568/611d89d9-12fb-4add-9218-13b10d2c086a) See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

DNN graph fusion with shared nodes opencv#24463 ### Pull Request Readiness Checklist For now, nodes from matched pattern are removed during the matching process so if nodes are used in similar subgraph, they cannot be found. required for opencv#24397 **Merge with extra**: opencv/opencv_extra#1115 A part from [model_name ](https://github.com/onnx/models/blob/main/vision/object_detection_segmentation/fcn/model/fcn-resnet101-11.onnx) with two Resize subgraphs with shared nodes: ![image](https://github.com/opencv/opencv/assets/25801568/611d89d9-12fb-4add-9218-13b10d2c086a) See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

…_dnn_sample Added PyTorch fcnresnet101 segmentation conversion cases opencv#24397 We write a sample code about transforming Pytorch fcnresnet101 to ONNX running on OpenCV. The input source image was shooted by ourself. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [X] I agree to contribute to the project under Apache 2 License. - [X] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [X] The PR is proposed to the proper branch - [ ] There is a reference to the original bug report and related work - [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [ ] The feature is well documented and sample code can be built with the project CMake

DNN graph fusion with shared nodes opencv#24463 ### Pull Request Readiness Checklist For now, nodes from matched pattern are removed during the matching process so if nodes are used in similar subgraph, they cannot be found. required for opencv#24397 **Merge with extra**: opencv/opencv_extra#1115 A part from [model_name ](https://github.com/onnx/models/blob/main/vision/object_detection_segmentation/fcn/model/fcn-resnet101-11.onnx) with two Resize subgraphs with shared nodes: ![image](https://github.com/opencv/opencv/assets/25801568/611d89d9-12fb-4add-9218-13b10d2c086a) See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

…_dnn_sample Added PyTorch fcnresnet101 segmentation conversion cases opencv#24397 We write a sample code about transforming Pytorch fcnresnet101 to ONNX running on OpenCV. The input source image was shooted by ourself. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [X] I agree to contribute to the project under Apache 2 License. - [X] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [X] The PR is proposed to the proper branch - [ ] There is a reference to the original bug report and related work - [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [ ] The feature is well documented and sample code can be built with the project CMake

DNN graph fusion with shared nodes opencv#24463 ### Pull Request Readiness Checklist For now, nodes from matched pattern are removed during the matching process so if nodes are used in similar subgraph, they cannot be found. required for opencv#24397 **Merge with extra**: opencv/opencv_extra#1115 A part from [model_name ](https://github.com/onnx/models/blob/main/vision/object_detection_segmentation/fcn/model/fcn-resnet101-11.onnx) with two Resize subgraphs with shared nodes: ![image](https://github.com/opencv/opencv/assets/25801568/611d89d9-12fb-4add-9218-13b10d2c086a) See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

…_dnn_sample Added PyTorch fcnresnet101 segmentation conversion cases opencv#24397 We write a sample code about transforming Pytorch fcnresnet101 to ONNX running on OpenCV. The input source image was shooted by ourself. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [X] I agree to contribute to the project under Apache 2 License. - [X] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [X] The PR is proposed to the proper branch - [ ] There is a reference to the original bug report and related work - [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [ ] The feature is well documented and sample code can be built with the project CMake

richard28039 added 2 commits October 13, 2023 15:48

add_fcnresnet101_to_opencv_sample_dnn

48525fd

Merge branch '4.x' into add_fcnresnet101_to_dnn_sample

d497058

asmorkalov added category: samples category: dnn labels Oct 13, 2023

dkurt reviewed Oct 13, 2023

View reviewed changes

samples/dnn/dnn_model_runner/dnn_conversion/pytorch/segmantation/pytorch_fcnresnet101.py Outdated Show resolved Hide resolved

dkurt reviewed Oct 13, 2023

View reviewed changes

samples/dnn/dnn_model_runner/dnn_conversion/pytorch/segmantation/data/cat_and_bike.jpg Outdated Show resolved Hide resolved

richard28039 and others added 4 commits October 13, 2023 21:14

remove picture and change input method

dbaa197

add_onnx_framework_and_std_to_segmentation

d50b644

Merge branch 'opencv:4.x' into add_fcnresnet101_to_dnn_sample

65df7fe

Delete samples/dnn/dnn_model_runner/dnn_conversion/pytorch/segmantati…

cd6951c

…on directory

dkurt reviewed Oct 18, 2023

View reviewed changes

samples/dnn/models.yml Outdated Show resolved Hide resolved

dkurt reviewed Oct 18, 2023

View reviewed changes

samples/dnn/models.yml Outdated Show resolved Hide resolved

remove_unnecessary_parameter

386f4b8

dkurt reviewed Oct 19, 2023

View reviewed changes

samples/dnn/segmentation.py Outdated Show resolved Hide resolved

remove_trailing_whitespace

f852143

dkurt reviewed Oct 19, 2023

View reviewed changes

samples/dnn/models.yml Outdated Show resolved Hide resolved

dkurt reviewed Oct 19, 2023

View reviewed changes

samples/dnn/models.yml Outdated Show resolved Hide resolved

set_classes_file_to_exist_file

1c831c3

asmorkalov assigned dkurt Oct 20, 2023

asmorkalov added this to the 4.9.0 milestone Oct 20, 2023

dkurt reviewed Oct 26, 2023

View reviewed changes

dkurt mentioned this pull request Oct 27, 2023

DNN graph fusion with shared nodes #24463

Merged

6 tasks

change the model url

ce450fe

dkurt approved these changes Oct 30, 2023

View reviewed changes

dkurt requested a review from asmorkalov October 31, 2023 06:33

asmorkalov approved these changes Oct 31, 2023

View reviewed changes

asmorkalov self-requested a review October 31, 2023 09:33

asmorkalov reviewed Oct 31, 2023

View reviewed changes

asmorkalov self-requested a review October 31, 2023 09:40

remove classes

675744d

asmorkalov merged commit e95c005 into opencv:4.x Nov 3, 2023

asmorkalov mentioned this pull request Jan 19, 2024

5.x merge 4.x #24862

Merged

WanliZhong mentioned this pull request Apr 17, 2024

Can't get useful results with segmentation sample. #25434

Closed

4 tasks

Uh oh!

Conversation

richard28039 commented Oct 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkurt Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

richard28039 Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

dkurt Oct 30, 2023

Choose a reason for hiding this comment

Uh oh!

richard28039 Oct 30, 2023

Choose a reason for hiding this comment

Uh oh!

asmorkalov left a comment

Choose a reason for hiding this comment

Uh oh!

asmorkalov left a comment

Choose a reason for hiding this comment

Uh oh!

asmorkalov commented Oct 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dkurt commented Oct 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asmorkalov commented Nov 1, 2023

Uh oh!

richard28039 commented Nov 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asmorkalov commented Nov 2, 2023

Uh oh!

dkurt commented Nov 2, 2023

Uh oh!

richard28039 commented Nov 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

richard28039 commented Oct 13, 2023 •

edited

Loading

asmorkalov commented Oct 31, 2023 •

edited

Loading

dkurt commented Oct 31, 2023 •

edited

Loading

richard28039 commented Nov 2, 2023 •

edited

Loading