Skip to content

Support for MobileNetV3-SSD from TensorFlow#16760

Merged
opencv-pushbot merged 1 commit intoopencv:3.4from
dkurt:dnn_mobilenet_v3
Mar 14, 2020
Merged

Support for MobileNetV3-SSD from TensorFlow#16760
opencv-pushbot merged 1 commit intoopencv:3.4from
dkurt:dnn_mobilenet_v3

Conversation

@dkurt
Copy link
Copy Markdown
Member

@dkurt dkurt commented Mar 8, 2020

Pull Request Readiness Checklist

Merge with extra: opencv/opencv_extra#725

Add support for MobileNetV3 SSD from TensorFlow Object Detection API: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md#mobile-models

Follow a guide to run the model: https://github.com/opencv/opencv/wiki/TensorFlow-Object-Detection-API

NOTE: considering discussion tensorflow/models#7727, only updated models work correctly: tensorflow/models#8057

NOTE: comparing to other SSD networks, these models expect normalized inputs ([-1, 1] input range):

import cv2 as cv

net = cv.dnn_DetectionModel('frozen_inference_graph.pb', 'graph.pbtxt')
net.setInputSize(320, 320)
net.setInputScale(1.0 / 127.5)
net.setInputMean((127.5, 127.5, 127.5))
net.setInputSwapRB(True)

frame = cv.imread('example.jpg')

classes, confidences, boxes = net.detect(frame, confThreshold=0.5)

for classId, confidence, box in zip(classes.flatten(), confidences.flatten(), boxes):
    print(classId, confidence)
    cv.rectangle(frame, box, color=(0, 255, 0))

cv.imshow('out', frame)
cv.waitKey()

res

force_builders=Custom,Custom Win,Custom Mac
build_image:Custom=ubuntu-openvino-2020.1.0:16.04
build_image:Custom Win=openvino-2020.1.0
build_image:Custom Mac=openvino-2020.1.0

test_modules:Custom=dnn,python2,python3,java
test_modules:Custom Win=dnn,python2,python3,java
test_modules:Custom Mac=dnn,python2,python3,java

buildworker:Custom=linux-1
# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*

@dkurt dkurt force-pushed the dnn_mobilenet_v3 branch from 2d87c68 to b927ce1 Compare March 8, 2020 18:09
@dkurt dkurt added this to the 3.4.10 milestone Mar 10, 2020
Copy link
Copy Markdown
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@opencv-pushbot opencv-pushbot merged commit 683910f into opencv:3.4 Mar 14, 2020
@alalek alalek mentioned this pull request Mar 17, 2020
@rokopi-byte
Copy link
Copy Markdown

Are setInputScale and setInputMean necessary ?
I'm currently testing this snippet and it works without them (I suppose these operation are already in the network model) while it does not work with them.. prediction is wrong (I suppose operation are doubled in this case)!

@dkurt
Copy link
Copy Markdown
Member Author

dkurt commented May 12, 2020

NOTE: considering discussion tensorflow/models#7727, only updated models work correctly: tensorflow/models#8057
NOTE: comparing to other SSD networks, these models expect normalized inputs ([-1, 1] input range):

@roccopietrini, Do you try the model from the model zoo or a custom one? Model ssd_mobilenet_v3_large_coco_2020_01_14 expects normalized input.

@rokopi-byte
Copy link
Copy Markdown

I tried with the model zoo model (ssd_mobilenet_v3_small_coco_2020_01_14), with just transfer learning for my custom class (so I just modified some parameters about the training in the pipeline.config, without any modification to the network). I generated the pbtxt file using this: https://github.com/opencv/opencv/blob/master/samples/dnn/tf_text_graph_ssd.py
I know the network expect normalized input, but I guess this is done by the network itself ? If I use setInputScale and setInputMea the prediction is wrong (the bounding box is much bigger), so I suspect the operations are doubled (because they already occur in the network..).

@dkurt
Copy link
Copy Markdown
Member Author

dkurt commented May 13, 2020

@roccopietrini, maybe something has been changed in the TensorFlow Object Detection API and published model does not reflect actual training code. Do you mind to open a separate issue so we won't miss it? Please add all the details - upload a newtork if possible, add origin image and expected results.

@rokopi-byte
Copy link
Copy Markdown

Hi, I don't think at this point that we have a bug in opencv .. so I'd better ask in stackoverflow because it's not fully clear to me weather the network include the normalization or not by default. I'll be back if I discover something .. :)

@boris-savic
Copy link
Copy Markdown

Hi,

using this exact code with installed:

numpy-1.18.5
opencv-python-4.2.0.34

I get error:

cv2.error: OpenCV(4.2.0) /io/opencv/modules/dnn/src/dnn.cpp:562: error: (-2:Unspecified error) Can't create layer "FeatureExtractor/MobilenetV3/Conv/hard_swish/add" of type "AddV2" in function 'getLayerInstance'

Any idea what did I miss?

@dkurt
Copy link
Copy Markdown
Member Author

dkurt commented Jun 11, 2020

@boris-savic, open .pbtxt and replace all "AddV2" to "Add"

@boris-savic
Copy link
Copy Markdown

@boris-savic, open .pbtxt and replace all "AddV2" to "Add"

Thanks. This did move things a bit forward but seems to get stuck on another issue:

[ERROR:0] global /io/opencv/modules/dnn/src/dnn.cpp (3066) getLayerShapesRecursively OPENCV/DNN: [Eltwise]:(FeatureExtractor/MobilenetV3/expanded_conv_3/squeeze_excite/mul): getMemoryShapes() throws exception. inputs=2 outputs=0/1
[ERROR:0] global /io/opencv/modules/dnn/src/dnn.cpp (3069) getLayerShapesRecursively     input[0] = [ 1 72 40 40 ]
[ERROR:0] global /io/opencv/modules/dnn/src/dnn.cpp (3069) getLayerShapesRecursively     input[1] = [ 1 72 1 1 ]
[ERROR:0] global /io/opencv/modules/dnn/src/dnn.cpp (3075) getLayerShapesRecursively Exception message: OpenCV(4.2.0) /io/opencv/modules/dnn/src/layers/eltwise_layer.cpp:209: error: (-215:Assertion failed) inputs[0][j] == inputs[i][j] in function 'getMemoryShapes'

Traceback (most recent call last):
  File "/home/boris/workspace/cv-dnn/detection.py", line 13, in <module>
    classes, confidences, boxes = net.detect(frame, confThreshold=0.5)
cv2.error: OpenCV(4.2.0) /io/opencv/modules/dnn/src/layers/eltwise_layer.cpp:209: error: (-215:Assertion failed) inputs[0][j] == inputs[i][j] in function 'getMemoryShapes'


Process finished with exit code 1

@dkurt
Copy link
Copy Markdown
Member Author

dkurt commented Jun 11, 2020

@boris-savic, please use the latest version of the script or see similar issue: #17058

@AnwarYangui
Copy link
Copy Markdown

Bonsoir, comment télécharger le modèle MobileNet-SSD v3 ?
Lorsque je clique sur poids ou configuration, rien ne se passe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

eltwise_layer.cpp:116: error: (-215:Assertion failed) inputs[0] == inputs[i] in function 'getMemoryShapes'

7 participants