Interactive Open Model Zoo as OpenCV module by dkurt · Pull Request #272 · openvinotoolkit/open_model_zoo

dkurt · 2019-08-03T13:31:07Z

import cv2 as cv

from cv2.open_model_zoo.topologies import mobilenet_ssd
from cv2.open_model_zoo import DnnDetectionModel

frame = cv.imread('example.jpg')

ssd = mobilenet_ssd()
net = DnnDetectionModel(ssd)

classIds, confidences, boxes = net.detect(frame, confThreshold=0.5)
for box in boxes:
    cv.rectangle(frame, box, (0, 255, 0))

cv.imshow('out', frame)
cv.waitKey()

🐍 🐺 🦁 🙉 🐘 🐧 🐙 🦈 🐭 🐫 🐸 🐜 🐆 🦊 🐟 😼 🪲 🐳 🐨 🕷️ 🐉 🕊️ 🦉 🐰 🐔 🦀 🐝

WIP

related: opencv/opencv#14730

Pipelines

example with text recognition demo API:

import cv2 as cv
import numpy as np

from cv2.open_model_zoo import TextRecognitionPipeline

frame = cv.imread('text.jpg')

p = TextRecognitionPipeline()

rects, texts, confs = p.process(frame)

for rect, text in zip(rects, texts):
    vertices = cv.boxPoints(rect)

    for j in range(4):
        p1 = (vertices[j][0], vertices[j][1])
        p2 = (vertices[(j + 1) % 4][0], vertices[(j + 1) % 4][1])
        cv.line(frame, p1, p2, (0, 255, 0), 1);

    x = np.min(vertices[:,0])
    y = np.min(vertices[:,1])
    cv.putText(frame, text, (x, y), cv.FONT_HERSHEY_SIMPLEX, 1.0, (0, 255, 0), thickness=2)

cv.imshow('frame', frame)
cv.waitKey()

import cv2 as cv
from cv2.open_model_zoo import HumanPoseEstimation

p = HumanPoseEstimation()

frame = cv.imread('example.png')
poses = p.process(frame)

p.render(frame, poses)

cv.imshow('res', frame)
cv.waitKey()

openvino-pushbot · 2019-08-04T01:19:34Z

Can one of the admins verify this patch?

openvino-pushbot · 2019-08-04T01:21:42Z

Can one of the admins verify this patch?

snosov1 · 2019-08-08T08:40:46Z

Is it really necessary to have such a long prefix? cv.open_model_zoo_TextRecognitionPipeline(det, rec)
Do you plan to add the example into the demos folder?

dkurt · 2019-08-08T11:37:34Z

Is it really necessary to have such a long prefix? cv.open_model_zoo_TextRecognitionPipeline(det, rec)

@snosov1, For now it's a kind of bug in OpenCV's wrappers (related feature request opencv/opencv#14730). For now there is a workaround for it to define a method with proper name:

class TextRecognitionPipelineImpl
{
    //...
};

Ptr<TextRecognitionPipelineImpl> TextRecognitionPipeline(...) { return new TextRecognitionPipelineImpl(...); }

corresponding Python code:

from cv2.open_model_zoo import TextRecognitionPipeline
p = TextRecognitionPipeline()

Do you plan to add the example into the demos folder?

It'd be nice. In example, add more Python demos for existing C++ ones. The only thing we have to decide is a code duplication: move C++ code from samples to the module and replace it to corresponding APIs.

snosov1 · 2019-08-08T11:59:05Z

Not sure I understand the answers, Dmitry =)

For now it's a kind of bug in OpenCV's wrappers

Are you saying that it will look "normal" after the merge of the respective PR?

add more Python demos for existing C++ ones.

That's a different story. For starters we should have at least one to showcase this type of usage.

dkurt · 2019-08-08T12:49:39Z

@snosov1,

from cv2.open_model_zoo import TextRecognitionPipeline
p = TextRecognitionPipeline()

would be available :) I'll provide the code which do it in future commits.

Maybe one of open questions is name scopes to differ "topologies" from "algorithms". In example,

from cv2.open_model_zoo.topologies import text_detection, text_recognition
from cv2.open_model_zoo import TextRecognitionPipeline

p = TextRecognitionPipeline()

jrosebr1 · 2019-08-11T13:54:09Z

@dkurt This is a really amazing PR, incredible work! If you need help putting together examples of how to use it let me know and I'll publish some content on the PyImageSearch blog 😄

dkurt · 2019-08-11T14:08:49Z

Adrian, thank you! That would be great!

jrosebr1 · 2019-08-12T12:46:49Z

Absolutely! I'll keep an eye on this PR and once it's merged I'll put something together 😄

I was looking at the TextRecognitionPipeline code -- I assume it's using EAST to localize text in the image, but what about the actual OCR component. What model is being used there?

dkurt · 2019-08-12T13:16:27Z

@jrosebr1, text-detection-0004: https://github.com/opencv/open_model_zoo/blob/master/intel_models/text-detection-0004/description/text-detection-0004.md.

I think that next step is to enable these pipelines to with with different kind of models (it could be as EAST text detection as Intel's models).

IRDonch · 2019-08-12T15:50:48Z

Without going into a detailed review, I have a few high level comments:

I'm confused as to how this is supposed to be used. OMZ is included into the OpenVINO toolkit as source code, while OpenCV is included as binaries. Thus an OpenVINO tookit user has no way to build this module, as his OpenCV is already built.
Half of this code is a reimplementation of Model Downloader/Converter. This is extremely fragile. The only supported way to download/convert/query models is by running the tools in tools/downloader. Anything else is liable to break as soon as the format of the config changes.

This code, from what I can see, is already non-functional for models with regex-based postprocessing and PyTorch models. And it's going to be completely broken after Split list_topologies.yml into per-model config files #276, and then broken some more by another PR I have planned. And I heard rumors about a planned Caffe model which requires a new kind of post-processing, so that's going to be broken too. And that's just the changes that I can foresee.

The bottom line is that this is unacceptable. You either need to run Model Downloader or just request that the user supply a directory with everything already downloaded. IMO, the latter option is way simpler.

dkurt · 2019-08-12T16:02:38Z

@IRDonch, thanks for review!

I'm confused as to how this is supposed to be used. OMZ is included into the OpenVINO toolkit as source code, while OpenCV is included as binaries. Thus an OpenVINO tookit user has no way to build this module, as his OpenCV is already built.

We can also build it as a part of OpenCV's binaries for OpenVINO. Just adding OPENCV_EXTRA_MODULES_PATH during build time.

Half of this code is a reimplementation of Model Downloader/Converter. This is extremely fragile. The only supported way to download/convert/query models is by running the tools in tools/downloader. Anything else is liable to break as soon as the format of the config changes.

Python code is based on OMZ's downloader and https://github.com/opencv/opencv_extra/blob/master/testdata/dnn/download_models.py. Binaries uses only default Python packages (urllib, zipfile, tarfile, etc.)

This code, from what I can see, is already non-functional for models with regex-based postprocessing and PyTorch models. And it's going to be completely broken after #276, and then broken some more by another PR I have planned. And I heard rumors about a planned Caffe model which requires a new kind of post-processing, so that's going to be broken too. And that's just the changes that I can foresee.

Most of postprocessing doesn't seem actual:

    postprocessing:
      - $type: regex_replace
        file: mtcnn-p.prototxt
        pattern: 'dim: 12'
        replacement: 'dim: 720'
        count: 1
      - $type: regex_replace
        file: mtcnn-p.prototxt
        pattern: 'dim: 12'
        replacement: 'dim: 1280'
        count: 1
    model_optimizer_args:
      - --framework=caffe
      - --data_type=FP32
      - --input_shape=[1,3,720,1280]
      - --input=data
      - --mean_values=data[127.5,127.5,127.5]
      - --scale_values=data[128.0]
      - --output=conv4-2,prob1
      - --input_model=$dl_dir/mtcnn-p.caffemodel
      - --input_proto=$dl_dir/mtcnn-p.prototxt

I believe MO can ignore input dims at prototxt if --input_shape=[1,3,720,1280] specified (I can check it). Haven't found PyTorch models for now.

I wanted to adapt this PR after changes proposed at #276. That shouldn't be a problem.

IRDonch · 2019-08-13T17:02:06Z

We can also build it as a part of OpenCV's binaries for OpenVINO. Just adding OPENCV_EXTRA_MODULES_PATH during build time.

This seems impractical. It means that the OpenCV team would have to rebuild their binaries after every update to the model configuration file(s). From past experience I can say the models tend to get updated close to the time of the release due to last-minute Model Optimizer bugfixes, so the chances of OpenCV shipping with an out-of-sync module are quite high.

Python code is based on OMZ's downloader and https://github.com/opencv/opencv_extra/blob/master/testdata/dnn/download_models.py. Binaries uses only default Python packages (urllib, zipfile, tarfile, etc.)

I can see that, but it does nothing to address my concern. Also, it's not entirely true - when you're running Model Optimizer, you depend on all of its dependencies.

I believe MO can ignore input dims at prototxt if --input_shape=[1,3,720,1280] specified (I can check it).

That's good, but I know at least one regex-based fix that cannot be skipped - the one in googlenet-v2. In that case, the original prototxt file is actually ill-formed, and we have to patch it to make it work.

Haven't found PyTorch models for now.

They were merged recently. You might want to rebase.

I wanted to adapt this PR after changes proposed at #276. That shouldn't be a problem.

I'm sure you can adapt to that specific change, but the problem is that you have to. Any new feature in the downloader/converter would require you (or someone else) to adapt the module to it. That's not sustainable.

dkurt · 2019-08-13T17:16:59Z

@IRDonch, Thanks!

I agree that adding new module is a kind of overhead. There is another one option is to add it to regular OpenCV distribution. However the name of the package will be a bit different:

from cv2.dnn.open_model_zoo import ...

Or

from cv2.dnn.zoo import ...

This way we can use OMZ's topologies as optional ones and refactored https://github.com/opencv/opencv_extra/blob/master/testdata/dnn/download_models.py as a base.

We can start with it and check it's stability first.

vladimir-dudnik · 2019-08-13T17:50:26Z

@IRDonch

I'm sure you can adapt to that specific change, but the problem is that you have to. Any new feature in the downloader/converter would require you (or someone else) to adapt the module to it. That's not sustainable.

Can we rely on downloader/converter API to minimize needs of changes in case of any new features in them?

dkurt · 2019-08-13T18:28:35Z

@vladimir-dudnik, Proposed changes use pure Python to download files by URL. The module depends only on list_topologies.yml (or set of them after #276) and source code of some demos (for now it's text_recognition and human_pose_estimation).

Do not run MO for DLDT models. Create aliases for OpenVINO models by highest version.

Move topologies to cv2.open_model_zoo.topologies Enable cv2.open_model_zoo.TextRecognitionPipeline

Add DnnClassificationModel. Add some docs.

IRDonch · 2019-08-15T15:29:12Z

Can we rely on downloader/converter API to minimize needs of changes in case of any new features in them?

That's what I'm suggesting...

dkurt · 2019-08-15T15:52:28Z

@IRDonch, I'd like to propose the following solution: the package is still generated by OMZ structure (intel/ and public/ subfolders). For every model it's name is preserved in any way. In case of success - YAML is parser as well.

Then, at download() we will use downloader.py as a first priority tool for downloading (using the name of model we have). In case of failed download.py command (in example, when OpenCV is compiled with OMZ without OpenVINO and script has not been found) - try to download by existing code.

dkurt · 2019-08-15T16:41:21Z

On the other hand - there is a lightweight solution with just Python module which can only download and convert models without demos.

I'll propose a separate PR with it.

mmphego

Some suggestions
Overral great PR can't wait for it to be merged.

Less is more!!!

mmphego · 2020-09-16T15:55:04Z

ocv_module/open_model_zoo/README.md

@@ -0,0 +1,108 @@
+# Open Model Zoo
+
+This is OpenCV module which let you have interactive Open Model Zoo in Python.


This could be reworded as:

Suggested change

This is OpenCV module which let you have interactive Open Model Zoo in Python.

This is OpenCV module which lets you interact Open Model Zoo in Python.

mmphego · 2020-09-16T15:55:47Z

ocv_module/open_model_zoo/README.md

+```
+
+
+If you already have modules such opencv_contrib, you can combine it. In example,


Suggested change

If you already have modules such opencv_contrib, you can combine it. In example,

If you already have modules such `opencv_contrib`, you can combine it. In example,

mmphego · 2020-09-16T15:58:12Z

ocv_module/open_model_zoo/README.md

+topology = squeezenet1_0()
+```
+
+To infer network you can use as just paths downloaded files:


Suggested change

To infer network you can use as just paths downloaded files:

To infer a network you can just use paths to the downloaded files:

mmphego · 2020-09-16T16:00:25Z

ocv_module/open_model_zoo/README.md

+Some of networks may have pretty complicated pre- or post- processing procedures.
+Another models can be combined to solve interesting problems. For these kind of
+topologies you can use ready Algorithms. In example, to recognize text:


Suggested change

Some of networks may have pretty complicated pre- or post- processing procedures.

Another models can be combined to solve interesting problems. For these kind of

topologies you can use ready Algorithms. In example, to recognize text:

Some networks may have pretty complicated pre- or post-processing procedures.

Other models can be combined to solve interesting problems. For this kind of

topologies you can use ready Algorithms. This example shows how to recognize text:

mmphego · 2020-09-16T16:03:27Z

ocv_module/open_model_zoo/gen.py

+        if len(files) > 1:
+            config['model_url'], config['model_sha256'], config['model_path'] = getSource(files[1])
+
+    s = ', '.join(['{"%s", "%s"}' % (key, value) for key, value in config.items()])


No need to cast into a list.

Suggested change

s = ', '.join(['{"%s", "%s"}' % (key, value) for key, value in config.items()])

s = ', '.join('{"%s", "%s"}' % (key, value) for key, value in config.items())

dkurt force-pushed the py_open_model_zoo branch 5 times, most recently from 0d73fbd to 9926897 Compare August 7, 2019 14:01

dkurt force-pushed the py_open_model_zoo branch 2 times, most recently from 09d5c37 to eaec181 Compare August 10, 2019 19:43

dkurt force-pushed the py_open_model_zoo branch from 7553cb5 to ce226b7 Compare August 11, 2019 18:52

dkurt force-pushed the py_open_model_zoo branch from 810d387 to 85b2d34 Compare August 13, 2019 20:35

dkurt added 5 commits August 15, 2019 14:45

initial module 🐍

98f1848

Download in constructor 🐺

815e987

initial parsing with pyyaml 🦁

2cdfc39

Download from Google Drive 🐵

5efb951

Headers generation 🐘

3af70dc

dkurt added 17 commits August 15, 2019 14:45

Add Model Optimizer conversion 🐫

2a37cf4

Specify location of generated IRs and do not exit after MO 🐸

0a59bee

Download DLDT networks with different precisions. 🐜

7531c45

Do not run MO for DLDT models. Create aliases for OpenVINO models by highest version.

set input shape for DNN runner 🐆

5255522

Downloading progress bar 🦊

96ce98c

TextRecognition pipeline intro 🐟

cc204f1

Fix Model Optimizer 2019R2 call 😼

ec26cad

Move topologies to cv2.open_model_zoo.topologies Enable cv2.open_model_zoo.TextRecognitionPipeline

Add some docs and Python tests 🪲

2a3ca65

Open Model Zoo in C++ 🐳

bfdf9ab

Default arguments for TextRecognitionPipeline 🐨

78fc275

Add parameters for TextRecognitionPipeline 🕷️

087e186

Add DnnClassificationModel. Add some docs.

Move description and license to docs. Zip archieves management 🐉

55aa360

Remove archives after exctract. Custom Model Optimizer flags 🕊️

0458ae0

Use Human body pose estimation demo 🦉

7b3b956

Reuse some code from text recognition demo 🐰

e578750

Set target device for human pose 🐔

0b4b914

Consider planar configs structure 🦀

f7e8462

dkurt force-pushed the py_open_model_zoo branch from 85b2d34 to f7e8462 Compare August 15, 2019 12:38

Select target devices for TextRecognitionPipeline 🐝

c0ae5f3

dkurt mentioned this pull request Aug 15, 2019

Dynamic Open Model Zoo as Python module #304

Closed

7 tasks

dkurt force-pushed the py_open_model_zoo branch 2 times, most recently from 99230a9 to c0ae5f3 Compare August 21, 2019 07:33

dkurt mentioned this pull request Oct 18, 2019

samples: refactor DNN model downloading opencv/opencv#12186

Closed

vshampor force-pushed the develop branch from 8de822c to 9df4afd Compare November 12, 2019 15:22

dkurt mentioned this pull request May 12, 2020

Suggestion for a OpenPose simplified module. opencv/opencv#17271

Closed

mmphego reviewed Sep 16, 2020

View reviewed changes

dkurt closed this Jan 21, 2021

		@@ -0,0 +1,108 @@
		# Open Model Zoo

		This is OpenCV module which let you have interactive Open Model Zoo in Python.

	This is OpenCV module which let you have interactive Open Model Zoo in Python.
	This is OpenCV module which lets you interact Open Model Zoo in Python.

		```


		If you already have modules such opencv_contrib, you can combine it. In example,

	If you already have modules such opencv_contrib, you can combine it. In example,
	If you already have modules such `opencv_contrib`, you can combine it. In example,

	To infer network you can use as just paths downloaded files:
	To infer a network you can just use paths to the downloaded files:

	s = ', '.join(['{"%s", "%s"}' % (key, value) for key, value in config.items()])
	s = ', '.join('{"%s", "%s"}' % (key, value) for key, value in config.items())

Conversation

dkurt commented Aug 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pipelines

Uh oh!

openvino-pushbot commented Aug 4, 2019

Uh oh!

openvino-pushbot commented Aug 4, 2019

Uh oh!

snosov1 commented Aug 8, 2019

Uh oh!

dkurt commented Aug 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

snosov1 commented Aug 8, 2019

Uh oh!

dkurt commented Aug 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jrosebr1 commented Aug 11, 2019

Uh oh!

dkurt commented Aug 11, 2019

Uh oh!

jrosebr1 commented Aug 12, 2019

Uh oh!

dkurt commented Aug 12, 2019

Uh oh!

IRDonch commented Aug 12, 2019

Uh oh!

dkurt commented Aug 12, 2019

Uh oh!

IRDonch commented Aug 13, 2019

Uh oh!

dkurt commented Aug 13, 2019

Uh oh!

vladimir-dudnik commented Aug 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dkurt commented Aug 13, 2019

Uh oh!

IRDonch commented Aug 15, 2019

Uh oh!

dkurt commented Aug 15, 2019

Uh oh!

dkurt commented Aug 15, 2019

Uh oh!

mmphego left a comment

Choose a reason for hiding this comment

Uh oh!

mmphego Sep 16, 2020

Choose a reason for hiding this comment

Uh oh!

mmphego Sep 16, 2020

Choose a reason for hiding this comment

Uh oh!

mmphego Sep 16, 2020

Choose a reason for hiding this comment

Uh oh!

mmphego Sep 16, 2020

Choose a reason for hiding this comment

Uh oh!

mmphego Sep 16, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dkurt commented Aug 3, 2019 •

edited

Loading

dkurt commented Aug 8, 2019 •

edited

Loading

dkurt commented Aug 8, 2019 •

edited

Loading

vladimir-dudnik commented Aug 13, 2019 •

edited

Loading