Support FlowNet2 model by l-bat · Pull Request #16575 · opencv/opencv

l-bat · 2020-02-13T13:32:59Z

Merge with extra: opencv/opencv_extra#708
relates #16319

Supported layers:

Added tests:

force_builders=Custom,Custom Win,Custom Mac
build_image:Custom=ubuntu-openvino-2020.1.0:16.04
Xbuild_image:Custom=ubuntu-openvino-2020.2.0:16.04
build_image:Custom Win=openvino-2020.1.0
build_image:Custom Mac=openvino-2020.1.0

test_modules:Custom=dnn,python2,python3,java
test_modules:Custom Win=dnn,python2,python3,java
test_modules:Custom Mac=dnn,python2,python3,java

buildworker:Custom=linux-1
# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*

Example

Inputs

Output

modules/dnn/src/layers/scale_layer.cpp

modules/dnn/src/layers/correlation_layer.cpp

modules/dnn/src/caffe/caffe_importer.cpp

modules/dnn/include/opencv2/dnn/all_layers.hpp

dkurt · 2020-04-09T14:57:54Z

modules/dnn/src/layers/normalize_bbox_layer.cpp

 }

+
+class ChannelNormLayerImpl CV_FINAL : public ChannelNormLayer


What do you think if instead of adding this custom layer we can modify .ptototxt replacing it to series of layers that give the sample result? In example,

x*x -> reshape (1, 1, C, -1) -> ave pool by H -> multiply by C -> reshape back

Or, which is better, to do the same in the importer

In this case, we need to store outShapes in importer. Maybe we will have problems with nets without .caffemodel.

Maybe we will have problems with nets without .caffemodel.

Why so?

We have already tried to calculate outShapes in caffe_importer. Some tests failed in getMemoryShapes method if we don't use .caffemodel for nets with weighted layers.

I modified channel_norm.prototxt to a series of layers.

modules/dnn/src/layers/resize_layer.cpp

modules/dnn/src/layers/scale_layer.cpp

asmorkalov · 2020-04-15T06:51:18Z

@dkurt Friendly reminder.

dkurt · 2020-04-15T07:22:39Z

modules/dnn/src/layers/resize_layer.cpp

+            outputs[0][3] = inputs[1][3];
+            if ((interpolation == "linear" && (factor_w != 1.0 || factor_h != 1.0)) ||
+                (interpolation == "nearest" && (factor_w < 1.0 || factor_h < 1.0)))
+                    CV_Error(Error::StsNotImplemented, "Unsupported Resize mode");


Why they are not supported?

Caffe result with these factors is not equal to OpenCV resize

asmorkalov · 2020-04-28T07:30:38Z

@dkurt Friendly reminder.

asmorkalov · 2020-05-14T11:47:01Z

@dkurt @l-bat Friendly reminder.

modules/dnn/src/layers/resize_layer.cpp

dkurt · 2020-05-18T10:56:42Z

samples/dnn/optical_flow.py

+
+To run model in OpenCV modify Flownetv2.prototxt.
+
+change ChannelNorm layers to series of layers:


How many ChannelNorm layers inside? Maybe it make sense to put modified .prototxt to opencv_extra?

I put modified .prototxt and .caffemodel to cloud.

samples/dnn/optical_flow.py

dkurt · 2020-05-18T14:20:42Z

samples/dnn/optical_flow.py

+You can download the converted .caffemodel model from https://drive.google.com/open?id=16qvE9VNmU39NttpZwZs81Ga8VYQJDaWZ
+or convert .h5 model to .caffemodel yourself.
+Download .prototxt from https://drive.google.com/open?id=19bo6SWU2p8ZKvjXqMKiCPdK8mghwDy9b
+or modify original Flownetv2.prototxt - change ChannelNorm layers to series of layers as in link above.


Please rephrase so user can understand which are required steps and that are optional. Otherwise one can download .caffemodel then "To get original model". I'd like to put everything in one place and say "Download this and that. Otherwise download original model from here".

Changes required in prototxt file I'd like to recommend to write right in the prototxt file as a reference to origin prototxt file.

modules/dnn/src/layers/resize_layer.cpp

dkurt

👍 Well done!

l-bat added feature category: dnn labels Feb 13, 2020

l-bat added this to the 3.4.10 milestone Feb 13, 2020

dkurt reviewed Feb 13, 2020

View reviewed changes

modules/dnn/src/layers/scale_layer.cpp Outdated Show resolved Hide resolved

dkurt reviewed Feb 13, 2020

View reviewed changes

modules/dnn/src/layers/scale_layer.cpp Outdated Show resolved Hide resolved

dkurt reviewed Feb 13, 2020

View reviewed changes

modules/dnn/src/layers/scale_layer.cpp Outdated Show resolved Hide resolved

dkurt reviewed Feb 13, 2020

View reviewed changes

modules/dnn/src/layers/scale_layer.cpp Outdated Show resolved Hide resolved

dkurt reviewed Feb 14, 2020

View reviewed changes

modules/dnn/src/layers/scale_layer.cpp Show resolved Hide resolved

dkurt reviewed Feb 18, 2020

View reviewed changes

modules/dnn/src/layers/correlation_layer.cpp Outdated Show resolved Hide resolved

l-bat force-pushed the flownet2 branch from bddb16a to 9ce8768 Compare February 19, 2020 07:46

l-bat added 13 commits February 21, 2020 10:34

Support DataAugmentation layer

c8d6a41

Fix warnings

a683707

Fix comments

0684d7b

Support Correlation layer

e45cf93

TEST

6bae3c3

Support Correlation layer

886d982

Supported Accum and FlowWarp layers

4bd7963

Supported ChannelNorm layer

c8fa35c

Supported Resample with inputs.size() > 1

e9b7400

Fixed comments

1e732f6

Refactoring

fa9c034

Added tests

c764bab

Add resample test

d31dd17

l-bat force-pushed the flownet2 branch from 4f11e53 to d31dd17 Compare February 21, 2020 09:25

l-bat added 5 commits February 21, 2020 17:24

Added asserts in resize layer

c02cb9c

Updated DataAugmentation layer

6783546

Update convolution layer

2235650

Refactoring

b953eab

Fix data augmentation layer

90c6a20

dkurt reviewed Mar 4, 2020

View reviewed changes

modules/dnn/src/caffe/caffe_importer.cpp Outdated Show resolved Hide resolved

dkurt reviewed Apr 9, 2020

View reviewed changes

modules/dnn/include/opencv2/dnn/all_layers.hpp Show resolved Hide resolved

dkurt reviewed Apr 9, 2020

View reviewed changes

modules/dnn/src/layers/resize_layer.cpp Outdated Show resolved Hide resolved

dkurt reviewed Apr 9, 2020

View reviewed changes

modules/dnn/src/layers/scale_layer.cpp Outdated Show resolved Hide resolved

Remove useless resize type

803babb

l-bat force-pushed the flownet2 branch from c9b7a47 to 803babb Compare April 10, 2020 12:33

Used ResizeLayer in Accum

5e5027a

l-bat force-pushed the flownet2 branch from 746f170 to 5e5027a Compare April 13, 2020 14:02

Split ChannelNormLayer

1605ca4

dkurt reviewed Apr 15, 2020

View reviewed changes

l-bat force-pushed the flownet2 branch 2 times, most recently from 50af357 to afc2e4b Compare May 15, 2020 11:56

l-bat added 2 commits May 15, 2020 14:57

Delete duplicate assert

afc2e4b

Add sample

9bf13b5

dkurt reviewed May 18, 2020

View reviewed changes

modules/dnn/src/layers/resize_layer.cpp Show resolved Hide resolved

dkurt reviewed May 18, 2020

View reviewed changes

samples/dnn/optical_flow.py Outdated Show resolved Hide resolved

dkurt reviewed May 18, 2020

View reviewed changes

samples/dnn/optical_flow.py Outdated Show resolved Hide resolved

dkurt reviewed May 18, 2020

View reviewed changes

samples/dnn/optical_flow.py Outdated Show resolved Hide resolved

Fix sample

8ea2a48

dkurt reviewed May 18, 2020

View reviewed changes

modules/dnn/src/layers/resize_layer.cpp Outdated Show resolved Hide resolved

dkurt linked an issue May 18, 2020 that may be closed by this pull request

DNN: Exception "inputs.size() >= 2" on calling Forward #16319

Closed

Added colormap

7de4fa5

dkurt approved these changes May 19, 2020

View reviewed changes

alalek merged commit d991c22 into opencv:3.4 May 19, 2020

alalek mentioned this pull request May 22, 2020

Merge 3.4 #17351

Merged

		}


		class ChannelNormLayerImpl CV_FINAL : public ChannelNormLayer


		To run model in OpenCV modify Flownetv2.prototxt.

		change ChannelNorm layers to series of layers:

Uh oh!

Conversation

l-bat commented Feb 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkurt Apr 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkurt Apr 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

asmorkalov commented Apr 15, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asmorkalov commented Apr 28, 2020

Uh oh!

asmorkalov commented May 14, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkurt May 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dkurt left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

l-bat commented Feb 13, 2020 •

edited

Loading

dkurt Apr 9, 2020 •

edited

Loading

dkurt Apr 10, 2020 •

edited

Loading

dkurt May 18, 2020 •

edited

Loading