Skip to content

Flownet2 with anysize#17919

Merged
opencv-pushbot merged 1 commit intoopencv:3.4from
zhaoyue-zephyrus:flownet2_with_anysize
Aug 26, 2020
Merged

Flownet2 with anysize#17919
opencv-pushbot merged 1 commit intoopencv:3.4from
zhaoyue-zephyrus:flownet2_with_anysize

Conversation

@zhaoyue-zephyrus
Copy link
Copy Markdown
Contributor

@zhaoyue-zephyrus zhaoyue-zephyrus commented Jul 22, 2020 •
edited by alalek
Loading

merge with extra: opencv/opencv_extra#780

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under OpenCV (BSD) License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
  • The PR is proposed to proper branch
  • There is reference to original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake
force_builders=Custom,Custom Win,Custom Mac
build_image:Custom=ubuntu-openvino-2020.4.0:16.04
build_image:Custom Win=openvino-2020.3.0
build_image:Custom Mac=openvino-2020.3.0

test_modules:Custom=dnn,python2,python3,java
test_modules:Custom Win=dnn,python2,python3,java
test_modules:Custom Mac=dnn,python2,python3,java

buildworker:Custom=linux-1
# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*
opencv_extra=data_aug_with_different_size

@zhaoyue-zephyrus
Copy link
Copy Markdown
Contributor Author

zhaoyue-zephyrus commented Jul 22, 2020

Hi, this PR helps to enhance the flownet2 (samples/dnn/optical_flow.py) example with arbitrary size.

Simply running python samples/dnn/optical_flow.py -i ${TEST_VIDEO} --height 240 --width 320 would produce the following error:

[ERROR:0] global /home/yzhao/mmaction-release/third_party/opencv-4.4.0/modules/dnn/src/dnn.cpp (3444) getLayerShapesRecursively OPENCV/DNN: [DataAugmentation]:(img0s_aug): getMemoryShapes() throws exception. inputs=1 outputs=0/1 blobs=3
[ERROR:0] global /home/yzhao/mmaction-release/third_party/opencv-4.4.0/modules/dnn/src/dnn.cpp (3447) getLayerShapesRecursively     input[0] = [ 1 3 240 320 ]
[ERROR:0] global /home/yzhao/mmaction-release/third_party/opencv-4.4.0/modules/dnn/src/dnn.cpp (3455) getLayerShapesRecursively     blobs[0] = CV_32FC1 [ 1 1 1 1 ]
[ERROR:0] global /home/yzhao/mmaction-release/third_party/opencv-4.4.0/modules/dnn/src/dnn.cpp (3455) getLayerShapesRecursively     blobs[1] = CV_32FC1 [ 1 3 320 448 ]
[ERROR:0] global /home/yzhao/mmaction-release/third_party/opencv-4.4.0/modules/dnn/src/dnn.cpp (3455) getLayerShapesRecursively     blobs[2] = CV_32FC1 [ 1 3 1 1 ]
[ERROR:0] global /home/yzhao/mmaction-release/third_party/opencv-4.4.0/modules/dnn/src/dnn.cpp (3457) getLayerShapesRecursively Exception message: OpenCV(4.4.0) /home/yzhao/mmaction-release/third_party/opencv-4.4.0/modules/dnn/src/layers/scale_layer.cpp:393: error: (-215:Assertion failed) blobs[1].total() == total(inputs[0], 1) in function 'getMemoryShapes'

Traceback (most recent call last):
  File "samples/dnn/optical_flow.py", line 84, in <module>
    flow = opt_flow.compute_flow(first_frame, second_frame)
  File "samples/dnn/optical_flow.py", line 31, in compute_flow
    flow = self.net.forward()
cv2.error: OpenCV(4.4.0) /home/yzhao/mmaction-release/third_party/opencv-4.4.0/modules/dnn/src/layers/scale_layer.cpp:393: error: (-215:Assertion failed) blobs[1].total() == total(inputs[0], 1) in function 'getMemoryShapes'

@dkurt dkurt requested a review from l-bat July 23, 2020 06:04
@asmorkalov asmorkalov added category: dnn pr: needs test New functionality requires minimal tests set labels Jul 23, 2020
@asmorkalov
Copy link
Copy Markdown
Contributor

asmorkalov commented Jul 23, 2020

@zhaoyue-zephyrus Could you add test for the case you implemented?

l-bat
l-bat reviewed Jul 23, 2020
View reviewed changes
samples/dnn/optical_flow.py
parser.add_argument('--proto', '-p', default='FlowNet2_deploy.prototxt', help='Path to prototxt.')
parser.add_argument('--height', default=320, type=int, help='Input height')
parser.add_argument('--width', default=448, type=int, help='Input width')
parser.add_argument('--proto', '-p', default='FlowNet2_deploy_anysize.prototxt', help='Path to prototxt.')
Copy link
Copy Markdown
Contributor

@l-bat l-bat Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to change the default proto name?

Copy link
Copy Markdown
Contributor Author

@zhaoyue-zephyrus zhaoyue-zephyrus Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure we can keep the default proto name. This name (with _anysize) is the one I attached in the google drive.

Copy link
Copy Markdown
Contributor

@l-bat l-bat Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In flownet2-models.tar.gz from https://github.com/lmb-freiburg/flownet2/blob/master/models/download-models.sh default name is FlowNet2_deploy.prototxt.template

Copy link
Copy Markdown
Contributor Author

@zhaoyue-zephyrus zhaoyue-zephyrus Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. However, the default FlowNet2_deploy.prototxt.template doesn't work with opencv-4.4.0 directly (maybe lacking certain layer's support). I modify the proto in https://drive.google.com/open?id=19bo6SWU2p8ZKvjXqMKiCPdK8mghwDy9b to have similar $-quoted variables link FlowNet2_deploy.prototxt.template.

@zhaoyue-zephyrus
Copy link
Copy Markdown
Contributor Author

zhaoyue-zephyrus commented Jul 31, 2020

The test samples have been added in this PR.

@l-bat
Copy link
Copy Markdown
Contributor

l-bat commented Jul 31, 2020

The test samples have been added in this PR.

Thank you! Please add test to https://github.com/opencv/opencv/blob/3.4/modules/dnn/test/test_layers.cpp

l-bat
l-bat reviewed Aug 6, 2020
View reviewed changes
l-bat
l-bat reviewed Aug 7, 2020
View reviewed changes
@l-bat l-bat removed the pr: needs test New functionality requires minimal tests set label Aug 10, 2020
@l-bat
Copy link
Copy Markdown
Contributor

l-bat commented Aug 10, 2020

Please rebase your PR to 3.4 branch

l-bat
l-bat reviewed Aug 10, 2020
View reviewed changes
@zhaoyue-zephyrus
Copy link
Copy Markdown
Contributor Author

zhaoyue-zephyrus commented Aug 10, 2020

Please rebase your PR to 3.4 branch

When you talk about rebase, do you literally mean using git rebase 3.4? It seems that conflicts frequently occur that have to be resolved manually.

@zhaoyue-zephyrus zhaoyue-zephyrus changed the base branch from master to 3.4 August 10, 2020 16:48
@l-bat
Copy link
Copy Markdown
Contributor

l-bat commented Aug 11, 2020

Can you squash all commits? Looks like this warning https://pullrequest.opencv.org/buildbot/builders/precommit_docs/builds/26251/steps/patch%20size%20opencv_extra/logs/stdio related to this PR.

@zhaoyue-zephyrus
support flownet2 with arbitary input size
e231be8 
revise default proto to match the filename in documentations

fix a bug

beautify python codes

fix bug

beautify codes

add test samples with larger/smaller size

remove unless code

using bytearray without creating tmp file

remove useless codes
l-bat
l-bat approved these changes Aug 12, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@l-bat l-bat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done! Thank you so much 👍

@l-bat l-bat added this to the 3.4.12 milestone Aug 13, 2020
@opencv-pushbot opencv-pushbot merged commit 7927228 into opencv:3.4 Aug 26, 2020
@alalek alalek mentioned this pull request Aug 26, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@l-bat l-bat l-bat approved these changes

Assignees

@l-bat l-bat

Labels

category: dnn

Projects

None yet

Milestone

3.4.12

Development

Successfully merging this pull request may close these issues.

4 participants

@zhaoyue-zephyrus @asmorkalov @l-bat @opencv-pushbot