Support Yolov4 by l-bat · Pull Request #17185 · opencv/opencv

l-bat · 2020-04-29T10:29:11Z

Merge with extra: opencv/opencv_extra#753
resolves: #17148

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under OpenCV (BSD) License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

force_builders=Custom,Custom Win,Custom Mac
build_image:Custom=ubuntu-openvino-2020.1.0:16.04
build_image:Custom Win=openvino-2020.2.0
build_image:Custom Mac=openvino-2020.2.0

test_modules:Custom=dnn,python2,python3,java
test_modules:Custom Win=dnn,python2,python3,java
test_modules:Custom Mac=dnn,python2,python3,java

buildworker:Custom=linux-1
# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*

Performance tests
CPU: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz

Net	Backend	branch 3.4	branch yolov4
TinyYoloVOC	DNN_BACKEND_INFERENCE_ENGINE	23.29	23.15
TinyYoloVOC	DNN_BACKEND_OPENCV	35.24	35.29
YoloVOC @ 416x416	DNN_BACKEND_INFERENCE_ENGINE	88.42	88.54
YoloVOC @ 416x416	DNN_BACKEND_OPENCV	129.50	130.03
YOLOv3 @ 416x416	DNN_BACKEND_INFERENCE_ENGINE	200.76	200.93
YOLOv3 @ 416x416	DNN_BACKEND_OPENCV	273.93	274.48
YOLOv4 @ 416x416	DNN_BACKEND_INFERENCE_ENGINE	-	537.48
YOLOv4 @ 416x416	DNN_BACKEND_OPENCV	-	712.38

CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz x8

	DNN_BACKEND_OPENCV	DNN_BACKEND_INFERENCE_ENGINE
TinyYoloVOC	24.34ms	17.68ms
YoloVOC @ 416x416	92.22ms	68.46ms
YOLOv3 @ 416x416	208.74ms	155.62ms
YOLOv4 @ 416x416	471.89ms	369.82ms

import cv2 as cv

net = cv.dnn_DetectionModel(yolov4.cfg', 'yolov4.weights)
net.setInputSize(608, 608)
net.setInputScale(1.0 / 255)
net.setInputSwapRB(True)

frame = cv.imread('example.jpg')

with open('coco.names', 'rt') as f:
    names = f.read().rstrip('\n').split('\n')

classes, confidences, boxes = net.detect(frame, confThreshold=0.1, nmsThreshold=0.4)
for classId, confidence, box in zip(classes.flatten(), confidences.flatten(), boxes):
    label = '%.2f' % confidence
    label = '%s: %s' % (names[classId], label)
    labelSize, baseLine = cv.getTextSize(label, cv.FONT_HERSHEY_SIMPLEX, 0.5, 1)
    left, top, width, height = box
    top = max(top, labelSize[1])
    cv.rectangle(frame, box, color=(0, 255, 0), thickness=3)
    cv.rectangle(frame, (left, top - labelSize[1]), (left + labelSize[0], top + baseLine), (255, 255, 255), cv.FILLED)
    cv.putText(frame, label, (left, top), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 0))

cv.imshow('out', frame)

Mish activation function
Mish activation function in origin paper
𝑓(𝑥) = 𝑥 * 𝑡𝑎𝑛ℎ(softplus(𝑥)) where softplus (𝑥) = ln(1 + 𝑒xp(𝑥))

Darknet implementation softplus

float softplus(float x, float threshold = 20) {
    if (x > threshold) return x;              // too large
    else if (x < -threshold) return expf(x);  // too small
    return logf(expf(x) + 1);
}

We can use the original softplus function for darknet.
Proof:

x < -20: let's compare ln (1 + exp( 𝑥 )) with exp(x). exp (x) - > 0 and ln(1 + exp(x)) - > 0 when x - > -infinity. The maximum difference between ln (1 + exp ( 𝑥 )) and x at the maximum value of x (x=-20). exp (-20)= 2.061153622438558e-09 vs ln(1 + exp (-20)) = 2.0611536900435727e-09. both values are very close.
x > 20: let's compare ln(1 + 𝑒xp(𝑥)) with x. ln(1 + 𝑒xp(𝑥)) -> ln(𝑒xp(𝑥)) = x at X->infinity
The maximum difference between ln (1 + exp( 𝑥 )) and x with a minimum of x (x=20). ln(1 + exp(20)) = 20.000000002061153. So the difference is lower than 10e-8.

dkurt · 2020-04-29T14:13:01Z

Can you please describe what "before" and "after" means?

dkurt · 2020-04-29T14:14:10Z

Yolov3 | OpenCV | 17210.52 | 14720.49

Have you used perf_dnn for bahcnmarking? Seems incorrect for me

modules/dnn/src/layers/elementwise_layers.cpp

l-bat · 2020-04-29T14:24:19Z

Can you please describe what "before" and "after" means?

“Before” means the results of running performance tests before this PR changes (without Yolov4 support)

l-bat · 2020-04-29T14:24:58Z

Yolov3 | OpenCV | 17210.52 | 14720.49

Have you used perf_dnn for bahcnmarking? Seems incorrect for me

Yes, I used perf_dnn

modules/dnn/src/layers/region_layer.cpp

dkurt

👍 Well done!

rgov · 2020-04-30T18:56:04Z

Great to see that this is in the 3.4.11 milestone. Will this reach the 4.x branch as well?

l-bat · 2020-05-01T08:53:43Z

Thank you for your interest @rgov, @wooseokyourself! These PR changes will be merged into master branch next week.

dkurt · 2020-05-01T14:47:43Z

@sqiprasanna, please read all the messages carefully - you're trying to use OpenCV 4.2.0. We just merged the changes yesterday.

AlexeyAB · 2020-05-01T16:01:34Z

@l-bat @dkurt
Is Yolov4 more than 2 times slower than Yolov3 due to MISH-activation?

sqiprasanna · 2020-05-01T16:24:31Z

@l-bat Thanks for the reply. I just used this yolov4 for training on my custom dataset. It really worked well on detecting images. I'm more excited to test this model on real-time detection using OpenCV. As I see here you're trying to include yolov4 in OpenCV dnn. I just want to really appreciate the work you've done and would like to see adding yolov4 into OpenCV.

Thank you again for your hard work. Have a nice day.

dkurt · 2020-05-01T17:49:31Z

@AlexeyAB, it might be, yes. I'd like to recommend to open an issue for YOLOv4 optimization (some research with per-layer performance will be very helpful). Thanks!

marcusbrito · 2020-05-21T14:08:13Z

Hello,
I had understood that yolov4 was already working on version 3.4.10, but I tried and got the error:

cv2.error: OpenCV(3.4.10) /opencv-3.4.10/modules/dnn/src/darknet/darknet_io.cpp:821: error: (-212:Parsing error) Unsupported activation: mish in function 'ReadDarknetFromCfgStream'

I'm buildin OpenCV 3.4.10 from source on a docker container. I'm cloning this repository:
https://github.com/Itseez/opencv/archive/3.4.10.zip

I'm not very familiar with this, so can someone please give me a simple answer like "It's not working yet" or "It's already working, you can git clone this link", instead of referencing another one confused thread.

mshabunin · 2020-05-21T17:16:13Z

It's not working yet, wait for the next release or get latest master or 3.4 branch version and build it manually.

OrriMandarin · 2020-05-27T08:23:38Z

Hi,
Any date for the next release ?

Thanks

akhilgkrishnan

good

dkurt · 2020-06-26T10:53:55Z

@l-bat @dkurt
Is Yolov4 more than 2 times slower than Yolov3 due to MISH-activation?

@AlexeyAB, please take a look at Mish optimization for CPU: #17624

l-bat added the category: dnn label Apr 29, 2020

l-bat added this to the 3.4.11 milestone Apr 29, 2020

l-bat linked an issue Apr 29, 2020 that may be closed by this pull request

Feature-request: State-of-art Yolo v4 Detector #17148

Closed

l-bat force-pushed the yolo_v4 branch 2 times, most recently from dbb3524 to 856b1f5 Compare April 29, 2020 11:50

Support Yolov4

856b1f5

l-bat force-pushed the yolo_v4 branch from 3b7fffc to 25c3e2f Compare April 29, 2020 13:06

Skip Mish on OpenVINO 2020.2

25c3e2f

dkurt reviewed Apr 29, 2020

View reviewed changes

modules/dnn/src/layers/elementwise_layers.cpp Outdated Show resolved Hide resolved

dkurt reviewed Apr 29, 2020

View reviewed changes

modules/dnn/src/layers/region_layer.cpp Outdated Show resolved Hide resolved

Revert Mish

8e36db4

l-bat force-pushed the yolo_v4 branch 2 times, most recently from cfa41e6 to 4bbb21d Compare April 30, 2020 10:24

Refactoring

4bbb21d

dkurt approved these changes Apr 30, 2020

View reviewed changes

dkurt self-assigned this Apr 30, 2020

dkurt linked an issue Apr 30, 2020 that may be closed by this pull request

Error when using cv.dnn.readNetFromDarknet function with OpenCV 4.2.0 version #17166

Closed

alalek merged commit a5696da into opencv:3.4 Apr 30, 2020

l-bat mentioned this pull request Apr 30, 2020

Feature-request: State-of-art Yolo v4 Detector #17148

Closed

shimat mentioned this pull request Apr 30, 2020

yolov4 Unsupported activation: mish shimat/opencvsharp#927

Closed

AlexeyAB mentioned this pull request May 3, 2020

YOLOv4 showing an error in opencv dnn module AlexeyAB/darknet#5465

Open

dkurt mentioned this pull request May 5, 2020

Opencv dnn python sample is slow with darknet model #17204

Closed

alalek mentioned this pull request May 6, 2020

Merge 3.4 #17231

Merged

IDGate-ai mentioned this pull request May 18, 2020

About yolov4? opencv/opencv-python#327

Closed

AlexeyAB mentioned this pull request May 27, 2020

Support with OpenCV dnn module for Yolov4 model. AlexeyAB/darknet#5671

Open

akhilgkrishnan reviewed May 27, 2020

View reviewed changes

opencv locked as too heated and limited conversation to collaborators May 27, 2020

Uh oh!

Conversation

l-bat commented Apr 29, 2020 • edited by dkurt Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

dkurt commented Apr 29, 2020

Uh oh!

dkurt commented Apr 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

l-bat commented Apr 29, 2020

Uh oh!

l-bat commented Apr 29, 2020

Uh oh!

Uh oh!

dkurt left a comment

Choose a reason for hiding this comment

Uh oh!

rgov commented Apr 30, 2020

Uh oh!

l-bat commented May 1, 2020

Uh oh!

dkurt commented May 1, 2020

Uh oh!

AlexeyAB commented May 1, 2020

Uh oh!

sqiprasanna commented May 1, 2020

Uh oh!

dkurt commented May 1, 2020

Uh oh!

marcusbrito commented May 21, 2020

Uh oh!

mshabunin commented May 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

OrriMandarin commented May 27, 2020

Uh oh!

akhilgkrishnan left a comment

Choose a reason for hiding this comment

Uh oh!

dkurt commented Jun 26, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

l-bat commented Apr 29, 2020 •

edited by dkurt

Loading

dkurt commented Apr 29, 2020 •

edited

Loading

mshabunin commented May 21, 2020 •

edited

Loading