using cv2 dnn interface to run yolov8 model by Tsai-chia-hsiang · Pull Request #24396 · opencv/opencv

Tsai-chia-hsiang · 2023-10-13T08:33:15Z

This is a sample code for using opencv dnn interface to run ultralytics yolov8 model for object detection.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
[] There is a reference to the original bug report and related work
[] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
[] The feature is well documented and sample code can be built with the project CMake

samples/dnn/dnn_model_runner/dnn_conversion/ultralytics/predict/dog1.jpg

samples/dnn/dnn_model_runner/dnn_conversion/ultralytics/yolov8_cv.py

samples/dnn/object_detection.py

Tsai-chia-hsiang · 2023-10-25T14:20:44Z

@dkurt
Hi, during refactoring the code, we got a question about line 54:

However, according to the comment above it, shoudn't it be score = detection[4:] ? Since detection[0,1,2,3] are the bbox x,y,w,h and [4: ] are the score.

Or Do I misunderstand the usage of this line ?

Thanks

samples/dnn/models.yml

dkurt · 2023-11-09T09:00:11Z

@Tsai-chia-hsiang, made some changes with sample update and common parts unification. Please give your comment on the padding preprocessing (I removed it for now). If you don't mind, we can introduce it in the next PR.

Also, I have replaced onnx URL to Ultralytics's .pt model with a comment how to convert to ONNX.

Now sample works with the following commands:
YOLOv8 (Ultralytics):

python samples/dnn/object_detection.py yolov8 --model ~/yolov8net.onnx --input kodim23.png

YOLOv4 (Darknet):

python samples/dnn/object_detection.py yolo --model ~/yolov4.weights --input kodim23.png

SSD (Caffe):

python samples/dnn/object_detection.py ssd_caffe --input kodim23.png

Tsai-chia-hsiang · 2023-11-09T09:20:05Z

@dkurt
Ok, so should I open another PR ? or just add the comment and commit in this PR is fine ?
Thanks.

dkurt · 2023-11-09T09:29:05Z

@Tsai-chia-hsiang, I recommend to keep this PR just enable YOLOv8 postprocessing. Need more time to think about the pre-processing proposal.

Tsai-chia-hsiang · 2023-11-09T11:16:10Z

OK, I see.

Thanks!

asmorkalov · 2023-11-09T12:34:05Z

 python samples/dnn/object_detection.py yolov8 --model ./samples/yolov8net.onnx --input ./samples/data/lena.jpg 
[ WARN:0@0.020] global samples.cpp:61 findFile cv::samples::findFile('object_detection_classes_yolov4.txt') => ''
Traceback (most recent call last):
  File "samples/dnn/object_detection.py", line 90, in <module>
    net = cv.dnn.readNet(args.model, args.config, args.framework)
cv2.error: OpenCV(4.8.0-dev) /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/onnx/onnx_importer.cpp:279: error: (-210:Unsupported format or combination of formats) Failed to parse ONNX model: ./samples/yolov8net.onnx in function 'ONNXImporter'

dkurt · 2023-11-09T12:50:17Z

@asmorkalov , dis you convert a model to ONNX or used https://github.com/JasonLin1110/ultralytics_yolov8_onnx_model/blob/main/yolov8net.onnx ? I tested second option, let me try again.

Update: convert step worked for me too. Can you please check that ./samples/yolov8net.onnx is a correct path? Maybe ./samples/yolov8n.onnx? Also, please make sure Torch for CPU is used (pip install torch torchvisoon --index-url https://download.pytorch.org/whl/cpu)

add yolov8 to object_detection.py and models.yml

change yoloV8 readNetFromONNX to readNet, change lastLayer.type == 'Identity' to args.postprocessing == 'yolov8'

add padding image to blobFromImage. Solve case yolov8 doen't have config(args.config == None), so cv.samples.findFile(args.config) will return error

add a if else in nms_indices = nms_indices[:, 0] if len(nms_indices) else [], because yolov8's nms_indices is 1-d array

Correct yolov8 onnx's url and classes

minScore, minClassLoc, x

asmorkalov · 2023-11-16T07:46:29Z

Ok. I found out the issue reason.

config for download_models.py contains link to pt, but not onnx. The file is renamed to .onnx as it's defined in yaml config.
In case if I use ONNX from https://github.com/JasonLin1110/ultralytics_yolov8_onnx_model/blob/main/yolov8net.onnx the sample works fine.

asmorkalov · 2023-11-16T07:40:09Z

samples/dnn/models.yml

+# model.export(format="onnx")
+yolov8:
+  load_info:
+    url: "https://github.com/ultralytics/assets/releases/download/v0.0.0/yolov8n.pt"


ONNX link is required here.

It's better to rename model: "yolov8net.onnx" to model: "yolov8n.pt".

config for download_models.py contains link to pt, but not onnx. The file is renamed to .onnx as it's defined in yaml config.

I have not seen entry for YOLOv8 in download_models.py except https://github.com/CVHub520/X-AnyLabeling/releases/download/v0.1.0/yolov8n.onnx which is not an official repo. Can we keep a link to .pt?

Oh, excuse me. I just missed that there is another https://github.com/opencv/opencv/blob/4.x/samples/dnn/download_models.py

Replaced to URL from opencv_extra

asmorkalov

👍
Tested manually with current OpenCV 4.x, Ubuntu 18.04 and ONNX model by link in yaml config.

dkurt · 2023-11-16T10:50:51Z

@asmorkalov, thanks for testing!

Using cv2 dnn interface to run yolov8 model opencv#24396 This is a sample code for using opencv dnn interface to run ultralytics yolov8 model for object detection. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [X] I agree to contribute to the project under Apache 2 License. - [X] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [X] The PR is proposed to the proper branch - [] There is a reference to the original bug report and related work - [] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [] The feature is well documented and sample code can be built with the project CMake

asmorkalov requested a review from Abdurrahheem October 13, 2023 08:57

asmorkalov added category: samples category: dnn labels Oct 13, 2023

dkurt reviewed Oct 13, 2023

View reviewed changes

samples/dnn/dnn_model_runner/dnn_conversion/ultralytics/predict/dog1.jpg Outdated Show resolved Hide resolved

dkurt reviewed Oct 13, 2023

View reviewed changes

samples/dnn/dnn_model_runner/dnn_conversion/ultralytics/yolov8_cv.py Outdated Show resolved Hide resolved

Tsai-chia-hsiang requested a review from dkurt October 18, 2023 06:26

Tsai-chia-hsiang marked this pull request as draft October 18, 2023 06:27

Tsai-chia-hsiang marked this pull request as ready for review October 18, 2023 06:28

dkurt reviewed Oct 23, 2023

View reviewed changes