Update: For anyone who needs this new feature right now on Python and can't wait 3 months for the next OpenCV version, read this post on how to compile a custom Python .whl package:

#15739 (comment)

Ping @dkurt @liuyuns @dmonterom @vpisarev @arthur-williams @alalek @AlexeyAB (everyone who has ever worked on OpenCV's darknet module).

New Neural Network Innovation!

There's an incredible new neural network innovation: YOLOv3-Tiny has been reduced to almost half CPU usage while retaining detection accuracy. They call it YOLOv3-Tiny-PRN. It's the new king of low-CPU usage networks! Here is the research and paper in progress (soon to be published): https://github.com/WongKinYiu/PartialResidualNetworks

Here's the performance difference on Darknet (it would be even faster in OpenCV):

If you do a diff between yolov3-tiny-prn config and the original yolov3-tiny, you will see that the changes are that many layers have half the filter sizes, and there are many [shortcut] layers added instead.

You can also read their paper at the repo above.

The problem is that OpenCV's darknet loader lacks full support for [shortcut] layers. So nobody can enjoy this big PRN innovation on OpenCV's ultra-fast CPU neural network implementation...

References

Reference: Darknet's implementation of shortcut layers.

• https://github.com/pjreddie/darknet/blob/b13f67bfdd87434e141af532cdb5dc1b8369aa3b/src/parser.c#L539-L556
• https://github.com/pjreddie/darknet/blob/f6d861736038da22c9eb0739dca84003c5a5e275/src/shortcut_layer.c#L9-L39

Reference: OpenCV's incomplete implementation of shortcut layers.

• opencv/modules/dnn/src/darknet/darknet_io.cpp

  Lines 631 to 641 in 8c0b071

  	else if (layer_type == "shortcut")
  	{
  	std::string bottom_layer = getParam<std::string>(layer_params, "from", "");
  	CV_Assert(!bottom_layer.empty());
  	int from = std::atoi(bottom_layer.c_str());
  	from += layers_counter;
  	current_channels = net->out_channels_vec[from];
  	setParams.setShortcut(from);
  	}

• opencv/modules/dnn/src/darknet/darknet_io.cpp

  Lines 412 to 432 in 8c0b071

  	void setShortcut(int from)
  	{
  	cv::dnn::LayerParams shortcut_param;
  	shortcut_param.name = "Shortcut-name";
  	shortcut_param.type = "Eltwise";
  	shortcut_param.set<std::string>("op", "sum");
  	darknet::LayerParameter lp;
  	std::string layer_name = cv::format("shortcut_%d", layer_id);
  	lp.layer_name = layer_name;
  	lp.layer_type = shortcut_param.type;
  	lp.layerParams = shortcut_param;
  	lp.bottom_indexes.push_back(fused_layer_names.at(from));
  	lp.bottom_indexes.push_back(last_layer);
  	last_layer = layer_name;
  	net->layers.push_back(lp);
  	layer_id++;
  	fused_layer_names.push_back(last_layer);
  	}

• We can see one big issue immediately: OpenCV's reading of the shortcut layers only reads the from= property. It totally ignores all other properties (mainly the important activation= property).
• As a sidenote: The PRN network uses [shortcut] with activation=leaky (leaky RELU).
• Here's @dkurt 16 months ago, confirming that OpenCV's implementation of [shortcut] is wrong (basically it just implements the bare-minimum "Linear" mode for equally-sized layers which are used by the main Yolov3 config, but more advanced usages do not work): dnn.readNetFromDarknet() dosn't support "avgpool" layer #11568 (comment)

Reference: OpenCV's "Eltwise" layer type, with the "op:sum" parameter, that the YOLO "shortcut" layer is being converted into.

• https://github.com/opencv/opencv/blob/master/modules/dnn/src/layers/eltwise_layer.cpp
• This will most likely need improvement to add a flag which says "allow summing layers of different sizes"... and check how darknet does it; I assume that summing differently shaped arrays is done via interpolated scaling of the source layer's output to the target layer's array size.

Reference: Official pjreddie configs which use shortcut in various ways.

• https://github.com/pjreddie/darknet/blob/master/cfg/yolov3.cfg
• All of the ResNet configs at https://github.com/pjreddie/darknet/tree/master/cfg

Reference: Python script which completely crashes python due to OpenCV failing to read the shortcut layers.

• cfg: https://github.com/WongKinYiu/PartialResidualNetworks/blob/master/cfg/yolov3-tiny-prn.cfg
• weights: https://github.com/WongKinYiu/PartialResidualNetworks/blob/master/model/yolov3-tiny-prn.weights

import os
import cv2

cfg = "yolov3-tiny-prn.cfg"
weights = "yolov3-tiny-prn.weights"

# True, True
print(os.path.isfile(cfg))
print(os.path.isfile(weights))

# This kills python silently. The second "x" does not execute.
print("x") # prints
foo = cv2.dnn.readNetFromDarknet(cfg, weights) # kills python
print("x")

Summary

• Innovative new Neural Network massively reduces CPU usage and inference time, while keeping the same detection accuracy.
• OpenCV's darknet parser does not support options (such as activation=leaky) on the [shortcut] layers. The darknet/parser.c that I linked above shows that the official list of [shortcut] options is activation=, from=, alpha= and beta=.
• OpenCV does not support "shortcut"-linking between layers of different sizes, apparently, according to @dkurt.
• OpenCV dies without throwing any descriptive error when trying to load the network's config, as seen in the python file (which shows no error output; it just dies).
• If this can be fixed, OpenCV will be the world's fastest CPU-based framework for running the world's fastest YOLO object detector for low-power devices. Yay. I hope that day is possible in the future...
• Sidenote regarding that last point: With the current official YOLOv3-Tiny in OpenCV, my i7-8750H is taking ~30ms per forward pass (33.3 FPS). If the performance ratio of YOLOv3-Tiny-PRN's speed improvement is the same in OpenCV as it was in Darknet, the new PRN-based network would take ~18.5ms per forward pass (54 FPS) if OpenCV could support it. That's indeed amazing for a CPU detector!