Skip to content

Wechat qrcode perfomance tests#3483

Merged
asmorkalov merged 3 commits intoopencv:4.xfrom
SinM9:wechat_qrcode_perf_tests
May 4, 2023
Merged

Wechat qrcode perfomance tests#3483
asmorkalov merged 3 commits intoopencv:4.xfrom
SinM9:wechat_qrcode_perf_tests

Conversation

@SinM9
Copy link
Copy Markdown

@SinM9 SinM9 commented Apr 26, 2023

Added perfomance tests for detection and decoding with and without nn.
Download model files from https://github.com/WeChatCV/opencv_3rdparty/tree/wechat_qrcode.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

Comment on lines +95 to +98
for(size_t i = 0; i < decoded_info.size(); i++)
{
ASSERT_FALSE(decoded_info[i].empty());
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Result check should be out of perf test loop.

Comment on lines +121 to +123
{
ASSERT_FALSE(decoded_info[i].empty());
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Result check should be out of perf test loop.

// of this distribution and at http://opencv.org/license.html.
#include "perf_precomp.hpp"

CV_PERF_TEST_MAIN(wechat_qrcode)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to copy initTests() code from objdetect/test to properly find DNN models.

std::string qrcode_images_multiple[] = {"2_qrcodes.png", "3_qrcodes.png", "3_close_qrcodes.png",
"4_qrcodes.png", "5_qrcodes.png", "7_qrcodes.png"};

bool Find_Models_Files(std::vector<std::string>& models) {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

function names should start with lowercase.

path_sr_prototxt = findDataFile("dnn/wechat"+model_version+"/sr.prototxt", false);
path_sr_caffemodel = findDataFile("dnn/wechat"+model_version+"/sr.caffemodel", false);
models = {path_detect_prototxt, path_detect_caffemodel, path_sr_prototxt, path_sr_caffemodel};
return true;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why always true? do we need bool result?

Comment on lines +185 to +188
ASSERT_TRUE(Find_Models_Files(models));
auto detector = wechat_qrcode::WeChatQRCode(models[0], models[1], models[2], models[3]);

TEST_CYCLE()
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With DNN models we should have "warmup" first call before measuring.

Comment on lines +113 to +114
ASSERT_TRUE(Find_Models_Files(models));
auto detector = wechat_qrcode::WeChatQRCode(models[0], models[1], models[2], models[3]);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

models[0], models[1], models[2], models[3]

createQRDetectorWithDNN()

RNG rng;
rng.fill(not_qr_code, RNG::UNIFORM, Scalar(0), Scalar(1));
}
if (type_gen == "chessboard")
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

else if

int i = r / not_qr_code.cols;
int j = r % not_qr_code.cols;
not_qr_code.ptr<uchar>(i)[j] = next_pixel;
next_pixel = 255 - next_pixel;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with even cols that code draws vertical lines.

Comment on lines +167 to +182
if (type_gen == "random")
{
RNG rng;
rng.fill(not_qr_code, RNG::UNIFORM, Scalar(0), Scalar(1));
}
if (type_gen == "chessboard")
{
uint8_t next_pixel = 0;
for (int r = 0; r < not_qr_code.rows * not_qr_code.cols; r++)
{
int i = r / not_qr_code.cols;
int j = r % not_qr_code.cols;
not_qr_code.ptr<uchar>(i)[j] = next_pixel;
next_pixel = 255 - next_pixel;
}
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code duplication

@opencv-alalek
Copy link
Copy Markdown

@MaximMilashchenko please review this PR.

@MaximMilashchenko
Copy link
Copy Markdown
Contributor

@MaximMilashchenko please review this PR.

reviewed, no comments from my side

@opencv-alalek opencv-alalek requested a review from mshabunin May 3, 2023 15:07
@mshabunin
Copy link
Copy Markdown
Contributor

Overall looks good to me 👍
Additionally I've run these tests on Core i5-11600 (Ubuntu) and Rockchip RK3588 (Armbian) and compared results:

Details
Name of Test x86_64 rk3588 rk3588 vs x86 64 (x-factor)
detect_and_decode::Perf_Objdetect_Not_QRCode::("", "chessboard", 640x480) 18.584 50.045 0.37
detect_and_decode::Perf_Objdetect_Not_QRCode::("", "chessboard", 1280x720) 65.713 182.732 0.36
detect_and_decode::Perf_Objdetect_Not_QRCode::("", "random", 640x480) 7.326 21.505 0.34
detect_and_decode::Perf_Objdetect_Not_QRCode::("", "random", 1280x720) 32.225 90.927 0.35
detect_and_decode::Perf_Objdetect_Not_QRCode::("", "zero", 640x480) 7.323 21.536 0.34
detect_and_decode::Perf_Objdetect_Not_QRCode::("", "zero", 1280x720) 32.210 94.080 0.34
detect_and_decode::Perf_Objdetect_Not_QRCode::("dnn/wechat_2021-01", "chessboard", 640x480) 3.277 19.265 0.17
detect_and_decode::Perf_Objdetect_Not_QRCode::("dnn/wechat_2021-01", "chessboard", 1280x720) 95.754 295.366 0.32
detect_and_decode::Perf_Objdetect_Not_QRCode::("dnn/wechat_2021-01", "random", 640x480) 3.277 19.209 0.17
detect_and_decode::Perf_Objdetect_Not_QRCode::("dnn/wechat_2021-01", "random", 1280x720) 3.377 20.193 0.17
detect_and_decode::Perf_Objdetect_Not_QRCode::("dnn/wechat_2021-01", "zero", 640x480) 3.268 19.283 0.17
detect_and_decode::Perf_Objdetect_Not_QRCode::("dnn/wechat_2021-01", "zero", 1280x720) 3.370 20.406 0.17
detect_and_decode::Perf_Objdetect_QRCode::("", "kanji.jpg") 1.695 3.844 0.44
detect_and_decode::Perf_Objdetect_QRCode::("", "link_wiki_cv.jpg") 19.699 45.967 0.43
detect_and_decode::Perf_Objdetect_QRCode::("", "russian.jpg") 3.732 7.320 0.51
detect_and_decode::Perf_Objdetect_QRCode::("", "version_1_top.jpg") 0.858 2.186 0.39
detect_and_decode::Perf_Objdetect_QRCode::("", "version_2_left.jpg") 2.454 6.550 0.37
detect_and_decode::Perf_Objdetect_QRCode::("", "version_2_top.jpg") 0.853 2.199 0.39
detect_and_decode::Perf_Objdetect_QRCode::("", "version_2_up.jpg") 0.861 2.229 0.39
detect_and_decode::Perf_Objdetect_QRCode::("", "version_3_down.jpg") 0.858 2.184 0.39
detect_and_decode::Perf_Objdetect_QRCode::("", "version_3_top.jpg") 0.876 2.219 0.39
detect_and_decode::Perf_Objdetect_QRCode::("", "version_4_top.jpg") 0.995 2.483 0.40
detect_and_decode::Perf_Objdetect_QRCode::("", "version_5_down.jpg") 1.149 2.775 0.41
detect_and_decode::Perf_Objdetect_QRCode::("", "version_5_left.jpg") 0.932 2.330 0.40
detect_and_decode::Perf_Objdetect_QRCode::("", "version_5_top.jpg") 3.045 6.297 0.48
detect_and_decode::Perf_Objdetect_QRCode::("", "version_5_up.jpg") 8.060 18.971 0.42
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "kanji.jpg") 4.642 23.082 0.20
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "link_wiki_cv.jpg") 5.153 24.557 0.21
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "russian.jpg") 4.989 24.859 0.20
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "version_1_top.jpg") 4.420 23.165 0.19
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "version_2_left.jpg") 4.238 22.127 0.19
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "version_2_top.jpg") 4.273 22.234 0.19
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "version_2_up.jpg") 5.828 25.864 0.23
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "version_3_down.jpg") 4.346 22.569 0.19
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "version_3_top.jpg") 4.418 22.489 0.20
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "version_4_top.jpg") 4.422 22.728 0.19
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "version_5_down.jpg") 4.461 22.098 0.20
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "version_5_left.jpg") 5.004 23.651 0.21
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "version_5_top.jpg") 4.560 23.045 0.20
detect_and_decode::Perf_Objdetect_QRCode::("dnn/wechat_2021-01", "version_5_up.jpg") 24.120 78.197 0.31
detect_and_decode::Perf_Objdetect_QRCode_Multi::("", "2_qrcodes.png") 2.014 5.821 0.35
detect_and_decode::Perf_Objdetect_QRCode_Multi::("", "3_close_qrcodes.png") 1.307 2.977 0.44
detect_and_decode::Perf_Objdetect_QRCode_Multi::("", "3_qrcodes.png") 1.757 4.508 0.39
detect_and_decode::Perf_Objdetect_QRCode_Multi::("", "4_qrcodes.png") 1.752 3.643 0.48
detect_and_decode::Perf_Objdetect_QRCode_Multi::("", "5_qrcodes.png") 5.163 10.627 0.49
detect_and_decode::Perf_Objdetect_QRCode_Multi::("", "7_qrcodes.png") 20.820 48.158 0.43
detect_and_decode::Perf_Objdetect_QRCode_Multi::("dnn/wechat_2021-01", "2_qrcodes.png") 59.649 183.353 0.33
detect_and_decode::Perf_Objdetect_QRCode_Multi::("dnn/wechat_2021-01", "3_close_qrcodes.png") 13.964 61.721 0.23
detect_and_decode::Perf_Objdetect_QRCode_Multi::("dnn/wechat_2021-01", "3_qrcodes.png") 34.269 110.550 0.31
detect_and_decode::Perf_Objdetect_QRCode_Multi::("dnn/wechat_2021-01", "4_qrcodes.png") 25.056 103.633 0.24
detect_and_decode::Perf_Objdetect_QRCode_Multi::("dnn/wechat_2021-01", "5_qrcodes.png") 4.914 22.592 0.22
detect_and_decode::Perf_Objdetect_QRCode_Multi::("dnn/wechat_2021-01", "7_qrcodes.png") 6.520 25.852 0.25

@asmorkalov asmorkalov merged commit 3a2ed1c into opencv:4.x May 4, 2023
@asmorkalov asmorkalov mentioned this pull request May 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants