[GSoC] OpenCV.js: WASM SIMD optimization 2.0 by lionkunonly · Pull Request #18068 · opencv/opencv

lionkunonly · 2020-08-11T04:48:38Z

Overview

Proposal: OpenCV.js: WASM SIMD optimization 2.0
Mentor: Vitaly Tuzov @terfendail, Ningxin Hu @huningxin,
Student: Kun Liang @lionkunonly

This pull request changes

Update the version of supported emscripten to 1.39.16 and modify the js files in test and perf test to fit it. Add required emscripten version into opencv.js build instructions.
Implement the type 64 intrinsic and add perf test for some kernels.
Optimize the perf test. Remove the redundant code in the perf test and move the functions that play similar roles into the file perf_helpfunc.js. Reconstruct current perf tests.
Add more perf tests for kernels. Supported perf tests: cvtColor, resize, threshold, Sobel, filter2D, Scharr, gaussianBlur, blur, medianBlur, erode, dilate, remap, warpAffine, warpPerspective, pyrDown.
Implement a loader that can detect the features of the browser and load the corresponding version of OpenCV.js automatically. It utilizes the help of WebAssembly Feature Detection.

The Test

Test Environment:

OS: Ubuntu Linux 18.04.4
Emscripten: 1.39.16, LLVM upstream backend
Browser: Chrome, Version 85.0.4183.26 dev (64-bit)
Hardware: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 12 logical cores

OpenCV.js tests: all passed
Universal Intrinsics WASM backend test: all passed
All perf tests: all run successfully

Results

Perf test for 64bit intrinsics

OS: Ubuntu Linux 18.04.4
Emscripten: 1.39.16, LLVM upstream backend
Browser: Chrome, Version 85.0.4183.26 dev (64-bit)
Hardware: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 12 logical cores

Function	Before 64bit implementation (ms)	After 64bit implementation (ms)	Speedup
countNonZero()	0.5386	0.5221	1.032x
Mat::dot()	0.8079	0.7960	1.015x
split()	2.1213	2.0483	1.036x
merge()	2.2383	2.2264	1.005x

The performance of kernels after 64-bit implementation is similar to the performance before implementation.

Performance of resize kernel with widen instructions

OS: Ubuntu Linux 18.04.4
Emscripten: 1.39.16, LLVM upstream backend
Browser: Chrome, Version 85.0.4183.26 dev (64-bit)
Hardware: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 12 logical cores

Parameters	Mean time of Scalar	Mean time of SIMD with `shift`(ms)	Speedup	Mean time of SIMD with `widen`(ms)	Speedup
(CV_8UC4, 1280x720, 640x480)	5.113	2.888	1.77x	4.031	1.27x
(CV_8UC1, 1280x720, 640x480)	1.870	1.128	1.66x	1.249	1.50x

The perform of the implementation with widen instructions does not bring improvement. One of the reasons is that the wasm_v8x16_shuffle and widen instructions have to be used together. The original implementation is left finally.

Perf tests

Supported Kernels now (lines)	Supported Kernels before (lines)
cvtColor (421)	cvtColor (572)
resize (165)	resize (262)
threshold (158)	threshold (217)
Sobel (170)	-
filter2D (127)	-
Scharr (156)	-
gaussianBlur (126)	-
blur (130)	-
medianBlur (118)	-
erode (117)	-
dilate (117)	-
remap (182)	-
warpAffine (130)	-
warpPerspective (143)	-
pyrDown (116)	-

The collected performance data is stored in the Google Drive. Performance data

Final Report
Perf tests and loader demo(May need the latest version of Chrome-Dev)
Video

Performance Analysis

Because there are more than ten kernels are tested and some of them are tested with parameters with different data type and channels, the table that records the performance data is large. So I put the performance data in my personal Google drive.

Performance data

Analysis
Based on the collected performance data, the SIMD optimization works as we expected in most situations. It can achieve similar performance as the SSE2 optimization for the native OpenCV. For example, for the kernel blur with the parameter (1280x720, CV_8UC1, BORDER_REPLICATE) ksize=3, the SIMD version has 1.357x speed up and the SSE2 has 1.415x speed up. Sometimes the SIMD optimization is better. For example, for the kernel pyrDown with the parameter (1920x1080, CV_32FC4), the SIMD version has 3.094x speed up and the SSE2 has 1.83 speed up. The data that achieves similar or better speed up is tagged with the color green.

However, there still exist some bad cases, for example, for the kernel blur with (1280x720, CV_32FC1, BORDER_REPLICATE) ksize=3, the SIMD version has 0.519 speed up which is 2x slower than the SSE optimization for native OpenCV. Such data is tagged with the color yellow.

The data tagged with color red means that it is unnormal. In my point of view, some yellow data is brought by the red unnormal data like the kernel medianBlur with the parameter (1280x720, CV_16SC1, 5).

I hope the collected data can help people know the gap between the SIMD optimization in the OpenCV.js and the optimization in the native OpenCV better.

force_builders_only=linux,docs,Custom
buildworker:Custom=linux-4
build_image:Docs=docs-js
build_image:Custom=javascript-simd:1.39.16

fix the trailing whitespace.

…n in perf test of warpAffine and warpPrespective

lionkunonly · 2020-08-25T09:22:25Z

Hi, I have implemented the OpenCV.js loader and modified the js_setup.markdown to explain the usage of the loader. But I am not sure my expression is clear enough. So, Cloud you give some suggestions?

huningxin · 2020-08-26T06:02:50Z

modules/js/src/loader.js

+    let threadsPath = "";
+    let mtSIMDPath = "";
+
+


Please remove the redundant blank line (just need one).

huningxin · 2020-08-26T06:03:41Z

modules/js/src/loader.js

+
+
+    this.judgeWASM = function() {
+        try{


Please insert a space between try and {.

actually, you can check WebAssembly by

return !(typeof WebAssembly === 'undefined')

huningxin · 2020-08-26T06:05:02Z

modules/js/src/loader.js

+    let mtSIMDPath = "";
+
+
+    this.judgeWASM = function() {


judge sounds not a good name. how about checkWasm?

huningxin · 2020-08-26T06:07:08Z

modules/js/src/loader.js

+        }
+    }
+
+    this.judgeSIMD = function() {


huningxin · 2020-08-26T06:07:24Z

modules/js/src/loader.js

+    }
+
+    this.judgeSIMD = function() {
+        return WebAssembly.validate(new Uint8Array([0,97,115,109,1,0,0,0,1,4,1,96,0,0,3,2,1,0,10,9,1,7,0,65,0,253,15,26,11]));


could you please document what's the content of the Unit8Array?

and you need to call checkWasm first.

By reading the docs of wasm-feature-detect, I think my implementation for detecting threads feature and simd feature is good enough. Because the browser will update in the future, which may block my implementation. So, I decide to utilize the wasm-feature-detect to solve my problem. Do you agree it ?

It sounds good to me.

huningxin · 2020-08-26T06:10:27Z

modules/js/src/loader.js

+        try {
+            let test1 = (new MessageChannel).port1.postMessage(new SharedArrayBuffer(1));
+            let result = WebAssembly.validate(new Uint8Array([0,97,115,109,1,0,0,0,1,4,1,96,0,0,3,2,1,0,5,4,1,3,1,1,10,11,1,9,0,65,0,254,16,2,0,26,11]));
+            return result;


return true?

huningxin · 2020-08-26T06:10:34Z

modules/js/src/loader.js

+            let result = WebAssembly.validate(new Uint8Array([0,97,115,109,1,0,0,0,1,4,1,96,0,0,3,2,1,0,5,4,1,3,1,1,10,11,1,9,0,65,0,254,16,2,0,26,11]));
+            return result;
+        } catch(e) {
+            return !1


return false?

huningxin · 2020-08-26T06:11:31Z

modules/js/src/loader.js

+        }
+    }
+
+    this.setPaths = function(paths) {


Please remove this method and merge its implementation into the constructor.

huningxin · 2020-08-26T06:13:44Z

modules/js/src/loader.js

+        }
+    }
+
+    this.loadOpenCV = function (onloadCallback) {


I think you can encapsulate all functions into this method. Do you really need a loader object?

function loadOpenCV(paths, onloadCallback) {...}

It would have simpler usage.

The loader object is not necessary. I think I can try to encapsulate all functions in loadOpenCV.

huningxin · 2020-08-26T06:16:57Z

modules/js/src/loader.js

+        if (simdSupported && threadsSupported && OPENCV_URL == "" && self.mtSIMDPath != "") {
+            OPENCV_URL = self.mtSIMDPath;
+        } else if (simdSupported && threadsSupported && OPENCV_URL == "") {
+            throw new Error("The browser supports simd and threads, but the path of OpenCV.js with simd and threads optimization is empty");


I think in this case, if developer only provides the threadsPath or simdPath, it should also work.

terfendail · 2020-08-26T17:20:31Z

platforms/js/build_js.py

    log.info("=====")
    builder.build_opencvjs()
-
+    


Please remove trailing whitespaces

…ve trailing whitespaces

lionkunonly · 2020-08-27T05:26:28Z

@huningxin @terfendail Hi, I have modified the implementation based on your comments in code review. However, the most important change in this implementation is that I use the WebAssembly Feature Detection in my implementation. The reason for using it is that I find the original implementation may fail when the browser update in the future. With the help of the WebAssembly Feature Detection would be available even if the browser update. Because this library would update at that time.

huningxin

@lionkunonly , could you please share the tests and the results you did for your loader?

huningxin · 2020-08-28T00:36:55Z

modules/js/src/loader.js

+        OPENCV_URL = simdPath;
+        console.log("The OpenCV.js with simd optimization is loaded now.");
+    } else if (threadsSupported && threadsPath != "") {
+        if (simdSupported && threadsSupported) {


if (simdSupported && threadsSimdPath === "")?

huningxin · 2020-08-28T00:39:43Z

modules/js/src/loader.js

+        console.log("The browser supports wasm, but the path of OpenCV.js for wasm is empty");
+    }
+
+    if(OPENCV_URL == "") {


Put a space between if and (

huningxin · 2020-08-28T00:41:03Z

modules/js/src/loader.js

-            throw new Error("The browser supports simd, but the path of OpenCV.js with simd optimization is empty");
+        OPENCV_URL = threadsPath;
+        console.log("The OpenCV.js with threads optimization is loaded now");
+    } else if (threadsPath) {


else if (threadsSupported)?

Sorry, it is my fault. This choice should be deleted.

huningxin · 2020-08-28T00:41:35Z

doc/js_tutorials/js_setup/js_setup/js_setup.markdown

-    loader = new Loader();
-
    //Set paths configuration
    pathsConfig = {


let pathsConfig?

huningxin · 2020-08-28T00:42:07Z

doc/js_tutorials/js_setup/js_setup/js_setup.markdown

-    // Create an instance 
-    loader = new Loader();
-
    //Set paths configuration


Add a space after //

huningxin · 2020-08-28T00:42:12Z

doc/js_tutorials/js_setup/js_setup/js_setup.markdown


-    //Load OpenCV.js and use main function as the param
-    loader.loadOpenCV(main);
+    //Load OpenCV.js and use the pathsConfiguration and main function as the params.


lionkunonly · 2020-08-28T02:43:05Z

@lionkunonly , could you please share the tests and the results you did for your loader?

The Scene1: SIMD and threads are enabled.

The path configuration of the example

The log for loading status and html elements

In this scene, the application load the opencv.js with simd and threads optimization successfully.

The Scene2: SIMD and threads are enabled, but the path to threads+simd version is invalid.

The path configuration in js file

Results

In this scene, the path for opencv.js with threads and simd optimization is empty. So the loader loads the opencv.js with simd optimization automatically.

The Scene3: Threads is enabled, simd is diabled.

The feature configuration in browser

Results

In this scene, the loader loads the opencv.js with threads optimization automatically without any failed log like the log in the scene2.

Finally, I am trying to design a demo to show the effect of the loader and deploy the demo on the github.io.

…loader.js

lionkunonly · 2020-08-29T08:53:14Z

@lionkunonly , could you please share the tests and the results you did for your loader?

You can try the example with the Demo -- Perf tests and loader demo

@huningxin @terfendail I think I have completed all the tasks in GSoC2020 now. Should I remove the [WIP] in this PR title?

lionkunonly · 2020-08-30T11:35:12Z

@terfendail @terfendail The final report is Here. You can leave your comment in the Google drive version .

huningxin · 2020-08-31T04:44:30Z

You can try the example with the Demo -- Perf tests and loader demo

The loader demo works great regarding to my test, even when disabling WASM in Chrome. Thanks for that!

My another suggestion would be using gaussianBlur kernel that has good threads speedup (4.98X) and simd speedup (3.36X). So user can see difference easily.

The final report is Here. You can leave your comment in the Google drive version .

Please add the summary and analysis of your performance data into the final report. Both of them are valuable. Thanks!

huningxin · 2020-08-31T04:45:15Z

Should I remove the [WIP] in this PR title?

It seems all tasks are done. Please remove that so the OpenCV maintainers can review.

lionkunonly · 2020-09-04T12:14:54Z

@terfendail @huningxin Hi, Vitaly and Ningxin. I have updated the video report please check it here

lionkunonly and others added 2 commits August 11, 2020 12:22

gsoc_2020_simd Add perf test for filter2d

6bbb2aa

add perf test for kernel scharr and kernel gaussianBlur

2f02dfc

alalek mentioned this pull request Aug 14, 2020

OpenCV.js --simd build fails with recent Emscripten #18097

Closed

add perf test for blur, medianBlur, erode, dilate

faf95a5

huningxin mentioned this pull request Aug 17, 2020

modify the perf test for kernel sobel huningxin/opencv#361

Merged

fix the errors for the opencv PR robot

c3d8bd3

fix the trailing whitespace.

lionkunonly force-pushed the gsoc_2020_simd branch from a7ad26e to c3d8bd3 Compare August 18, 2020 08:44

lionkun added 5 commits August 20, 2020 17:09

add perf tests for kernel remap, warpAffine, warpPersepective, pyrDown

e7080be

fix a bug in modules/js/perf/perf_imgproc/perf_remap.js

fb4291c

add function smoothBorder in helpfun.js and remove replicated functio…

961336e

…n in perf test of warpAffine and warpPrespective

fix the trailing white space issues

15f96c6

add OpenCV.js loader

6d06647

huningxin reviewed Aug 26, 2020

View reviewed changes

terfendail reviewed Aug 26, 2020

View reviewed changes

platforms/js/build_js.py Outdated

log.info("=====")

builder.build_opencvjs()

Copy link
Copy Markdown

Contributor

terfendail Aug 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove trailing whitespaces

Implement the Loader with help of WebAssembly Feature Detection, remo…

f839367

…ve trailing whitespaces

huningxin reviewed Aug 28, 2020

View reviewed changes

modify the explantion for loader in js_setup.markdown and fix bug in …

b201371

…loader.js

lionkunonly changed the title ~~WIP: [GSoC] OpenCV.js: WASM SIMD optimization 2.0.~~ [GSoC] OpenCV.js: WASM SIMD optimization 2.0 Aug 29, 2020

terfendail approved these changes Oct 16, 2020

View reviewed changes

alalek assigned terfendail Oct 18, 2020

alalek merged commit c824176 into opencv:3.4 Oct 18, 2020

alalek mentioned this pull request Oct 21, 2020

Merge 3.4 #18639

Merged

alalek mentioned this pull request Nov 27, 2020

Merge 4.x #18951

Merged

alalek mentioned this pull request Mar 20, 2021

Wasm perf tests RuntimeError: abort(TypeError: cv.Size is not a constructor). #19754

Closed

Uh oh!

Conversation

lionkunonly commented Aug 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

This pull request changes

The Test

Results

Performance Analysis

Uh oh!

lionkunonly commented Aug 25, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lionkunonly Aug 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lionkunonly commented Aug 27, 2020

Uh oh!

huningxin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lionkunonly commented Aug 28, 2020

The Scene1: SIMD and threads are enabled.

The path configuration of the example

The log for loading status and html elements

The Scene2: SIMD and threads are enabled, but the path to threads+simd version is invalid.

The path configuration in js file

Results

The Scene3: Threads is enabled, simd is diabled.

The feature configuration in browser

Results

Uh oh!

lionkunonly commented Aug 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

lionkunonly commented Aug 11, 2020 •

edited

Loading

lionkunonly Aug 27, 2020 •

edited

Loading

lionkunonly commented Aug 29, 2020 •

edited

Loading

lionkunonly commented Sep 4, 2020 •

edited

Loading