Overview

• Proposal: Develop OpenCV.js DNN modules for promising web use cases together with their tutorials
• Mentor: Ningxin Hu @huningxin
• Student: Zhiming Zeng @akineeic
• Report
• Video report on Youtube

Introduction

OpenCV.js comes out as a JavaScript binding for a selected subset of OpenCV functions for the open web platform. It allows emerging web applications involving multimedia processing to benefit from the wide variety of vision functions available in OpenCV. OpenCV.js leverages Emscripten to compile OpenCV functions into ASM.js or WebAssembly targets, and provides JavaScript APIs for web developers.

The dnn module of OpenCV supports several frameworks including caffe, tensorflow, torch, darknet, onnx. It also supports so many types of layers as shown in the wiki, which reveal its potential to develop deep learning applications. OpenCV.js also binds some dnn API which is enough to do deep learning inference on the Web. At the same time, with the optimization through multi-threads and SIMD, the performance of the dnn module in OpenCV.js is impressive. All of these means that OpenCV.js also has the ability and potential to develop attractive deep learning applications on the Web.

But there are lack of tutorials to show the ability or potential of the dnn module of OpenCV.js. Very little information a developer can get from the tutorial even if he is interested in the dnn module of OpenCV.js. Lack of documentation and tutorials becomes a major issue that prevents widespread use of the dnn module in OpenCV.js.

Therefore, the main goal of this project is to develop several deep learning example with OpenCV.js and add them into the tutorial.

Work structure

Develop DNN(Deep Neural Networks) examples

This project added several DNN examples into OpenCV.js tutorial, including image classification, object detection, style transfer, semantic segmentation and pose estimation. The process flow of each example is similar. So condensed a function main to do inference once, which could be divided into several parts as follows:

1. Load labels from txt file and process it into an array.
2. Fetch model file and save into emscripten file system.
3. Get blob from image as input for net, and standardize it with parameter like mean, std according to model.
4. Use binding functions setInput and forward to get the output data.
5. Post-processing to get the final result from the output data.

The first three part have been written into a JS file as help functions since these parts can be re-use between examples. You should only re-write the post-processing function to develop different example and change the parameters to run differents models.

Usage of DNN example

Follow the tutorial to build OpenCV.js and append option --build_doc to build documents. To try the DNN examples, launch a local web server in <build_dir>/doc/doxygen/html folder. Then click OpenCV.js Tutorials and Deep Neural Networks (dnn module) example in turn will navigate into the example webpage. You can download models according to the table in the bottom of the webpage or use their own models. Click the modelFile button (and configFile button if neeed) to upload model and modify the parameters in the first codeSnippet according to the uploaded model. Finally click Try it button will run function main to do inference once and show the result in the webpage.

Result

Test Environment

OS: Ubuntu 18.04

Emscripten: 1.39.15, LLVM upstream backend

Browser: Google Chrome 85.0.4183.83 (Official Build) (64-bit)

Hardware: Core(TM) i7-5960X CPU @ 3.00GHz with 16 logical cores

Test Result

Examples show(test in scalar)

• Image Classification

• Object Detection

• Style Transfer

• Semantic Segmentation

• Pose Estimation

Future Work

1. Investigate more models and add their infermation into the tutorials for each example.
2. Try to enable DetectionModel for object detection example with the new version of OpenCV.

force_builders_only=docs,Custom
buildworker:Custom=linux-1,linux-2,linux-4
build_image:Docs=docs-js
build_image:Custom=javascript