[GAPI] Support basic inference in OAK backend by smirnov-alexey · Pull Request #21504 · opencv/opencv

smirnov-alexey · 2022-01-24T08:17:53Z

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

force_builders=Custom,Custom Win,Custom Mac
build_gapi_standalone:Linux x64=ade-0.1.1f
build_gapi_standalone:Win64=ade-0.1.1f
build_gapi_standalone:Mac=ade-0.1.1f
build_gapi_standalone:Linux x64 Debug=ade-0.1.1f

Xbuild_image:Custom=centos:7
Xbuildworker:Custom=linux-1
build_gapi_standalone:Custom=ade-0.1.1f

build_image:Custom=ubuntu-openvino-2021.4.2:20.04
build_image:Custom Win=openvino-2021.4.2
build_image:Custom Mac=openvino-2021.4.2

test_modules:Custom=gapi,python2,python3,java
test_modules:Custom Win=gapi,python2,python3,java
test_modules:Custom Mac=gapi,python2,python3,java

buildworker:Custom=linux-1
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*

This PR brings several new features in OAK backend:

Camera config
- Introduced rather basic config for the user to provide for ColorCamera. Note, that it still doesn't support all of the DAI parameters, e.g. resolutions
Inference support
- Added support for infer() operation via a kernel in OAK backend. Currently it only accepts GFrame as input - not sure if other types are applicable since the stream comes from the camera. Meta information is extracted from the DAI library. Preprocessing is currently tied to camera - this will be refactored separately and done as an additional node before the infer kernel (hidden from the user).
Additional heterogeneity support for the backend
- Possible via Copy operation in the backend. It allows not to split OAK island in two when we want an original stream from the camera or any other node in the island. Copy can be graph's output, passthrough operation (connects two nodes in the island) or both - the behavior in those cases is different (described and implemented in the backend).
2 samples
- Basic sample where we get stream from camera (wasn't possible before) via Copy
- Infer sample with rendering

smirnov-alexey · 2022-01-24T08:19:28Z

@TolyaTalamanov @dmatveev @rgarnov fyi. This is just a basic proposal, so there are several major problems with the current design.

smirnov-alexey · 2022-02-10T14:09:22Z

Note: nn metadata can be extracted from DAI once the feature is enabled there: luxonis/depthai-core#368

smirnov-alexey · 2022-03-10T15:02:46Z

@TolyaTalamanov @rgarnov please, review

smirnov-alexey · 2022-03-10T17:53:54Z

Note:

There are some accuracy issues with the infer (could be model, could be our side - I will double check as a separate task/PR)
Infer API is hard-coded now (GFrame as a single input, GMat as a single output). This will also be considered a separate task/PR (probably along with other infers (ROI, List, etc))
Copy and infer are rather bad-designed currently but hopefully it will be resolved during code review

rgarnov

Just a few questions for now, will add more on goakbackend.cpp later

modules/gapi/include/opencv2/gapi/oak/oak.hpp

modules/gapi/samples/oak_basic_infer.cpp

modules/gapi/include/opencv2/gapi/oak/oak.hpp

modules/gapi/src/backends/oak/oak_memory_adapters.hpp

modules/gapi/src/backends/oak/goak_memory_adapters.cpp

modules/gapi/src/backends/oak/goakbackend.cpp

modules/gapi/include/opencv2/gapi/oak/oak.hpp

modules/gapi/src/backends/oak/goakbackend.cpp

smirnov-alexey · 2022-03-28T11:40:59Z

@rgarnov please, take a look

modules/gapi/src/backends/oak/goakbackend.cpp

smirnov-alexey · 2022-03-29T16:03:14Z

@alalek can we merge this, please?

dmatveev

Looks ok, but please bear in mind I didn't dive deep into the backend code.

Just a simple question

Is there any tests we could run with OAK connected?

What happens on this:

cv::GFrame in;
cv::GFrame copy = cv::gapi::oak::copy(cv::gapi::oak::copy(cv::gapi::oak::copy(in)));
cv::GFrame render = cv::gapi::renderFrame(copy, ...);
cv::GMat out = cv::gapi::streaming::BGR(cv::gapi::oak::copy(render));

As a user, I am able to write code like this and from the general G-API concept it is perfectly legal. What will happen here at graph compile time? At graph execution time?

dmatveev · 2022-04-01T09:03:07Z

modules/gapi/include/opencv2/gapi/oak/infer.hpp

+
+    @param model Path to model (.blob file)
+    */
+    Params(const std::string &model) {


should be explicit

dmatveev · 2022-04-01T09:03:48Z

modules/gapi/include/opencv2/gapi/oak/infer.hpp

+* which is specific to OAK models.
+*/
+struct ParamDesc {
+    std::string blob_file;


No input/output layer names here? How is this supposed to work? So far you test with 1-in 1-out models only?

Currently only 1-in 1-out models are supported. Will extend once supported. Although, not sure if this info will be needed from the user since we extract all meta information from the model directly and utilize in the backend

dmatveev · 2022-04-01T09:05:58Z

modules/gapi/include/opencv2/gapi/oak/oak.hpp

+G_API_OP(GCopy, <GFrame(GFrame)>, "org.opencv.oak.copy") {
+    static GFrameDesc outMeta(const GFrameDesc& in) {
+        return in;
+    }
+};


Is a new operation really required?
https://docs.opencv.org/4.5.5/d6/d91/group__gapi__transform.html#ga1dc3b1987d6685689ecc0d590abddba8

Right, this can be refactored. But for now I'd keep Copy as a separate operation in the backend

dmatveev · 2022-04-01T09:07:32Z

modules/gapi/samples/oak_basic_infer.cpp

+    "{ detector            |             | Path to compiled .blob face detector model }"
+    "{ duration            | 1           | Number of frames to pull from camera and run inference on }";
+
+#ifdef HAVE_OAK


Not sure if this #ifdef at a right place, considering you including oak.hpp explicitly above.

If code compiles with NO OAK but WITH those headers included, do we need it here at all?

dmatveev · 2022-04-01T09:09:02Z

modules/gapi/samples/oak_basic_infer.cpp

+
+    // Initialize graph structure
+    cv::GFrame in;
+    cv::GFrame copy = cv::gapi::oak::copy(in); // NV12 transfered to host + passthrough copy for infer


I believe this copy should be isp if we want to trick that it is camera

Didn't get the idea, could you explain in more details, please?

dmatveev · 2022-04-01T09:10:05Z

modules/gapi/samples/oak_basic_infer.cpp

+int main() {
+    GAPI_Assert(false && "Built without OAK support");
+    return -1;
+}


If the app code above can compile with no OAK present, I believe we don't need this either

smirnov-alexey · 2022-04-04T09:38:10Z

Will address open comments in a separate PR. @alalek can we merge this one, please?

…ities of OAK backend

smirnov-alexey · 2022-04-05T15:04:58Z

@dmatveev answering your questions above:

There is a simple test in opencv_test_gapi under HAVE_OAK guard which checks that the backend compiles
There is a check with a meaningful error message that there are no back-to-back copy operations in the backend. Also it checks that there is only 1 OAK island in graph

[GAPI] Support basic inference in OAK backend * Combined commit which enables basic inference and other extra capabilities of OAK backend * Remove unnecessary target options from the cmakelist

smirnov-alexey changed the title ~~[WIP] [GAPI] Support basic inference in OAK backend~~ [GAPI] Support basic inference in OAK backend Mar 10, 2022

rgarnov reviewed Mar 17, 2022

View reviewed changes

rgarnov requested changes Mar 21, 2022

View reviewed changes

rgarnov requested changes Mar 22, 2022

View reviewed changes

rgarnov approved these changes Mar 29, 2022

View reviewed changes

modules/gapi/src/backends/oak/goakbackend.cpp Outdated Show resolved Hide resolved

dmatveev self-assigned this Apr 1, 2022

dmatveev added the category: g-api / gapi label Apr 1, 2022

dmatveev added this to the 4.6.0 milestone Apr 1, 2022

dmatveev approved these changes Apr 1, 2022

View reviewed changes

Combined commit which enables basic inference and other extra capabil…

f714de8

…ities of OAK backend

smirnov-alexey force-pushed the as/oak_infer branch from e1fa522 to f714de8 Compare April 5, 2022 14:59

Remove unnecessary target options from the cmakelist

5c91090

alalek merged commit 7ed5574 into opencv:4.x Apr 5, 2022

opencv-pushbot mentioned this pull request Apr 23, 2022

(5.x) Merge 4.x #21903

Merged

Uh oh!

Conversation

smirnov-alexey commented Jan 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

smirnov-alexey commented Jan 24, 2022

Uh oh!

smirnov-alexey commented Feb 10, 2022

Uh oh!

smirnov-alexey commented Mar 10, 2022

Uh oh!

smirnov-alexey commented Mar 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgarnov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smirnov-alexey commented Mar 28, 2022

Uh oh!

Uh oh!

smirnov-alexey commented Mar 29, 2022

Uh oh!

dmatveev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smirnov-alexey commented Apr 4, 2022

Uh oh!

smirnov-alexey commented Apr 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

smirnov-alexey commented Jan 24, 2022 •

edited

Loading

smirnov-alexey commented Mar 10, 2022 •

edited

Loading

smirnov-alexey commented Apr 5, 2022 •

edited

Loading