G-API: oneVPL DX11 inference by sivanov-work · Pull Request #21232 · opencv/opencv

sivanov-work · 2021-12-10T12:02:26Z

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

Build Configuration

force_builders=XCustom,Custom Win,Custom Mac
build_gapi_standalone:Linux x64=ade-0.1.1f
build_gapi_standalone:Win64=ade-0.1.1f
build_gapi_standalone:Mac=ade-0.1.1f
build_gapi_standalone:Linux x64 Debug=ade-0.1.1f

Xbuild_image:Custom=centos:7
Xbuildworker:Custom=linux-1
build_gapi_standalone:Custom=ade-0.1.1f

build_image:Custom=ubuntu-openvino-2021.3.0:20.04
Xbuild_image:Custom Win=openvino-2021.2.0
build_image:Custom Mac=openvino-2021.2.0

test_modules:Custom=gapi,python2,python3,java
test_modules:Custom Win=gapi,python2,python3,java
test_modules:Custom Mac=gapi,python2,python3,java

buildworker:Custom=linux-1
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*

build_image:Custom Win=gapi-onevpl-2021.6.0
buildworker:Custom Win=windows-3
build_contrib:Custom Win=OFF

sivanov-work · 2021-12-15T09:39:17Z

one more commit is expected to fix CustomMac CI in a way to fix comparison between InferenceEngines types

mpashchenkov · 2021-12-21T07:34:36Z

modules/gapi/src/backends/ie/giebackend.cpp

@@ -510,15 +510,51 @@ inline IE::Blob::Ptr extractRemoteBlob(IECallContext& ctx, std::size_t i) {
    using ParamType = std::pair<InferenceEngine::TensorDesc,


Not for this changes but:
auto ie_core = cv::gimpl::ie::wrap::getCore(); - Is unused variable?

you right, it seems is unused

mpashchenkov · 2021-12-21T07:34:39Z

modules/gapi/src/backends/ie/giebackend.cpp

+    if (ctx.uu.params.device_id.find("GPU") != std::string::npos) {
+        GAPI_LOG_DEBUG(nullptr, "Extract remote blob for GPU DX11 device_id: " <<
+                                ctx.uu.params.device_id);
+        // despite of layout, blob dimensions always follow in N,C,H,W order


Will only 4D blob be supported?

it depends from openvino and code taken from openvino code make_shared_nv12_blob

mpashchenkov · 2021-12-21T07:34:42Z

modules/gapi/src/backends/ie/giebackend.cpp

+        // TODO NV12 surface supported only
+        if (static_cast<InferenceEngine::ColorFormat>(frame_blob_params->second["COLOR_FORMAT"]) ==
+            InferenceEngine::ColorFormat::NV12) {
+            InferenceEngine::Blob::Ptr y_blob =


Can be used InferenceEngine::NV12Blob for this case?

it has huge class hierarchy, but actually final blob must be one of descendant of ClBlob which represents remote GPU memory. Maybe NV12Blob is present in class inheritance in the middle chain, i'm not sure

mpashchenkov · 2021-12-21T07:34:45Z

modules/gapi/src/backends/ie/giebackend.cpp

+        if (static_cast<InferenceEngine::ColorFormat>(frame_blob_params->second["COLOR_FORMAT"]) ==
+            InferenceEngine::ColorFormat::NV12) {
+            InferenceEngine::Blob::Ptr y_blob =
+                           std::dynamic_pointer_cast<InferenceEngine::Blob>(


Can this be done with some make_..._blob functions? For interest.

yes it is: 524 - 550 lines related to make_shared_nv12_blob API function and to declare that function we need to include specific OV include file with this make* API but this header file has a required dependency from OpenCL -HPP package which is not a part of openCV and should be not according to license type. So all make_* functions from OV simply unwrapped from its definition and copied here

mpashchenkov · 2021-12-21T07:34:50Z

modules/gapi/src/backends/ie/giebackend.cpp

-                                *(ctx->uu.params.input_names.begin()),
-                                IE::make_shared_blob(this_blob, toIE(this_roi)),
-                                *ctx);
+                        setROIBlob(req,


I think, the function should be used for InferList and InferList2.
Can GPU ROI support appear in future?

Do you mean - place such changes SetBlob -> SetROIBlob in other place? could you point me out, please?

make_shared_blob for ROIs + SetBlob in InferList and InferList2. Is it necessary align this with SetRoiBlob? Now it is called in one place.

opencv/modules/gapi/src/backends/ie/giebackend.cpp

Line 1285 in e81100e

IE::Blob::Ptr roi_blob = IE::make_shared_blob(this_blob, toIE(rc));

opencv/modules/gapi/src/backends/ie/giebackend.cpp

Line 1444 in e81100e

this_blob = IE::make_shared_blob(blob_0, toIE(vec[list_idx]));

mpashchenkov · 2021-12-21T07:36:08Z

modules/gapi/src/backends/ie/giebackend.cpp

+                       const cv::Rect &roi,
+                       const IECallContext&           ctx) {
+    if (ctx.uu.params.device_id.find("GPU") != std::string::npos) {
+        GAPI_LOG_DEBUG(nullptr, "Skip ROI blob creation for device_id: " <<


This can confuse. M b it should be assert.

definitely no: GPU blob doesn't support ROI but need to continue with SeBlob further
Let's apply limitation that ROI for GPU (right before OpenVINO 2.0 adaptation) must be done on G-API / GSource side manually

modules/gapi/src/streaming/onevpl/engine/decode/decode_engine_legacy.hpp

modules/gapi/include/opencv2/gapi/streaming/onevpl/cfg_params.hpp

modules/gapi/samples/onevpl_infer_single_roi.cpp

modules/gapi/src/streaming/onevpl/accelerators/dx11_alloc_resource.cpp

modules/gapi/src/streaming/onevpl/cfg_params_parser.cpp

modules/gapi/src/streaming/onevpl/engine/decode/decode_session.hpp

modules/gapi/src/streaming/onevpl/engine/decode/decode_engine_legacy.cpp

modules/gapi/src/streaming/onevpl/engine/transcode/transcode_session.cpp

modules/gapi/src/streaming/onevpl/engine/transcode/transcode_engine_legacy.cpp

mpashchenkov

Some unanswered questions.

mpashchenkov · 2022-01-13T07:12:48Z

modules/gapi/src/streaming/onevpl/cfg_params.cpp

+                            static_cast<uint64_t>(value), false);
+}
+
+


Extra empty line.

dmatveev

Approved with comments

dmatveev · 2022-01-20T14:01:05Z

modules/gapi/samples/onevpl_infer_single_roi.cpp

+                     " For exploring list of supported transformations please find out "
+                     " vpp_* related stuff in"
+                     " gapi/include/opencv2/gapi/streaming/onevpl/cfg_params.hpp\n"
+                     " Pay attention that to obtain expected result In this case VPP "


no \n here

dmatveev · 2022-01-20T14:01:22Z

modules/gapi/samples/onevpl_infer_single_roi.cpp

+                     " Please consider param \"source_preproc_enable=1\" and specify "
+                     " appropriated media frame transformation using oneVPL::VPP primitives"


no \n here as well

dmatveev · 2022-01-20T14:01:38Z

modules/gapi/samples/onevpl_infer_single_roi.cpp

+                     " For exploring list of supported transformations please find out "
+                     " vpp_* related stuff in"


wow here too

dmatveev · 2022-01-20T14:04:19Z

modules/gapi/src/backends/ie/giebackend.cpp

+    ParamType* frame_blob_params = cv::util::any_cast<ParamType>(&any_blob_params);
+    if (frame_blob_params == nullptr) {
        GAPI_Assert(false && "Incorrect type of blobParams: "
                              "expected std::pair<InferenceEngine::TensorDesc,"
                                                 "InferenceEngine::ParamMap>");
    }


so there should be bad_any_cast already so no need to use *

dmatveev · 2022-01-20T14:05:21Z

modules/gapi/src/backends/ie/giebackend.cpp

-    return ctx.uu.rctx->CreateBlob(blob_params->first,
-                                   blob_params->second);
+    if (ctx.uu.params.device_id.find("GPU") != std::string::npos) {
+        GAPI_LOG_DEBUG(nullptr, "Extract remote blob for GPU DX11 device_id: " <<


I don't see the whole context but happens on Linux? GPU is perfectly valid there, but there's no DX11.

no MediaFrame Adapter for it - DX11 only
There is a check with assert whilst we create Source that available device type

OrestChura

Some remaining comments can be answered and addressed in the next PR

sivanov-work · 2022-01-20T14:50:05Z

it seems before merge need to rebase with latter nv12 changes in giebackend

alalek · 2022-01-20T19:58:33Z

dmatveev added this to the category: g-api milestone 6 hours ago

WTF???
Again, No new milestones please.

… due to OV issue

sivanov-work · 2022-01-24T13:55:18Z

@alalek Could you merge it please? all un-addressed comments would by applied in the next PR #21363
with some little refactoring

G-API: oneVPL DX11 inference * Draft GPU infer * Fix incorrect subresource_id for array of textures * Fix for TheOneSurface in different Frames * Turn on VPP param configuration * Add cropIn params * Remove infer sync sample * Remove comments * Remove DX11AllocResource extra init * Add condition for NV12 processing in giebackend * Add VPP frames pool param configurable * -M Remove extra WARN & INFOs, Fix custom MAC * Remove global vars from example, Fix some comments, Disable blobParam due to OV issue * Conflict resolving * Revert back pointer cast for cv::any

sivanov-work force-pushed the vpl_gpu_remote_infer branch from a20815b to 10d558a Compare December 13, 2021 06:43

sivanov-work requested review from OrestChura, mpashchenkov and smirnov-alexey December 15, 2021 09:38

sivanov-work marked this pull request as ready for review December 15, 2021 09:38

sivanov-work changed the title ~~G-API: oneVPL DX11 inference (WIP)~~ G-API: oneVPL DX11 inference Dec 15, 2021

sivanov-work added the category: g-api / gapi label Dec 15, 2021

mpashchenkov reviewed Dec 21, 2021

View reviewed changes

OrestChura reviewed Dec 22, 2021

View reviewed changes

OrestChura reviewed Dec 24, 2021

View reviewed changes

sivanov-work mentioned this pull request Dec 29, 2021

G-API: oneVPL Add VPP IE preproc #21363

Closed

6 tasks

smirnov-alexey mentioned this pull request Jan 11, 2022

Adapt remote inference to operate with NV12 blobs #21424

Merged

6 tasks

mpashchenkov reviewed Jan 13, 2022

View reviewed changes

dmatveev self-assigned this Jan 20, 2022

dmatveev added this to the category: g-api milestone Jan 20, 2022

dmatveev approved these changes Jan 20, 2022

View reviewed changes

OrestChura approved these changes Jan 20, 2022

View reviewed changes

sivanov-work added 5 commits January 21, 2022 08:15

Draft GPU infer

4049cf0

Fix incorrect subresource_id for array of textures

7880745

Fix for TheOneSurface in different Frames

805b88f

Turn on VPP param configuration

bdbfe93

Add cropIn params

eae7429

sivanov-work added 7 commits January 21, 2022 08:15

Remove infer sync sample

3e4f8f0

Remove comments

910522c

Remove DX11AllocResource extra init

e4a0142

Add condition for NV12 processing in giebackend

d184ed3

Add VPP frames pool param configurable

0041ffd

-M Remove extra WARN & INFOs, Fix custom MAC

b9a8d74

Remove global vars from example, Fix some comments, Disable blobParam…

8251811

… due to OV issue

sivanov-work force-pushed the vpl_gpu_remote_infer branch from 362b597 to 8251811 Compare January 21, 2022 12:35

sivanov-work added 2 commits January 21, 2022 15:36

Conflict resolving

1a85798

Revert back pointer cast for cv::any

e33c11d

alalek merged commit 266835c into opencv:4.x Jan 24, 2022

alalek mentioned this pull request Feb 22, 2022

(5.x) Merge 4.x #21651

Merged

		@@ -510,15 +510,51 @@ inline IE::Blob::Ptr extractRemoteBlob(IECallContext& ctx, std::size_t i) {
		using ParamType = std::pair<InferenceEngine::TensorDesc,

		" Please consider param \"source_preproc_enable=1\" and specify "
		" appropriated media frame transformation using oneVPL::VPP primitives"

		" For exploring list of supported transformations please find out "
		" vpp_* related stuff in"

Uh oh!

Conversation

sivanov-work commented Dec 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Build Configuration

Uh oh!

sivanov-work commented Dec 15, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mpashchenkov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmatveev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

OrestChura left a comment

Choose a reason for hiding this comment

Uh oh!

sivanov-work commented Jan 20, 2022

sivanov-work commented Dec 10, 2021 •

edited

Loading