[G-API]: Adding reshape for CNN input. by mpashchenkov · Pull Request #18240 · opencv/opencv

mpashchenkov · 2020-09-01T10:24:19Z

Adding reshape for CNN input

Reshape changes shape for CNN input dims.

To use input reshape, you should call function cfgInputReshape for Params:

.cfgInputReshape( { {layer1_name, shape1}, {layer3_name, shape3} } ) - for some layers;
.cfgInputReshape(layer1_name, shape1) - for single layer;
.cfgInputReshape( { layer1_name, layer3_name } ) - for reshaping by input image.

layer..._name - name of input layer, which should to be reshaped.
shape... - vector of dimensions, which reshape will use (1,3,300,300 for example).

Behavior:

input_reshape_table from IEUnit contains pairs layer's name --- dimensions for reshape;
Unfortunately, now current CNN input layers dimensions are not checked, but if provided incorrect dimensions in 1st and 2nd cases of cfgInputReshape then IE throw error message;
In 3rd case of cfgInputReshape are constructing element of input_reshape_table by current CNN dimensions and height and width input image;
If user call 1st, 2nd and 3rd case of .cfgInputReshape at the same time then reshape by dimensions is preferred then by input image;
Call of InferenceEngine::CNNNetwork::reshape function is located in OutMeta() of Infer, InferROI, InferList and InferList2.

Tests:
New fixture InferWithReshape:

TestInfer - simple infer by shapes;
TestInferInImage - infer by input image size;
TestInferBehavior - launch with 1st and 3rd case of .cfgInputReshape at the same time;
TestInferForSingleLayer - launch with 2nd case of .cfgInputReshape;
TestInferList - infer by ROIs with CNN input reshape;
TestInferList2;
TestInferListBGR - infer GFrame (BGR).
New fixture InferWithReshapeNV12:
TestInferListYUV - infer GFrame (YUV)..

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

Magic commands

force_builders=Custom,Custom Win,Custom Mac
build_gapi_standalone:Linux x64=ade-0.1.1f
build_gapi_standalone:Win64=ade-0.1.1f
build_gapi_standalone:Mac=ade-0.1.1f
build_gapi_standalone:Linux x64 Debug=ade-0.1.1f

build_image:Custom=centos:7
buildworker:Custom=linux-1
build_gapi_standalone:Custom=ade-0.1.1f

xbuild_image:Custom=ubuntu-openvino-2021.1.0:20.04
xbuild_image:Custom Win=openvino-2021.1.0
xbuild_image:Custom Mac=openvino-2021.1.0

test_modules:Custom=gapi
test_modules:Custom Win=gapi
test_modules:Custom Mac=gapi

# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*

mpashchenkov · 2020-09-01T10:25:27Z

@OrestChura, @aDanPin, please review.

modules/gapi/src/backends/ie/giebackend.cpp

modules/gapi/test/infer/gapi_infer_ie_test.cpp

modules/gapi/src/backends/ie/giebackend.cpp

OrestChura

Overall looks consistent

modules/gapi/include/opencv2/gapi/infer/ie.hpp

modules/gapi/src/backends/ie/giebackend.cpp

modules/gapi/include/opencv2/gapi/infer/ie.hpp

OrestChura

Great! Thanks!
But please check if PR causes IE tests failure

mpashchenkov · 2020-11-23T09:32:49Z

@dmatveev

asmorkalov · 2020-11-26T07:56:37Z

@dmatveev @alalek The patch is ready, let's merge it.

dmatveev · 2020-12-01T12:25:32Z

cfgInputReshape( { {layer1_name, shape1}, {layer3_name, shape3} } )

So what is shape here? In case of IE, isn't reshape just a boolean flag and the input blob is reshaped to the input image dimensions? Why specify a shape here at all?

dmatveev · 2020-12-01T12:26:03Z

..also, if this function is not called for all N inputs, then I'd limit it to configuring a single input only.

dmatveev · 2020-12-01T12:27:09Z

modules/gapi/include/opencv2/gapi/infer/ie.hpp

+    Params<Net>& cfgInputReshape(const std::map<std::string, std::vector<std::size_t>> &r) {
+        desc.reshape_inputs = r;
+        return *this;
+    }


Really I'd add an overload with const std::string, const std::vector<size_t>.

@mpashchenkov what about this?

Consider to add rvalue implementation, because it will be used in most of cases

Added overloads.

dmatveev · 2020-12-01T12:27:57Z

modules/gapi/src/backends/ie/giebackend.cpp

-            // cv::Mat-wrapped blob there to support IE's optimal "GetBlob idiom"
-            // Still, constant data is to set only once.
-            this_request.SetBlob(p.first, wrapIE(p.second.first, p.second.second));
-        }


Where did it go?

It was a duplicate code. Already fixed.

dmatveev · 2020-12-01T12:28:36Z

modules/gapi/src/backends/ie/giebackend.cpp

+        // Reshape for configured map with layers' names and their new shapes
+        if (!params.reshape_inputs.empty()) {
+            GAPI_Assert(params.reshape_inputs.size() <= params.num_in);
+            net.reshape(params.reshape_inputs);


So how it works for your purpose? Why we're not looking at the input data dimensions? Isn't this the case?

dmatveev · 2020-12-01T12:29:16Z

Another question - how does this reshape thing co-exists with the automatic preprocessing we set?

asmorkalov · 2020-12-09T14:34:49Z

@mpashchenkov Please take a look on new comments and merge conflict.

modules/gapi/include/opencv2/gapi/infer/ie.hpp

dmatveev · 2021-01-12T17:50:41Z

modules/gapi/include/opencv2/gapi/infer/ie.hpp

+    Params<Net>& cfgInputReshape(const std::map<std::string, std::vector<std::size_t>> &r) {
+        desc.reshape_inputs = r;
+        return *this;
+    }


@mpashchenkov what about this?

dmatveev · 2021-01-12T17:52:25Z

modules/gapi/src/backends/ie/giebackend.cpp

+    size_t to_hw_shift = 0;
+    switch (layout) {
+        case InferenceEngine::NHWC:
+        {


you don't need {/} as you don't declare local variables here

dmatveev · 2021-01-12T17:53:10Z

modules/gapi/src/backends/ie/giebackend.cpp

+    const auto input_name = ii->name();
+    const auto idx = getIdxByName(uu->params.reshape_in_names, input_name);
+    uu->params.reshape_in_shapes[uu->params.reshape_in_names.at(idx)] = input_dims;
+}


Can you please explain what happens here in this function?

Added comments about steps.

dmatveev · 2021-01-12T17:55:01Z

modules/gapi/src/backends/ie/giebackend.cpp

        auto &&ii = uu.inputs.at(uu.params.input_names.at(0));
        auto &&mm = in_metas.at(1u);
-        configureInputInfo(ii, mm);
+        configureInputInfo(ii, const_cast<IEUnit*>(&uu), mm);


Generally, const_cast is not good. It breaks the above contracts where we declare our inputs as const.

Added FIXME.

dmatveev · 2021-01-12T17:58:19Z

modules/gapi/src/backends/ie/giebackend.cpp

+                if (!uu->params.reshape_in_names.empty()) {
+                    configureReshape(ii, uu, {meta.size.height, meta.size.width});


Not sure if this logic is all correct.

Hypothetically, one kind of reshape shouldn't exclude the other one - and so for an N-input network the user could specify particular shapes for layers X and Y and keep it dynamic (based on input) for Z.

Also, is {W,H} the right shape? Does it work well with SetResizeAlgorithm() / SetBlob thing we also use?

New behavior:
You can call reshape by shapes parameter for some layers and call reshape by image size for another layers.
If both calls contain same layers names, then reshaping by shapes is preferred.

dmatveev · 2021-01-12T17:59:07Z

@TolyaTalamanov please have a look on this.

modules/gapi/include/opencv2/gapi/infer/ie.hpp

TolyaTalamanov · 2021-01-18T09:22:01Z

modules/gapi/include/opencv2/gapi/infer/ie.hpp

+    Params<Net>& cfgInputReshape(const std::map<std::string, std::vector<std::size_t>> &r) {
+        desc.reshape_inputs = r;
+        return *this;
+    }


Consider to add rvalue implementation, because it will be used in most of cases

modules/gapi/src/backends/ie/giebackend.cpp

modules/gapi/test/infer/gapi_infer_ie_test.cpp

modules/gapi/src/backends/ie/giebackend.cpp

modules/gapi/include/opencv2/gapi/infer/ie.hpp

modules/gapi/src/backends/ie/giebackend.cpp

mpashchenkov · 2021-01-25T11:25:07Z

Updated magic commands (openvino-2021.1.0 version). Old version doesn't support some blob functionality. @alalek, please restart build for this PR, it should work correctly.

alalek · 2021-01-25T11:40:58Z

Old version doesn't support some blob functionality

You should add #if guards with version check to handle that:

do not fail compilation. Use NonImplemented exceptions in #else branch.
do not fail tests. Skip them.

P.S. OV 2020.3 LTS should be supported too (with partially disabled functionality)

/cc @opencv/intel-gapi

modules/gapi/include/opencv2/gapi/infer/ie.hpp

modules/gapi/src/backends/ie/giebackend.cpp

TolyaTalamanov · 2021-03-03T09:44:46Z

Good job, thanks! 👍

mpashchenkov · 2021-03-03T13:47:10Z

@alalek, CN Jenkins can't reach https://github.com/opencv/opencv.git, but build bot's build is correct. Do need to restart CN Jenkins or can merge it by build bot's result?

alalek · 2021-03-03T14:03:17Z

Ignore, these failures are not related to any patch.

mpashchenkov · 2021-03-04T07:32:02Z

@alalek, can you merge this PR?

…nn-reshape [G-API]: Adding reshape for CNN input. * Added CNN input IE reshape * rbs * Added unordered_set instead vector * Alignment

OrestChura reviewed Sep 2, 2020

View reviewed changes

modules/gapi/src/backends/ie/giebackend.cpp Outdated Show resolved Hide resolved

modules/gapi/test/infer/gapi_infer_ie_test.cpp Outdated Show resolved Hide resolved

modules/gapi/src/backends/ie/giebackend.cpp Outdated Show resolved Hide resolved

mpashchenkov force-pushed the mp/ocv-gapi-input-cnn-reshape branch from 20d59f4 to d1887e4 Compare November 12, 2020 12:51

mpashchenkov requested a review from OrestChura November 12, 2020 14:46

OrestChura reviewed Nov 13, 2020

View reviewed changes

OrestChura approved these changes Nov 16, 2020

View reviewed changes

mpashchenkov requested a review from dmatveev November 27, 2020 14:09

dmatveev reviewed Dec 1, 2020

View reviewed changes

mpashchenkov force-pushed the mp/ocv-gapi-input-cnn-reshape branch from 991afe9 to 1189a29 Compare January 12, 2021 09:26

dmatveev reviewed Jan 12, 2021

View reviewed changes

dmatveev requested a review from TolyaTalamanov January 12, 2021 17:58

TolyaTalamanov reviewed Jan 18, 2021

View reviewed changes

mpashchenkov commented Jan 18, 2021

View reviewed changes

modules/gapi/src/backends/ie/giebackend.cpp Outdated Show resolved Hide resolved

TolyaTalamanov reviewed Jan 19, 2021

View reviewed changes

modules/gapi/include/opencv2/gapi/infer/ie.hpp Show resolved Hide resolved

modules/gapi/src/backends/ie/giebackend.cpp Show resolved Hide resolved

TolyaTalamanov reviewed Jan 19, 2021

View reviewed changes

modules/gapi/src/backends/ie/giebackend.cpp Outdated Show resolved Hide resolved

mpashchenkov changed the title ~~[G-API]: Add reshape for CNN input.~~ [G-API]: Adding reshape for CNN input. Jan 20, 2021

mpashchenkov requested a review from dmatveev January 22, 2021 15:20

mpashchenkov requested a review from TolyaTalamanov January 28, 2021 10:37

TolyaTalamanov requested changes Feb 8, 2021

View reviewed changes

modules/gapi/include/opencv2/gapi/infer/ie.hpp Outdated Show resolved Hide resolved

TolyaTalamanov reviewed Feb 8, 2021

View reviewed changes

modules/gapi/include/opencv2/gapi/infer/ie.hpp Outdated Show resolved Hide resolved

TolyaTalamanov reviewed Feb 8, 2021

View reviewed changes

modules/gapi/include/opencv2/gapi/infer/ie.hpp Outdated Show resolved Hide resolved

TolyaTalamanov reviewed Feb 8, 2021

View reviewed changes

modules/gapi/include/opencv2/gapi/infer/ie.hpp Outdated Show resolved Hide resolved

TolyaTalamanov requested changes Feb 8, 2021

View reviewed changes

TolyaTalamanov reviewed Feb 9, 2021

View reviewed changes

mpashchenkov force-pushed the mp/ocv-gapi-input-cnn-reshape branch from a705a8a to 7bb1fe3 Compare February 26, 2021 13:13

Maxim Pashchenkov added 2 commits March 1, 2021 11:49

Added CNN input IE reshape

af9a884

rbs

1e65dae

mpashchenkov force-pushed the mp/ocv-gapi-input-cnn-reshape branch from 7bb1fe3 to 1e65dae Compare March 1, 2021 08:54

Added unordered_set instead vector

5f0fbd5

TolyaTalamanov reviewed Mar 3, 2021

View reviewed changes

modules/gapi/src/backends/ie/giebackend.cpp Outdated Show resolved Hide resolved

Alignment

8dae695

TolyaTalamanov approved these changes Mar 3, 2021

View reviewed changes

mpashchenkov assigned TolyaTalamanov Mar 4, 2021

TolyaTalamanov approved these changes Mar 4, 2021

View reviewed changes

alalek merged commit 12fa8d8 into opencv:master Mar 10, 2021

alalek mentioned this pull request Apr 9, 2021

(5.x) Merge 4.x #19885

Merged

alalek mentioned this pull request Nov 20, 2022

gapi: fix InferWithReshape test crash when data is not found #22831

Merged

		if (!uu->params.reshape_in_names.empty()) {
		configureReshape(ii, uu, {meta.size.height, meta.size.width});

Uh oh!

Conversation

mpashchenkov commented Sep 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Adding reshape for CNN input

Pull Request Readiness Checklist

Magic commands

Uh oh!

mpashchenkov commented Sep 1, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OrestChura left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OrestChura left a comment

Choose a reason for hiding this comment

Uh oh!

mpashchenkov commented Nov 23, 2020

Uh oh!

asmorkalov commented Nov 26, 2020

Uh oh!

dmatveev commented Dec 1, 2020

Uh oh!

dmatveev commented Dec 1, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mpashchenkov Jan 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmatveev commented Dec 1, 2020

Uh oh!

asmorkalov commented Dec 9, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmatveev commented Jan 12, 2021

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mpashchenkov commented Sep 1, 2020 •

edited

Loading

mpashchenkov Jan 20, 2021 •

edited

Loading