Added ResizeBilinear op for tf by dmonterom · Pull Request #11050 · opencv/opencv

dmonterom · 2018-03-10T12:54:17Z

This pullrequest changes

Added ResizeBilinear op for tensorflow importer.

dkurt · 2018-03-12T08:00:36Z

modules/dnn/include/opencv2/dnn/all_layers.hpp

        static Ptr<ResizeNearestNeighborLayer> create(const LayerParams& params);
    };

+    class CV_EXPORTS ResizeBilinearLayer : public Layer


@dmonterom, what do you think about renaming ResizeNearestNeighborLayer to ResizeLayer and use interpolation strategy as an additional flag? Please add a test similar to nearest neighbor layer

opencv/modules/dnn/test/test_tf_importer.cpp

Lines 325 to 328 in 5b868cc

TEST(Test_TensorFlow, resize_nearest_neighbor)

{

runTensorFlowNet("resize_nearest_neighbor");

}

Using script from opencv/opencv_extra:
https://github.com/opencv/opencv_extra/blob/b467369489529d8eb2e847c13b117ec4e253a638/testdata/dnn/tensorflow/generate_tf_models.py#L258-L262

Hi @dkurt, thats a good idea, I will change it as soon as possible. I will also add the new test soon.

@dmonterom, thank you! It also can help us to resolve an issue #10985 with Interp layer from some of Caffe's forks smoother.

Hey @dkurt, I made the changes (I also tested the two layers and everything seems ok). If you agree with them i will replace the resize nearest neighbor test with a new one.

@dmonterom please add *_net, *_in and *_out files generated by script as a PR to opencv_extra with modified version of the script.

dkurt · 2018-03-12T13:26:54Z

modules/dnn/src/tensorflow/tf_importer.cpp


            layerParams.set("height", outSize.at<int>(0, 0));
            layerParams.set("width", outSize.at<int>(0, 1));
+            layerParams.set("interpolation", 0);


Enumerations INTER_NEAREST and INTER_LINEAR are also integers. You can just set them as arguments, layerParams.set("interpolation", INTER_NEAREST); and use interpolation like resize(getPlane(inp, n, ch), getPlane(out, n, ch), Size(outWidth, outHeight), 0, 0, interpolation);

dkurt · 2018-03-12T13:28:27Z

modules/dnn/src/tensorflow/tf_importer.cpp

+
+            connect(layer_id, dstNet, parsePin(layer.input(0)), id, 0);
+        }
+        else if (type == "ResizeBilinear")


Body of that case is absolutely the same. Can you just extend the previous one by else if (type == "ResizeNearestNeighbor" || type == "ResizeBilinear") and vary "interpolation" flag?

dkurt · 2018-03-12T13:39:07Z

modules/dnn/src/init.cpp

    CV_DNN_REGISTER_LAYER_CLASS(Reshape,        ReshapeLayer);
    CV_DNN_REGISTER_LAYER_CLASS(Flatten,        FlattenLayer);
-    CV_DNN_REGISTER_LAYER_CLASS(ResizeNearestNeighbor, ResizeNearestNeighborLayer);
+    CV_DNN_REGISTER_LAYER_CLASS(Resize, ResizeLayer);


Please align ResizeLayer with the other layers names.

dkurt · 2018-03-13T09:37:03Z

modules/dnn/src/tensorflow/tf_importer.cpp

            layerParams.set("height", outSize.at<int>(0, 0));
            layerParams.set("width", outSize.at<int>(0, 1));
+            if (type == "ResizeNearestNeighbor")
+                layerParams.set("interpolation", 0);


I think INTER_NEAREST or INTER_LINEAR should be here.

Hi, do you mean replacing the values by them? Then I should define them at the beginning or include imgproc.hpp, right?

Why? They are just defined in OpenCV: https://docs.opencv.org/master/da/d54/group__imgproc__transform.html#ga5bb5a1fea74ea38e1a5445ca803ff121 . And please do not forget to add a test.

dmonterom · 2018-03-14T09:14:38Z

Hi @dkurt, I have created a test for resize bilinear using the generator script, but when i run the test it fails:

error: Expected: (normL1) <= (l1), actual: 0.122411 vs 1e-05
error: Expected: (normInf) <= (lInf), actual: 1.06526 vs 0.0001

I have also regenerate the resize nearest one but it successes, so its not about my tensorflow version.
Maybe they are using different bilinear interpolation approaches?

dkurt · 2018-03-14T09:23:28Z

@dmonterom, that is why we need to test it. Perhaps the problem is in borders.Try to figure out what values are different. Test on small inputs like 1x5x5x1 and compare outputs.

dmonterom · 2018-03-14T14:21:35Z

@dkurt, I have been doing several tests with different sizes and they always fail. I found this information:
https://hackernoon.com/how-tensorflows-tf-image-resize-stole-60-days-of-my-life-aba5eb093f35
https://stackoverflow.com/questions/48628736/bilinear-interpolation-implentations-in-tensorflow-and-opencv?rq=1
So it seems that there is a bug in the tensorflow resize op. Anyway, the actual implementation of the layer does the trick: I have imported blitznet with dnn (which has several resize_bilinear layers) and the results are very close to the ones I get with tensorflow.

dkurt · 2018-03-14T15:31:21Z

@dmonterom, it'd nice if you can attach the results of your experiments. We need to know if difference is only at boundaries or in corners or in a whole image. Thanks!

dmonterom · 2018-03-14T16:20:55Z

@dkurt, here is an example of blitznet with tensorflow and with OpenCV

Here is the result with tensorflow:

And here with OpenCV:

As you can see, there is a small accuracy drop with OpenCV but the results are quite good.

dkurt · 2018-03-14T16:49:58Z

Thank you but I meant numerical experiments with resize of 5x5 inputs. We need to compare numbers first.

dkurt · 2018-03-21T10:39:27Z

@dmonterom, thank you for your work and research! It seems that TensorFlow's resize_bilinear is a quite different from OpenCV's INTER_LINEAR. Let me introduce you an experimental mechanic of custom layers. All that you need is just use changes from PR #11129 (work in progress for now). You may find a tutorial there.

dkurt · 2018-06-04T15:25:40Z

@dmonterom, We slightly modified your PR to merge it. May I ask you to try it with your model again?

alalek · 2018-06-04T15:31:52Z

modules/dnn/src/layers/resize_layer.cpp

+                float input_y = y * heightScale;
+                int y0 = static_cast<int>(input_y);
+                const float* inpData_row0 = (float*)inp.data + y0 * inpWidth;
+                const float* inpData_row1 = (y0 + 1 < inpHeight) ? (inpData_row0 + inpWidth) : inpData_row0;


Use .ptr<float>(y0)
and .ptr<float>(std::min(y0 + 1, inpHeight - 1))

Thanks for a review! Replaced it.

alalek · 2018-06-04T15:32:01Z

modules/dnn/src/layers/resize_layer.cpp

+                    int x0 = static_cast<int>(input_x);
+                    int x1 = std::min(x0 + 1, inpWidth - 1);
+
+                    float* outData = (float*)out.data + y * outWidth + x;


alalek · 2018-06-04T15:39:00Z

modules/dnn/src/layers/resize_layer.cpp

+                        *outData = inpData_row0_c[x0] +
+                            (input_y - y0) * (inpData_row1_c[x0] - inpData_row0_c[x0]) +
+                            (input_x - x0) * (inpData_row0_c[x1] - inpData_row0_c[x0] +
+                            (input_y - y0) * (inpData_row1_c[x1] - inpData_row0_c[x1] - inpData_row1_c[x0] + inpData_row0_c[x0]));


Why OpenCV cv::resize() with BORDER_REPLICATE doesn't work here? (answer: 0.5 pixel center)

Anyway, if you want to reuse OpenCV's INTER_LINEAR constant then it should have the same behavior as defined by this constant.

@alalek, There are different topics that TenorFlow's resize_bilinear doesn't equal to OpenCV's one.

Replaced constants to string values.

@alalek, if we extend cv::resize in the master branch and add the optional floating-point shift, will that make cv::resize suitable for implementation of this layer?

alternatively, can we use cv::warpAffine with the transformation matrix [fx, 0, shift_x; 0, fy, shift_y] to emulate the proper resize + shift?

Two different formulas are usually used for resize operation:

x * widthScale

Something like (x + 0.5) * widthScale - 0.5

The second formula is used in OpenVX, OpenCV's INTER_LINEAR resize (and probably generic INTER_AREA).
The first is used in OpenCV INTER_NEAREST (not accurate, but it is fast) / INTER_AREA (fast path).

Here, we need to use the first formula with "linear" interpolation (to match original framework operation).

@alalek, it means that cv::warpAffine can do that?

Combined ResizeNearestNeighbor and ResizeBilinear layers into Resize (with an interpolation param). Minor changes to tf_importer and resize layer to save some code lines Minor changes in init.cpp Minor changes in tf_importer.cpp

dkurt self-assigned this Mar 12, 2018

dkurt reviewed Mar 12, 2018

View reviewed changes

dkurt reviewed Mar 13, 2018

View reviewed changes

dkurt mentioned this pull request Mar 21, 2018

Custom layers for deep learning networks #11129

Merged

vpisarev assigned vpisarev and unassigned vpisarev Apr 9, 2018

dkurt force-pushed the added_resize_bilinear_layer branch 2 times, most recently from c1510ed to d796353 Compare June 4, 2018 15:23

alalek reviewed Jun 4, 2018

View reviewed changes

dmontero and others added 3 commits June 7, 2018 14:26

Added ResizeBilinear op for tf

bb3f791

Combined ResizeNearestNeighbor and ResizeBilinear layers into Resize (with an interpolation param). Minor changes to tf_importer and resize layer to save some code lines Minor changes in init.cpp Minor changes in tf_importer.cpp

Replaced implementation of a custom ResizeBilinear layer to all layers

bcac26b

Use Mat::ptr. Replace interpolation flags

5a7ca1b

dkurt force-pushed the added_resize_bilinear_layer branch from 59ee2aa to 5a7ca1b Compare June 7, 2018 11:31

vpisarev merged commit 7175f25 into opencv:3.4 Jun 7, 2018

alalek mentioned this pull request Jun 13, 2018

Merge 3.4 #11756

Merged

alalek mentioned this pull request Sep 2, 2021

Fix resize+concat optimization #20647

Merged

6 tasks

	TEST(Test_TensorFlow, resize_nearest_neighbor)
	{
	runTensorFlowNet("resize_nearest_neighbor");
	}

Uh oh!

Conversation

dmonterom commented Mar 10, 2018

This pullrequest changes

Uh oh!

dkurt Mar 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmonterom commented Mar 14, 2018

Uh oh!

dkurt commented Mar 14, 2018

Uh oh!

dmonterom commented Mar 14, 2018 • edited by dkurt Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dkurt commented Mar 14, 2018

Uh oh!

dmonterom commented Mar 14, 2018

Uh oh!

dkurt commented Mar 14, 2018

Uh oh!

dkurt commented Mar 21, 2018

Uh oh!

dkurt commented Jun 4, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vpisarev Jun 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dkurt Mar 12, 2018 •

edited

Loading

dmonterom commented Mar 14, 2018 •

edited by dkurt

Loading

vpisarev Jun 6, 2018 •

edited

Loading