python: attempts to fix 3d mat parsing problem for dnn by fengyuentau · Pull Request #25810 · opencv/opencv

fengyuentau · 2024-06-24T04:06:28Z

Fixes #25762 #23242
Relates #25763 #19091

Although cv.Mat has already been introduced to workaround this problem, people do not know it and it kind of leads to confusion with numpy.array. This patch adds a "switch" to turn off the auto multichannel feature when the API is from cv::dnn::Net (more specifically, setInput) and the parameter is of type Mat. This patch only leads to changes of three places in pyopencv_generated_types_content.h:

static PyObject* pyopencv_cv_dnn_dnn_Net_setInput(PyObject* self, PyObject* py_args, PyObject* kw)
{
...
- pyopencv_to_safe(pyobj_blob, blob, ArgInfo("blob", 0)) &&
+ pyopencv_to_safe(pyobj_blob, blob, ArgInfo("blob", 8)) &&
...
}

// I guess we also need to change this as one-channel blob is expected for param
static PyObject* pyopencv_cv_dnn_dnn_Net_setParam(PyObject* self, PyObject* py_args, PyObject* kw)
{
...
- pyopencv_to_safe(pyobj_blob, blob, ArgInfo("blob", 0)) )
+ pyopencv_to_safe(pyobj_blob, blob, ArgInfo("blob", 8)) )
...
- pyopencv_to_safe(pyobj_blob, blob, ArgInfo("blob", 0)) )
+ pyopencv_to_safe(pyobj_blob, blob, ArgInfo("blob", 8)) )
...
}

Others are unchanged, e.g. dnn_SegmentationModel and stuff like that.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

…v::dnn::Net

asmorkalov · 2024-06-24T06:44:33Z

@fengyuentau should work for DNN as the most popular case, but what about other functions, e.g. contrib? There is no way to define the same behaviour there.

modules/python/src2/gen2.py

fengyuentau · 2024-06-24T08:04:57Z

@fengyuentau should work for DNN as the most popular case, but what about other functions, e.g. contrib? There is no way to define the same behaviour there.

Could you give an example on functions in contrib? I think, if we extract the filter self.mat_single_channel = (namespace == "cv.dnn" and classname == "dnn_Net") and put it in any place we want, we can do the same for other functions as well.

vpisarev · 2024-06-28T06:31:51Z

@fengyuentau, thank you! I'd suggest to add another function/method parameter attribute CV_INP_ND to bindings generator (hdr_parser.py) so that Python bindings generator could call ndarray=>cv::Mat conversion with proper flags.

e.g.

class Net { ...
CV_WRAP void setInput(CV_INP_ND InputArray tensor);
};

fengyuentau · 2024-06-28T06:42:34Z

@fengyuentau, thank you! I'd suggest to add another function/method parameter attribute CV_INP_ND to bindings generator (hdr_parser.py) so that Python bindings generator could call ndarray=>cv::Mat conversion with proper flags.

e.g.
class Net { ...
CV_WRAP void setInput(CV_INP_ND InputArray tensor);
};

Discussed and decided to use CV_ND for the macro.

fengyuentau · 2024-06-28T08:56:23Z

Done refactor with CV_ND.

asmorkalov · 2024-06-28T12:12:44Z

@fengyuentau Please add/modify python tests to cover the feature.

modules/dnn/misc/python/test/test_dnn.py

opencv-alalek

Output arrays should be handled too.

E.g. forward(OutputArrayOfArrays

opencv-alalek · 2024-06-29T04:52:42Z

There is dedicated header which is used to test binding generator features: https://github.com/opencv/opencv/blob/4.x/modules/core/include/opencv2/core/bindings_utils.hpp

fengyuentau · 2024-06-29T05:25:22Z

Output arrays should be handled too.

E.g. forward(OutputArrayOfArrays

Specifically speaking for numpy, there is no multi-channel numpy array. So there is no need to mark output.

opencv-alalek · 2024-06-29T19:40:29Z

There is ability to pass pre-allocated arrays via such API. So shapes of these arrays should be properly passed to C++ code.

asmorkalov

👍

fengyuentau · 2024-07-01T09:14:42Z

There is ability to pass pre-allocated arrays via such API. So shapes of these arrays should be properly passed to C++ code.

@opencv-alalek I updated code according to your comment. This introduce a convention:

The pre-allocated is a np.array, we treat it as a Mat and parse with ND flag.
The pre-allocated is a tuple of np.arrayl, we treat it as a vector of Mat and the ND flag can still apply if any.

Before this commit, the first dimension of a pre-allocated np.array is treated as the length of vector and the inner arrays are parsed as the elements of the vector.

asmorkalov · 2024-07-03T09:07:01Z

@opencv-alalek could you take a look again?

…d_mat_in_dnn python: attempts to fix 3d mat parsing problem for dnn opencv#25810 Fixes opencv#25762 opencv#23242 Relates opencv#25763 opencv#19091 Although `cv.Mat` has already been introduced to workaround this problem, people do not know it and it kind of leads to confusion with `numpy.array`. This patch adds a "switch" to turn off the auto multichannel feature when the API is from cv::dnn::Net (more specifically, `setInput`) and the parameter is of type `Mat`. This patch only leads to changes of three places in `pyopencv_generated_types_content.h`: ```.diff static PyObject* pyopencv_cv_dnn_dnn_Net_setInput(PyObject* self, PyObject* py_args, PyObject* kw) { ... - pyopencv_to_safe(pyobj_blob, blob, ArgInfo("blob", 0)) && + pyopencv_to_safe(pyobj_blob, blob, ArgInfo("blob", 8)) && ... } // I guess we also need to change this as one-channel blob is expected for param static PyObject* pyopencv_cv_dnn_dnn_Net_setParam(PyObject* self, PyObject* py_args, PyObject* kw) { ... - pyopencv_to_safe(pyobj_blob, blob, ArgInfo("blob", 0)) ) + pyopencv_to_safe(pyobj_blob, blob, ArgInfo("blob", 8)) ) ... - pyopencv_to_safe(pyobj_blob, blob, ArgInfo("blob", 0)) ) + pyopencv_to_safe(pyobj_blob, blob, ArgInfo("blob", 8)) ) ... } ``` Others are unchanged, e.g. `dnn_SegmentationModel` and stuff like that. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

turn off multichannel mat parsing in Python interface if it is from c…

d0ddfdf

…v::dnn::Net

fengyuentau added the category: python bindings label Jun 24, 2024

fengyuentau requested review from VadimLevin, asmorkalov, opencv-alalek and vpisarev June 24, 2024 04:06

asmorkalov reviewed Jun 24, 2024

View reviewed changes

modules/python/src2/gen2.py Outdated Show resolved Hide resolved

fengyuentau added RFC pr: Discussion Required labels Jun 27, 2024

refactor with CV_ND macro

b47dd4f

fengyuentau added this to the 4.11.0 milestone Jun 28, 2024

asmorkalov added pr: needs test New functionality requires minimal tests set and removed RFC pr: Discussion Required labels Jun 28, 2024

fengyuentau commented Jun 28, 2024

View reviewed changes

modules/dnn/misc/python/test/test_dnn.py Show resolved Hide resolved

add python test for setParam

a8589c9

fengyuentau commented Jun 28, 2024

View reviewed changes

modules/dnn/misc/python/test/test_dnn.py Show resolved Hide resolved

opencv-alalek reviewed Jun 29, 2024

View reviewed changes

This comment was marked as outdated.

Sign in to view

asmorkalov removed the pr: needs test New functionality requires minimal tests set label Jul 1, 2024

asmorkalov approved these changes Jul 1, 2024

View reviewed changes

handle forward(OutputArrayOfArrays, ...) ND case

74976e8

VadimLevin approved these changes Jul 1, 2024

View reviewed changes

asmorkalov assigned asmorkalov and opencv-alalek and unassigned asmorkalov Jul 3, 2024

opencv-alalek approved these changes Jul 3, 2024

View reviewed changes

asmorkalov merged commit 5510718 into opencv:4.x Jul 4, 2024

fengyuentau deleted the python/fix_parsing_3d_mat_in_dnn branch July 4, 2024 05:34

fengyuentau mentioned this pull request Jul 4, 2024

ONNX broken for identity function #25763

Open

4 tasks

asmorkalov mentioned this pull request Jul 16, 2024

(5.x) Merge 4.x #25915

Merged

Uh oh!

Conversation

fengyuentau commented Jun 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

asmorkalov commented Jun 24, 2024

Uh oh!

Uh oh!

fengyuentau commented Jun 24, 2024

Uh oh!

vpisarev commented Jun 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fengyuentau commented Jun 28, 2024

Uh oh!

fengyuentau commented Jun 28, 2024

Uh oh!

asmorkalov commented Jun 28, 2024

Uh oh!

Uh oh!

Uh oh!

opencv-alalek left a comment

Choose a reason for hiding this comment

Uh oh!

opencv-alalek commented Jun 29, 2024

Uh oh!

fengyuentau commented Jun 29, 2024

Uh oh!

opencv-alalek commented Jun 29, 2024

Uh oh!

This comment was marked as outdated.

asmorkalov left a comment

Choose a reason for hiding this comment

Uh oh!

fengyuentau commented Jul 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asmorkalov commented Jul 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fengyuentau commented Jun 24, 2024 •

edited

Loading

vpisarev commented Jun 28, 2024 •

edited

Loading

fengyuentau commented Jul 1, 2024 •

edited

Loading