Skip to content

ONNX: upsample subgraph fusion added#16573

Merged
alalek merged 1 commit intoopencv:3.4from
ashishkrshrivastava:opencvonnx
Feb 29, 2020
Merged

ONNX: upsample subgraph fusion added#16573
alalek merged 1 commit intoopencv:3.4from
ashishkrshrivastava:opencvonnx

Conversation

@ashishkrshrivastava
Copy link
Copy Markdown
Contributor

@ashishkrshrivastava ashishkrshrivastava commented Feb 13, 2020

Merge with extra: opencv/opencv_extra#707
resolves #16323
resolves #16475
resolves #14713

opencv_extra=opencvonnx

Pull Request Readiness Checklist

  • I agree to contribute to the project under OpenCV (BSD) License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
  • The PR is proposed to proper branch
  • There is reference to original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

This PR adds

Subgraph fusion for upsample layer when exported from pytorch 1.3
Screenshot from 2020-02-13 17-43-16

If model is exported using pytorch 1.2, it does not produce two extra cast layers after Gather layer.
Screenshot from 2020-02-13 17-46-06

The additions I made are according to latest version of Pytorch.

allow_multiple_commits=1

@ashishkrshrivastava
Copy link
Copy Markdown
Contributor Author

There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.

@dkurt , do both opencv and opencv_extra branches need to have same name ?

unknown file: Failure
C++ exception with description "OpenCV(3.4.9-dev) /build/precommit_macosx/3.4/opencv/modules/ts/src/ts.cpp:1052: error: (-2:Unspecified error) OpenCV tests: Can't find required data file: dnn/onnx/models/upsample_unfused.onnx in function 'findData'
" thrown in the test body.

@dkurt
Copy link
Copy Markdown
Member

dkurt commented Feb 13, 2020

@ashishkrshrivastava, by default yes. If you already have one - add it's name to PR description

@ashishkrshrivastava
Copy link
Copy Markdown
Contributor Author

ashishkrshrivastava commented Feb 18, 2020

@dkurt

node = onnx.helper.make_node(
    'Resize',
    inputs=['X', 'roi', 'scales'],
    outputs=['Y'],
    mode='cubic',
)

data = np.array([[[
    [1, 2, 3, 4],
    [5, 6, 7, 8],
    [9, 10, 11, 12],
    [13, 14, 15, 16],
]]], dtype=np.float32)

roi = np.array([], dtype=np.float32)
scales = np.array([1.0, 1.0, 0.8, 0.8], dtype=np.float32)

# [[[[ 1.47119141  2.78125     4.08251953]
#    [ 6.71142578  8.02148438  9.32275391]
#    [11.91650391 13.2265625  14.52783203]]]]
output = interpolate_nd(
    data, cubic_coeffs, scale_factors=scales).astype(np.float32)

expect(node, inputs=[data, roi, scales], outputs=[output],

Here, In ONNX resize node example ROI is empty, and hence ONNX model can contain an empty constant node. I am getting an empty node when exporting pytorch model containing upsample layer using ONNX opset version 11. I want your suggestion regarding this empty constant node.
It need to be handled otherwise it would create problem here,

Mat getMatFromTensor(opencv_onnx::TensorProto& tensor_proto)
{
    CV_Assert(!tensor_proto.raw_data().empty() || !tensor_proto.float_data().empty()
                    || !tensor_proto.double_data().empty() || !tensor_proto.int64_data().empty());

I think we can add some dummy data or can remove the node.

[SOLVED]

i have removed the node at finalize function.

@alalek
Copy link
Copy Markdown
Member

alalek commented Feb 18, 2020

Conflicting files

Please rebase patch on latest commits from 3.4 branch:

git fetch updstream
git rebase -i upstream/3.4

( looks like it conflicts with #16472 - also please split related "Resize" test on smaller parts )

@ashishkrshrivastava
Copy link
Copy Markdown
Contributor Author

ashishkrshrivastava commented Feb 18, 2020

Conflicting files

Please rebase patch on latest commits from 3.4 branch:

Okay, thanks. Let me do that.

@ashishkrshrivastava
Copy link
Copy Markdown
Contributor Author

ashishkrshrivastava commented Feb 21, 2020

@dkurt , I just encountered a problem while loading
net = cv2.dnn.readNetFromONNX("/home/ashish/Downloads/Object Detection & Image Segmentation/yolov3/yolov3.onnx")
Here is the model.
It attempted to match SoftmaxSubgraph ( div node ).

cv2.error: OpenCV(4.2.0-dev) /home/ashish/opencv/modules/dnn/src/graph_simplifier.cpp:84: error: (-212:Parsing error) Input node with name const_fold_opt__570 not found in function 'getInputNodeId'

Here, const_fold_opt__570 is initializer.
Screenshot from 2020-02-21 15-53-22

So, it is neither node nor output of some node, I guess. What do you suggest about it?

@dkurt
Copy link
Copy Markdown
Member

dkurt commented Feb 21, 2020

@ashishkrshrivastava, I'd like to recommend to move it to separate issue / PR if you do not mind.

@ashishkrshrivastava
Copy link
Copy Markdown
Contributor Author

@dkurt , I just encountered a problem while loading
net = cv2.dnn.readNetFromONNX("/home/ashish/Downloads/Object Detection & Image Segmentation/yolov3/yolov3.onnx")
Here is the model.
It attempted to match SoftmaxSubgraph ( div node ).

cv2.error: OpenCV(4.2.0-dev) /home/ashish/opencv/modules/dnn/src/graph_simplifier.cpp:84: error: (-212:Parsing error) Input node with name const_fold_opt__570 not found in function 'getInputNodeId'

Here, const_fold_opt__570 is initializer.
Screenshot from 2020-02-21 15-53-22

So, it is neither node nor output of some node, I guess. What do you suggest about it?

@dkurt, Are you talking about moving only this issue to another issue/PR or everything ?

@dkurt
Copy link
Copy Markdown
Member

dkurt commented Feb 21, 2020

Yeah. So if this PR can already fix two mentioned issues (#16323 and #16475) - let's finish it. Or they also require this SoftMax fusion?

@ashishkrshrivastava
Copy link
Copy Markdown
Contributor Author

okay. for these two issues we dont need to worry about softamx fusion.

@dkurt
Copy link
Copy Markdown
Member

dkurt commented Feb 21, 2020

@ashishkrshrivastava, thank you!

@ashishkrshrivastava
Copy link
Copy Markdown
Contributor Author

@dkurt , In issue 16475 Unet model is obtained using Pytorch version (1.3) while modifications in this PR are according to Pytorch version (1.4). should I make changes according to version 1.3 or it is fine to continue with version 1.4(latest) ? I am able to load Unet exported using version 1.4 with current modifications.

@dkurt
Copy link
Copy Markdown
Member

dkurt commented Feb 22, 2020

@ashishkrshrivastava, If this PR can fix the issues - let's focus on current state. We need to be user oriented and if one have 1.4 version - we need to fix it here.

opencv_onnx::TensorProto tensor_proto = constant_node->attribute(0).t();
nodes_attributes.push_back(getMatFromTensor(tensor_proto));
}
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What this loop actually does?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sequence of our subgraph is

Shape Constant Gather Cast Constant Mul Cast Floor Shape Constant Gather Cast Constant Mul Cast Floor Unsqueeze Unsqueeze

While matchedNodesIds contain vector ids of nodes

Shape Gather Cast Mul Cast Floor Shape Gather Cast Mul Cast Floor Unsqueeze Unsqueeze
i.e. without constant.
so this loop is searching for consant nodes that are missing from matchedNodeIds because we need to extract data from those constant nodes.

Copy link
Copy Markdown
Member

@dkurt dkurt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Thank you!

@dkurt dkurt self-assigned this Feb 29, 2020
@alalek alalek merged commit 599a595 into opencv:3.4 Feb 29, 2020
@ashishkrshrivastava ashishkrshrivastava deleted the opencvonnx branch March 4, 2020 02:39
@alalek alalek mentioned this pull request Mar 4, 2020
@BadMachine
Copy link
Copy Markdown

@ashishkrshrivastava excuse me, can u explain how can i use your fix in OpenCV 4.2.0 ?

@dkurt
Copy link
Copy Markdown
Member

dkurt commented Mar 16, 2020

@BadMachine, you need to build OpenCV from source. The fix has been merged after OpenCV 4.2.0

@BadMachine
Copy link
Copy Markdown

BadMachine commented Mar 16, 2020

@dkurt Thanks for your answer!
But source code now build with errors...

image

PS: I know it should be in issue section, just for u know

@dkurt
Copy link
Copy Markdown
Member

dkurt commented Mar 16, 2020

@BadMachine, Please do not use closed PRs for bug reports - open a new issue including build flags and error logs (but not a screenshot).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants