Skip to content

error: (-215:Assertion failed) Passed input shapes do not match with parsed input shapes! in function 'cv::dnn::LayerEinsumImpl::getMemoryShapes' #25077

@LaurentBerger

Description

@LaurentBerger

System Information

General configuration for OpenCV 4.9.0-dev =====================================
Version control: 4.9.0-189-g2d204243af

Extra modules:
Location (extra): C:/lib/opencv_contrib/modules
Version control (extra): 4.9.0-21-gd870648f

Platform:
Timestamp: 2024-02-22T10:48:57Z
Host: Windows 10.0.22631 AMD64
CMake: 3.26.1
CMake generator: Visual Studio 17 2022
CMake build tool: C:/Program Files/Microsoft Visual Studio/2022/Community/MSBuild/Current/Bin/amd64/MSBuil

Detailed description

Model is loaded. Error is thrown in inference process
I check result with numpy and onnx.
Opencv thrown an error :

[ERROR:0@6387.898] global net_impl.cpp:1165 cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively OPENCV/DNN: [Einsum]:(onnx_node_output_0!out1): getMemoryShapes() throws exception. inputs=3 outputs=0/1 blobs=0
[ERROR:0@6387.899] global net_impl.cpp:1168 cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively     input[0] = [ 1 300 8 32 ]
[ERROR:0@6387.899] global net_impl.cpp:1168 cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively     input[1] = [ 1 8 32 1 ]
[ERROR:0@6387.899] global net_impl.cpp:1168 cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively     input[2] = [ 1 300 ]
[ERROR:0@6387.899] global net_impl.cpp:1178 cv::dnn::dnn4_v20231225::Net::Impl::getLayerShapesRecursively Exception message: OpenCV(4.9.0-dev) C:\lib\opencv\modules\dnn\src\layers\einsum_layer.cpp:440: error: (-215:Assertion failed) Passed input shapes do not match with parsed input shapes! in function 'cv::dnn::LayerEinsumImpl::getMemoryShapes'

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 47, in <module>
cv2.error: OpenCV(4.9.0-dev) C:\lib\opencv\modules\dnn\src\layers\einsum_layer.cpp:440: error: (-215:Assertion failed) Passed input shapes do not match with parsed input shapes! in function 'cv::dnn::LayerEinsumImpl::getMemoryShapes'

Steps to reproduce

Model is generated in code :
image

Source

import numpy as np
import onnx
import onnxsim
import onnx.reference
import cv2 as cv



eqn = "nlhd, nhdv, nlh -> nlhv"
init_Q = np.ones(shape=(1, 300, 8, 32))
init_KV = np.ones(shape=(1, 8, 32, 1))
init_onnx = np.ones(shape=(1, 300, 8))
numpy_res=np.einsum(eqn,init_Q, init_KV, init_onnx)
out1 = onnx.helper.make_tensor_value_info('out1', onnx.TensorProto.FLOAT, [None, None, None, None])
inp_init_Q = onnx.helper.make_tensor_value_info('inp_init_Q', onnx.TensorProto.FLOAT, init_Q.shape)
inp_init_KV = onnx.helper.make_tensor_value_info('inp_init_KV', onnx.TensorProto.FLOAT, init_KV.shape)
inp_init_onnx = onnx.helper.make_tensor_value_info('inp_init_onnx', onnx.TensorProto.FLOAT, init_onnx.shape)
node1 = onnx.helper.make_node( "Einsum", inputs=["inp_init_Q","inp_init_KV", "inp_init_onnx"], outputs=["out1"], equation=eqn)
graph = onnx.helper.make_graph([node1], 'test_einsum',  [inp_init_Q, inp_init_KV, inp_init_onnx], [out1])
onnx_model = onnx.helper.make_model(graph)

feeds = {'inp_init_Q': init_Q, 'inp_init_KV': init_KV, 'inp_init_onnx': init_onnx}
sess = onnx.reference.ReferenceEvaluator(onnx_model)
res_onnx = sess.run(["out1"], feeds)
print("ONNX result")
print("Max diff onnx numpy " , abs(res_onnx[0]-numpy_res).max())
onnx_name = eqn.replace('>', '')
print("Writting model")
with open(onnx_name + ".onnx", "wb") as f:
    f.write(onnx_model.SerializeToString())
print("Reading model")

net = cv.dnn.readNet(onnx_name+'.onnx')
net.enableWinograd(True)

net.setInput(init_Q.astype(np.float32), 'inp_init_Q')
net.setInput(init_KV.astype(np.float32), 'inp_init_KV')
net.setInput(init_onnx.astype(np.float32), 'inp_init_onnx')
res_ocv = net.forward()


in c++ with debug

        vector<int> szInitQ = { 1, 300, 8, 32 };
        vector<int> szInitKV = { 1, 8, 32, 1 };
        vector<int> szInitOnnx = { 1, 300, 8 };
        Mat inpInitQ(szInitQ, CV_32FC1, Scalar::all(1));
        Mat inpInitKV(szInitKV, CV_32FC1, Scalar::all(1));
        Mat inpInitOnnx(szInitOnnx, CV_32FC1, Scalar::all(1));


        Net netEinsum = readNet("nlhd, nhdv, nlh - nlhv.onnx");
        netEinsum.setInput(inpInitQ, "inp_init_Q");
        netEinsum.setInput(inpInitKV, "inp_init_KV");
        netEinsum.setInput(inpInitOnnx, "inp_init_onnx");
        //netEinsum.setInput(b, "inp1");
        Mat c = netEinsum.forward();

and error is

[ INFO:0@3.599] global onnx_importer.cpp:806 cv::dnn::dnn4_v20231225::ONNXImporter::populateNet DNN/ONNX: loading ONNX v9 model produced by ''. Number of nodes = 1, initializers = 0, inputs = 3, outputs = 1
[ INFO:0@3.599] global onnx_importer.cpp:699 cv::dnn::dnn4_v20231225::ONNXImporter::parseOperatorSet DNN/ONNX: ONNX opset version = 19
[ INFO:0@3.601] global onnx_importer.cpp:977 cv::dnn::dnn4_v20231225::ONNXImporter::handleNode DNN/ONNX: processing node with 3 inputs and 1 outputs: [Einsum]:(onnx_node_output_0!out1) from domain='ai.onnx'
OpenCV(4.9.0-dev) Error: Unspecified error (> Real output can not be shaped in to _**requred**_ output (expected: 'reqProd == realProd'), where
>     'reqProd' is 2400
> must be equal to
>     'realProd' is 19200
) in void __cdecl cv::dnn::LayerEinsumImpl::forward(const class cv::debug_build_guard::_InputArray &,const class cv::debug_build_guard::_OutputArray &,const class cv::debug_build_guard::_OutputArray &), file C:\lib\opencv\modules\dnn\src\layers\einsum_layer.cpp, line 550

Issue submission checklist

  • I report the issue, it's not a question
  • I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
  • I updated to the latest OpenCV version and the issue is still there
  • There is reproducer code and related data files (videos, images, onnx, etc)

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions