ONNX does not work for BMM function #25762
Description
System Information
OpenCV python version: 4.10.0.82
PyTorch version: 2.0.0+cu117
Operating System / Platform: Ubuntu 22.04
Python version: 3.10.6
Detailed description
torch.bmm does not work when exported to ONNX and imported into OpenCV, where it complains about shape mismatch.
I test with this by creating a minimal model with two inputs
- A: Batch of B number of C x C matrices
- V: Batch of B number of C x N matrices
BMM works such that bmm(A,V) should multiply every matrix in A with every matrix in V independently along batch dimension, but it seems to try to multiply across the batch dimension (I set the B, C and N dimensions to 3, 5 and 7 to make it easy to debug).
Output:
Output shape from PyTorch: torch.Size([3, 5, 7])
============= Diagnostic Run torch.onnx.export version 2.0.0+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================
[ERROR:0@0.053] global net_impl.cpp:1162 getLayerShapesRecursively OPENCV/DNN: [MatMul]:(onnx_node!/MatMul): getMemoryShapes() throws exception. inputs=2 outputs=0/1 blobs=0
[ERROR:0@0.053] global net_impl.cpp:1168 getLayerShapesRecursively input[0] = [ 3 5 ]
[ERROR:0@0.053] global net_impl.cpp:1168 getLayerShapesRecursively input[1] = [ 3 5 ]
[ERROR:0@0.053] global net_impl.cpp:1178 getLayerShapesRecursively Exception message: OpenCV(4.10.0) /io/opencv/modules/dnn/src/layers/matmul_layer.cpp:72: error: (-2:Unspecified error) in function 'virtual bool cv::dnn::MatMulLayerImpl::getMemoryShapes(const std::vector<std::vector<int> >&, int, std::vector<std::vector<int> >&, std::vector<std::vector<int> >&) const'
> DNN/MatMul: invalid dimension K (expected: 'K_A == K_B'), where
> 'K_A' is 5
> must be equal to
> 'K_B' is 3
Traceback (most recent call last):
File "/home/cdeln/Downloads/opencv-onnx-reshape-bug(1).py", line 30, in <module>
net.forward(['output'])[0].shape
cv2.error: OpenCV(4.10.0) /io/opencv/modules/dnn/src/layers/matmul_layer.cpp:72: error: (-2:Unspecified error) in function 'virtual bool cv::dnn::MatMulLayerImpl::getMemoryShapes(const std::vector<std::vector<int> >&, int, std::vector<std::vector<int> >&, std::vector<std::vector<int> >&) const'
> DNN/MatMul: invalid dimension K (expected: 'K_A == K_B'), where
> 'K_A' is 5
> must be equal to
> 'K_B' is 3
Steps to reproduce
Minimal reproducing example:
import torch
import cv2 as cv
class BMM(torch.nn.Module):
def __init__(self):
super().__init__()
def forward(self, A, V):
return torch.bmm(A, V)
model = BMM()
B, C, N = 3, 5, 7
A = torch.zeros(B, C, C)
V = torch.zeros(B, C, N)
input_values = (A, V)
input_names = ('A', 'V')
print('Output shape from PyTorch:', model(*input_values).shape)
torch.onnx.export(model, input_values, '/tmp/model.onnx', input_names=input_names, output_names=['output'])
net = cv.dnn.readNetFromONNX('/tmp/model.onnx')
net.setInput(A.numpy(), 'A')
net.setInput(V.numpy(), 'V')
net.forward(['output'])[0].shapeIssue submission checklist
- I report the issue, it's not a question
- I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
- I updated to the latest OpenCV version and the issue is still there
- There is reproducer code and related data files (videos, images, onnx, etc)