Skip to content

LSTM layer with batch_firs=True fails #23602

@Abdurrahheem

Description

@Abdurrahheem

System Information

OpenCV version: 4.6.0
Operating System / Platform: Ubuntu 20.04
Compiler & compiler version: GCC 7.5.0
Python version: 3.8.10

Detailed description

When I convert LSTM torch model with batch_first=True and run it in OpenCV it fails with the following error:

OpenCV(4.7.0-dev) Error: Assertion failed ((int)_numAxes == inputs[0].size()) in getMemoryShapes, file /home/abduragim/projects/opencv_proj/opencv/modules/dnn/src/layers/permute_layer.cpp, line 163
[ERROR:0@0.080] global net_impl.cpp:1164 getLayerShapesRecursively OPENCV/DNN: [Permute]:(onnx_node!/rnns/Transpose): getMemoryShapes() throws exception. inputs=1 outputs=0/1 blobs=0
[ERROR:0@0.080] global net_impl.cpp:1167 getLayerShapesRecursively     input[0] = [ 5 2 3 1 ]
[ERROR:0@0.080] global net_impl.cpp:1177 getLayerShapesRecursively Exception message: OpenCV(4.7.0-dev) /home/abduragim/projects/opencv_proj/opencv/modules/dnn/src/layers/permute_layer.cpp:163: error: (-215:Assertion failed) (int)_numAxes == inputs[0].size() in function 'getMemoryShapes'

Traceback (most recent call last):
  File "test_lstm.py", line 190, in <module>
    out_opencv = net.forward()
cv2.error: OpenCV(4.7.0-dev) /home/abduragim/projects/opencv_proj/opencv/modules/dnn/src/layers/permute_layer.cpp:163: error: (-215:Assertion failed) (int)_numAxes == inputs[0].size() in function 'getMemoryShapes'

Steps to reproduce

import torch
import torch.nn as nn
import onnxruntime
import numpy as np
import cv2
import random

# Set random seeds
seed = 42
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.set_printoptions(precision=15)


class LayerLSTM(nn.Module):
    def __init__(self, features, hidden_size, layers=1, batch_first=False):
        super(LayerLSTM, self).__init__()
        self.rnns = nn.LSTM(
            input_size=features,
            hidden_size=hidden_size,
            num_layers=layers,
            batch_first=batch_first
        )
        self.set_weights_to_ones()

    def forward(self, x, hx, cx):
        x, (hx, cx) = self.rnns(x, (hx,cx))
        return x, (hx, cx)

    def set_weights_to_ones(self):
        with torch.no_grad():
            for name, param in self.rnns.named_parameters():
                if 'weight' in name or 'bias' in name:
                    param.fill_(1.0)

if __name__ == "__main__":

    features    = 3
    hidden_size = 7
    batch_size  = 5
    seq_len     = 2
    layout      = True

    model = LayerLSTM(features, hidden_size, batch_first=layout)

    with torch.no_grad():
        if layout:
            x  = torch.ones(batch_size, seq_len, features)
        else:
            x  = torch.ones(seq_len, batch_size, features)

        hx = torch.ones(1, batch_size, hidden_size)
        cx = torch.ones(1, batch_size, hidden_size)

        print(x.shape, hx.shape, cx.shape, sep='\n')
        out, (hn, cn) = model(x, hx, cx)
        
    torch.onnx.export(
       model,
       (x, hx, cx),
       './sample_lstm.onnx',
       verbose=True,
       input_names=['x', 'hx', 'cx'],
       output_names=['output']
    )
    
    net = cv2.dnn.readNetFromONNX('./sample_lstm.onnx')

    x_opencv  = x.numpy()[..., None]
    hx_opencv = hx.numpy()[..., None]
    cx_opencv = cx.numpy()[..., None]

    # Set input data
    net.setInput(x_opencv)
    net.setInput(hx_opencv, name="hx")
    net.setInput(cx_opencv, name="cx")
    
    out_opencv = net.forward()

Issue submission checklist

  • I report the issue, it's not a question
  • I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
  • I updated to the latest OpenCV version and the issue is still there
  • There is reproducer code and related data files (videos, images, onnx, etc)

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions