Reshape layer of keras (tensorflow) is interpreted incorrectly by readNetFromTensorflow() in dnn #19489
Description
System information (version)
- OpenCV => 4.5.1 (tested on 4.4.0 as well)
- Operating System / Platform => Ubuntu 20.04
- Compiler => Pre-built library (for OpenCV 4.5.1) and gcc (for OpenCV 4.4.0)
Detailed description
I tried to use a deep learning model trained on keras (tensorflow) on OpenCV via dnn. The output of dnn was very different from that of keras, so I started debugging. It looks like the outputs are different because the Reshape layer of keras is interpreted incorrectly in OpenCV dnn. If I remove the Reshape layer the model works just fine. I searched as much as I can; I found some similar issues but I think this exact bug is not mentioned before.
Steps to reproduce
The code below should reproduce the error. You can also use this .zip file, which also contains the test.jpg file that's used below. As you can see, if you set the add_reshape_layer variable to False, the model will not contain the reshape layer and will work just fine -- the output of keras will match that of OpenCV dnn.
import tensorflow.keras as keras
import tensorflow as tf
from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2
import numpy as np
import cv2
# If you turn this off the code will work fine -- the outputs of keras and OpenCV dnn will match
add_reshape_layer = True
# This is the simplified model that I constructed for debugging
model_simple = keras.models.Sequential([keras.layers.Conv2D(64, kernel_size=7, activation='relu', strides=(2,2), padding="same"),
keras.layers.BatchNormalization(),
keras.layers.Conv2D(64, kernel_size=3, activation='relu', strides=(2,2), padding="same"),
keras.layers.BatchNormalization(),
keras.layers.Conv2D(32, kernel_size=3, activation='relu', strides=(2,2), padding="same", use_bias=False),
keras.layers.BatchNormalization(),
keras.layers.Conv2D(12, kernel_size=3, activation='relu', padding="same", use_bias=False),
keras.layers.BatchNormalization(),
keras.layers.Conv2D(2, kernel_size=3, activation='relu', padding="same", use_bias=False),
keras.layers.BatchNormalization(),
keras.layers.Flatten(),
keras.layers.Dense(16*16*6, activation='relu', use_bias=False)
])
if add_reshape_layer:
model_simple.add(keras.layers.Reshape((16,16,6), input_shape=(16*16*6, )))
# Building and freezing model
model_simple.build(input_shape=(None,256,256,3))
model_simple.trainable = False
model_simple.save('SIMPLE_MODEL')
loaded = tf.saved_model.load('SIMPLE_MODEL')
infer = loaded.signatures['serving_default']
f = tf.function(infer).get_concrete_function(conv2d_input=tf.TensorSpec(shape=[1, 256, 256, 3], dtype=tf.float32))
f2 = convert_variables_to_constants_v2(f)
graph_def = f2.graph.as_graph_def()
# Export frozen graph
with tf.io.gfile.GFile('simple_model.pb', 'wb') as f:
f.write(graph_def.SerializeToString())
# Try on test image
im = cv2.imread('test.jpg')/255.0
rim = tf.reshape(im, (-1, 256, 256, 3))
# y_keras: output for test image from keras interface
y_keras = model_simple.predict(rim)
model_dnn = cv2.dnn.readNetFromTensorflow('./simple_model.pb')
model_dnn.setInput(cv2.dnn.blobFromImage(np.float32(im)))
y_dnn = model_dnn.forward()
if add_reshape_layer:
print(np.max(np.abs(y_keras[0,:,:,0]-y_dnn[0,0,:,:]))/np.max(np.abs(y_keras))) # prints large difference
else:
print(np.max(np.abs(y_keras-y_dnn))/np.max(np.abs(y_keras))) # prints very small difference -- practically zero