Can't use fine-tuned MobileNetV3 from DNN #20540
Description
System information (version)
- OpenCV => OpenCV 4.5.3 (built from master, aaca498)
- Operating System / Platform => Windows 10 21H1
- Compiler => MSVC 14.29 (VS 2019 Version 16.10.4)
Detailed description
I trained MobileNetV3 (https://tfhub.dev/google/imagenet/mobilenet_v3_large_100_224/feature_vector/5) with fine-tuning (simple classifier with 2 classes). Then I converted it to ONNX using tf2onnx (https://github.com/onnx/tensorflow-onnx) - got model.onnx. I tried to use it from DNN (cv::dnn::readNetFromONNX) and got assert:
dnn\\src\\layers\\fully_connected_layer.cpp:181: error: (-215:Assertion failed) srcMat.dims == 2 && srcMat.cols == weight ...
And the output / output size looks wrong to me: 1 x 1280 x 1 x 2.
cv::dnn::readFromTensorflow for TF model doesn't work either. I froze TF model - got model.pb - and tried to run inference using DNN: no asserts, but got wrong output / output size and "dense" (result) output is missing from dumped model.
Some notes:
- Inference of model.onnx using ONNX Runtime works OK.
- Inference of model.pb using Tensorflow works OK.
- Inference of another not fine-tuned MobileNetV3 model using DNN works OK.
- OpenCV with changes from dnn : Fix convolution asymmetric padding bug in onnx importer #20394 must be used
Something in DNN is broken or I'm doing something wrong.
Steps to reproduce
Inference of model.onnx using OpenCV DNN
#include <opencv2/dnn.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/imgproc.hpp>
void test()
{
auto model = cv::dnn::readNetFromONNX("model.onnx");
model.setPreferableTarget(cv::dnn::DNN_TARGET_CPU);
const auto image = cv::imread("test2.png");
constexpr auto input_image_size = 224;
cv::Mat blob;
cv::resize(image, blob, cv::Size{ input_image_size, input_image_size });
blob.convertTo(blob, CV_32F, 1.0/255.0, -0.5);
blob = blob.reshape(1, {1, input_image_size, input_image_size, 3});
model.setInput(blob);
model.dumpToFile("dumped_model"); // OK - has "dense" output
const auto output = model.forward(); // asserts; output.size is 1 x 1280 x 1 x 2 - wrong?
}
Inference of model.pb using OpenCV DNN
#include <opencv2/dnn.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/imgproc.hpp>
void test()
{
auto model = cv::dnn::readNetFromTensorflow("model.pb");
model.setPreferableTarget(cv::dnn::DNN_TARGET_CPU);
const auto image = cv::imread("test2.png");
constexpr auto input_image_size = 224;
cv::Mat blob = cv::dnn::blobFromImage(image, 1.0/255.0, cv::Size{ input_image_size, input_image_size});
model.setInput(blob);
model.dumpToFile("dumped_model"); // ??? - "dense" output is missing
const auto output = model.forward(); // no asserts; output.size == 1 x 1280 - wrong?
}Loading of model.onnx using ONNX Runtime
#include <onnxruntime/core/session/onnxruntime_cxx_api.h>
#include <iostream>
template<typename T>
std::ostream& operator<<(std::ostream& os, const std::vector<T>& v)
{
os << "[";
for (int i = 0; i < v.size(); ++i)
{
os << v[i];
if (i != v.size() - 1)
{
os << ", ";
}
}
os << "]";
return os;
}
void test()
{
Ort::Env env;
Ort::Session session{env, L"model.onnx", Ort::SessionOptions{nullptr}};
Ort::AllocatorWithDefaultOptions allocator;
size_t numInputNodes = session.GetInputCount();
size_t numOutputNodes = session.GetOutputCount();
std::cout << "Number of Input Nodes: " << numInputNodes << std::endl;
std::cout << "Number of Output Nodes: " << numOutputNodes << std::endl;
const char* inputName = session.GetInputName(0, allocator);
std::cout << "Input Name: " << inputName << std::endl;
Ort::TypeInfo inputTypeInfo = session.GetInputTypeInfo(0);
auto inputTensorInfo = inputTypeInfo.GetTensorTypeAndShapeInfo();
ONNXTensorElementDataType inputType = inputTensorInfo.GetElementType();
std::cout << "Input Type: " << inputType << std::endl;
std::vector<int64_t> inputDims = inputTensorInfo.GetShape();
std::cout << "Input Dimensions: " << inputDims << std::endl;
const char* outputName = session.GetOutputName(0, allocator);
std::cout << "Output Name: " << outputName << std::endl;
Ort::TypeInfo outputTypeInfo = session.GetOutputTypeInfo(0);
auto outputTensorInfo = outputTypeInfo.GetTensorTypeAndShapeInfo();
ONNXTensorElementDataType outputType = outputTensorInfo.GetElementType();
std::cout << "Output Type: " << outputType << std::endl;
std::vector<int64_t> outputDims = outputTensorInfo.GetShape();
std::cout << "Output Dimensions: " << outputDims << std::endl;
}
/*
Number of Input Nodes: 1
Number of Output Nodes: 1
Input Name: input_1
Input Type: 1
Input Dimensions: [-1, 224, 224, 3]
Output Name: dense
Output Type: 1
Output Dimensions: [-1, 2]
*/
Issue submission checklist
- I report the issue, it's not a question
- I checked the problem with documentation, FAQ, open issues,
forum.opencv.org, Stack Overflow, etc and have not found solution - I updated to latest OpenCV version and the issue is still there
- There is reproducer code and related data files: videos, images, onnx, etc