Could you use one of the existing image in this repo, such as https://github.com/opencv/opencv_extra/blob/4.x/testdata/dnn/dog_orig_size.png? You can resize it to the target scale, run inference, then collect the corresponding output, so that the total size can be cut by half, which passes the limit check.

Could you use one of the existing image in this repo, such as https://github.com/opencv/opencv_extra/blob/4.x/testdata/dnn/dog_orig_size.png? You can resize it to the target scale, run inference, then collect the corresponding output, so that the total size can be cut by half, which passes the limit check.

Sorry for getting back late to this. Since it's an optical flow model, the input should be two images. I can use the same image twice for both inputs, but it wouldn't really make much sense. Another approach would be finding two lightweight images, but a question is what is the limit check?

@Aser-Abdelfatah, you may use images from https://github.com/opencv/opencv_extra/tree/4.x/testdata/gpu/opticalflow. Preprocess them in runtime.

@dkurt I didn't get the part about pre-processing in runtime. The testing code in opencv/modules/dnn/test/test_onnx_importer.cpp is essentially,

TEST_P(Test_ONNX_layers, RAFT) { testONNXModels("RAFT", npy, 0, 0, false, true, 2); }
So, the model expects two pre-processed inputs.
And of course, when I pre-process the images using a separate python code and convert the pre-processed numpy array to an npy file, the size of each input is the same, essentially 2.0 mega bytes.

And sorry for keeping getting back late to this. The time difference is crazy

And of course, when I pre-process the images using a separate python code and convert the pre-processed numpy array to an npy file

So you can do the same in C++ - imread two images and preprocess using OpenCV methods.

I'm struggling with the C++ implementation. Essentially, I can't find a way to export the 4-dimensional opencv Mat to npy file in C++.

You can do it in Python with OpenCV and NumPy.

import cv2 as cv
import numpy as np

img1 = cv.imread("/path/to/img1")
img2 = cv.imread("/path/to/img2")

import onnxruntime as ort
net = ort.InferenceSession("/path/to/raft/model")
output = net.run(["output_name"], {"input1_name": img1, "input2_name": img2})
np.save("output_file_name.npy", output[0])

Maybe you need to debug a little bit to run the code above, but it should be mostly correct.

I did it before. The problem is that the resulting npy file is also 2 mega bytes

2MB is fine. IIRC, the limit is 3MB.

2MB is fine. IIRC, the limit is 3MB.

@fengyuentau, this is not fine. The preprocessing might be done in C++ without any issues. Such 2MB will be added to git history forever.

I'm struggling with the C++ implementation.

@Aser-Abdelfatah, share Python implementation for preprocessing. Show how .npy are saved so we can propose an alternative C++ impl.

How is it possible to test model accuracy if the output reference is not preserved in this repository?

What I am saying is we don't need to check the exact correct results as what common users expect, we don't need to perform preprocessings to make results correct and meaningful. Just put two input images that has the input scale that the model requires, save outputs as a npy file, put the output file here, and thats it.

There is no problem with output - it takes just 21Kb (see testdata/dnn/onnx/data/output_RAFT.npy). This is a reference for test. I propose the following:

1. Use two images in Python and native RAFT to get output reference. Keep only output in .npy file
2. Use the same images in C++ test with the same preprocessing which was done in Python.

No need to add preprocessed input .npy to this PR.

Sorry, seems like there is a misunderstanding. Proposed solution by @fengyuentau at #1082 (comment) is correct. I don't understand what is testdata/dnn/onnx/data/output_RAFT.npy and why it's 21Kb but not 2MB as @Aser-Abdelfatah mentioned. Let's upload actual files first.

import onnxruntime as ort
import onnx
from imread_from_url import imread_from_url
import cv2 as cv


img1 = imread_from_url("https://github.com/opencv/opencv_extra/blob/4.x/testdata/gpu/opticalflow/frame0.png?raw=true")
img2 = imread_from_url("https://github.com/opencv/opencv_extra/blob/4.x/testdata/gpu/opticalflow/frame1.png?raw=true")
input_width = 480
input_height = 360

image1 = cv.cvtColor(img1, cv.COLOR_BGR2RGB)
img_input1 = cv.resize(image1, (input_width, input_height))
img_input1 = img_input1.transpose(2, 0, 1)
img_input1 = img_input1[np.newaxis,:,:,:]
img_input1 = img_input1.astype(np.float32)

image2 = cv.cvtColor(img2, cv.COLOR_BGR2RGB)
img_input2 = cv.resize(image2, (input_width, input_height))
img_input2 = img_input2.transpose(2, 0, 1)
img_input2 = img_input2[np.newaxis,:,:,:]
img_input2 = img_input2.astype(np.float32)


model = ort.InferenceSession("path\\to\\model\\optical_flow_estimation_raft_2023aug.onnx", providers=['CUDAExecutionProvider', 'CPUExecutionProvider']) // the model is here https://github.com/Aser-Abdelfatah/opencv_zoo/blob/gsoc_RAFT_Model_ONNX/models/optical_flow_estimation_raft/optical_flow_estimation_raft_2023aug.onnx


model_inputs = model.get_inputs()
input_names = [model_inputs[i].name for i in range(len(model_inputs))]
model_outputs = model.get_outputs()
output_names = [model_outputs[i].name for i in range(len(model_outputs))]

y = model.run(output_names, {input_names[0]: img_input1, input_names[1]: img_input2})

np.save("input_RAFT_0.npy", img_input1)
np.save("input_RAFT_1.npy", img_input2)
np.save("output_RAFT.npy", y[0])

This is the python code for pre-processing, inference, and saving output. The input files are numpy float arrays of dimension (1, 3, 360, 480). The output is an optical flow map not an image.

@Aser-Abdelfatah, thanks! This is C++ function you need to use:

Mat image1 = cv::imread("frame0.png");
Mat input1 = cv::dnn::blobFromImage(image1, 1.0, Size(input_width, input_height), Scalar(), true);