DNN: load fp16 ONNX model as fp32 by zihaomu · Pull Request #22337 · opencv/opencv

zihaomu · 2022-08-03T03:42:44Z

How to convert ONNX fP32 model to FP16 model?

from onnx import load_model, save_model
from onnxmltools.utils import float16_converter

dir = "MODEL_DIR_PATH"
onnx_model = load_model(os.path.join(dir, "resnet50v1.onnx"))
trans_model = float16_converter.convert_float_to_float16(onnx_model, keep_io_types=False)
save_model(trans_model, os.path.join(dir, "resnet50v1_FP16.onnx"))

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

vpisarev · 2022-08-03T06:24:25Z

the PR is important and is ready to be merged

rogday

I think we should log this behavior so that users are not surprised that OpenCV DNN takes twice as much RAM as other frameworks, otherwise looks good.

modules/dnn/src/onnx/onnx_graph_simplifier.cpp

fengyuentau

👍

zihaomu · 2022-08-09T08:23:36Z

Hi @alalek and @asmorkalov, can you please check if the update can fix the data alignment issue?

asmorkalov · 2022-08-11T10:41:06Z

@zihaomu I checked other tickets and OpenCV patches that are alignment related and found general solution for it that is managed by macro CV_STRONG_ALIGNMENT. See

opencv/modules/core/include/opencv2/core/cvdef.h

Line 351 in bb71cb2

    
           #if !defined(CV_STRONG_ALIGNMENT) && defined(__arm__) && !(defined(__aarch64__) || defined(_M_ARM64))

.

I propose to use it and rework the patch in the same way as it's done for int64 tensors in the same function (Macro check + AutoBuffer).
Also related issue: #16373

asmorkalov · 2022-08-11T10:41:58Z

Also it looks like fp64 branch suffers from the same problem, but it's not popular and the issue is invisible.

zihaomu · 2022-08-11T10:45:04Z

Great thanks! @asmorkalov I will follow your suggestion and re-implement it with CV_STRONG_ALIGNMENT .

asmorkalov · 2022-08-15T08:28:25Z

[  FAILED  ] Test_ONNX_nets.MobileNet_v2_FP16/0, where GetParam() = OCV/OCL
[  FAILED  ] Test_ONNX_nets.MobileNet_v2_FP16/1, where GetParam() = OCV/OCL_FP16
[  FAILED  ] Test_ONNX_nets.MobileNet_v2_FP16/2, where GetParam() = OCV/CPU

zihaomu · 2022-08-15T08:47:46Z

Thanks for the reminder, @asmorkalov. there seems to be a depth bug on float16 and I'm trying to fix it.

zihaomu · 2022-08-20T03:53:26Z

Hi @asenyaev, other CI works fine, only Win10 can not download the new added testing model from Github. Can you please check what's happening? Win10 CI error

asenyaev · 2022-08-22T06:26:54Z

@zihaomu sure, I'll check it.

asenyaev · 2022-08-22T08:53:23Z

@zihaomu, the model was downloaded manually on Windows machine and the pipeline for Windows passed.

zihaomu · 2022-08-22T11:35:41Z

Hi, ONNX_importer.cpp may encounter bugs when load float16 model.
When the format version of ONNX is lower than 7, dnn/misc/opencv_onnx.pb.cc will parse float 16 tensor into int32, which will cause the weight to be all 0.
Since the dnn/misc/opencv_onnx.pb.cc is generated by opencv-onnx.proto which was taken from https://github.com/onnx/onnx/blob/main/onnx/onnx.proto, I think this bug is not our responsibility.

In my test, this path works fine with model format version >= 6. So, I think we can merge this first, and then I will discuss the specific solution with the ONNX team.

Some example:
mobilenetv2 with ONNX format V3 (No)
mobilenetv2 with ONNX format V7 (Yes)

alalek

LGTM 👍

rogday

👍

fengyuentau · 2022-08-24T03:25:47Z

modules/dnn/src/onnx/onnx_graph_simplifier.cpp

        {
-            const ::google::protobuf::RepeatedField<float> field = tensor_proto.float_data();
-            Mat(sizes, CV_32FC1, (void*)field.data()).copyTo(blob);
+            const int offset = isLittleEndianCPU() ? 0 : 1;


IIRC, we have this check in CMake scripts. Can we use a precomp macro here?

Please reuse WORDS_BIGENDIAN macro (from cvconfig.h)

fengyuentau · 2022-08-24T03:27:20Z

modules/dnn/test/test_onnx_importer.cpp

 {
    return findDataFile(std::string("dnn/onnx/") + filename, required);
 }
+        void printShape(InputArray blob_)


unnecessary commit

Thx! Fixed, wrong commit.

modules/dnn/test/test_onnx_importer.cpp

zihaomu · 2022-08-24T03:42:18Z

Hi, with the help of ONNX teams, we can load all the ONNX float16 models now!
ONNX saves float 16 data to two different way: int32_t and raw_data.
ONNX convert float to float16

zihaomu added category: dnn pr: needs test New functionality requires minimal tests set labels Aug 3, 2022

vpisarev self-requested a review August 3, 2022 06:09

vpisarev approved these changes Aug 3, 2022

View reviewed changes

asmorkalov requested a review from rogday August 3, 2022 07:18

rogday reviewed Aug 3, 2022

View reviewed changes

zihaomu requested review from rogday August 5, 2022 05:41

rogday reviewed Aug 5, 2022

View reviewed changes

modules/dnn/src/onnx/onnx_graph_simplifier.cpp Outdated Show resolved Hide resolved

alalek reviewed Aug 5, 2022

View reviewed changes

modules/dnn/src/onnx/onnx_graph_simplifier.cpp Outdated Show resolved Hide resolved

alalek reviewed Aug 5, 2022

View reviewed changes

modules/dnn/src/onnx/onnx_graph_simplifier.cpp Show resolved Hide resolved

fengyuentau self-requested a review August 5, 2022 08:59

zihaomu force-pushed the load_ONNX_fp16_as_fp32 branch from 838b4a4 to 3d0bea5 Compare August 5, 2022 09:30

fengyuentau approved these changes Aug 6, 2022

View reviewed changes

zihaomu requested a review from alalek August 9, 2022 08:23

zihaomu force-pushed the load_ONNX_fp16_as_fp32 branch from fc98f49 to 4901de2 Compare August 12, 2022 07:27

asmorkalov approved these changes Aug 12, 2022

View reviewed changes

asmorkalov assigned vpisarev Aug 12, 2022

zihaomu mentioned this pull request Aug 12, 2022

DNN: add MobilenetV2 FP16 download link opencv/opencv_extra#993

Merged

zihaomu added test and removed pr: needs test New functionality requires minimal tests set labels Aug 15, 2022

zihaomu force-pushed the load_ONNX_fp16_as_fp32 branch from 52f5d5b to 0b155fa Compare August 20, 2022 23:56

zihaomu marked this pull request as draft August 21, 2022 04:27

zihaomu marked this pull request as ready for review August 22, 2022 11:23

zihaomu force-pushed the load_ONNX_fp16_as_fp32 branch 2 times, most recently from 8325b2b to 6204381 Compare August 22, 2022 11:26

alalek approved these changes Aug 22, 2022

View reviewed changes

zihaomu mentioned this pull request Aug 22, 2022

Can I convert the model format to specific version? onnx/onnx#4460

Closed

zihaomu requested review from asmorkalov and vpisarev August 23, 2022 01:44

asmorkalov approved these changes Aug 23, 2022

View reviewed changes

rogday approved these changes Aug 23, 2022

View reviewed changes

fengyuentau reviewed Aug 24, 2022

View reviewed changes

zihaomu force-pushed the load_ONNX_fp16_as_fp32 branch from cb62e4f to a970e58 Compare August 24, 2022 03:31

load fp16 as fp32 and align fp16 and double in onnx_graph_simplifie

7eaec9d

zihaomu force-pushed the load_ONNX_fp16_as_fp32 branch from a970e58 to 7eaec9d Compare August 26, 2022 02:05

vpisarev approved these changes Aug 26, 2022

View reviewed changes

reuse WORDS_BIGENDIAN.

9638e34

zihaomu force-pushed the load_ONNX_fp16_as_fp32 branch from 9da9699 to 9638e34 Compare August 26, 2022 23:43

asmorkalov merged commit 2619099 into opencv:4.x Aug 29, 2022

zihaomu mentioned this pull request Sep 8, 2022

crnnFP16 opencv/opencv_zoo#79

Merged

zihaomu mentioned this pull request Oct 10, 2022

opencv dnn unsupported data type float16 #18735

Closed

cocoa-xu mentioned this pull request Dec 17, 2022

text recogintion error cocoa-xu/evision#144

Closed

alalek mentioned this pull request Jan 8, 2023

(5.x) Merge 4.x #23113

Merged

asmorkalov added this to the 4.7.0 milestone Jan 23, 2023

zihaomu mentioned this pull request Aug 8, 2023

OpenCV4.8.0 dnn can not load fp16 onnx model which converts from onnxmltools #24110

Open

4 tasks

Uh oh!

Conversation

zihaomu commented Aug 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to convert ONNX fP32 model to FP16 model?

Pull Request Readiness Checklist

Uh oh!

vpisarev commented Aug 3, 2022

Uh oh!

rogday left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fengyuentau left a comment

Choose a reason for hiding this comment

Uh oh!

zihaomu commented Aug 9, 2022

Uh oh!

asmorkalov commented Aug 11, 2022

Uh oh!

asmorkalov commented Aug 11, 2022

Uh oh!

zihaomu commented Aug 11, 2022

Uh oh!

asmorkalov commented Aug 15, 2022

Uh oh!

zihaomu commented Aug 15, 2022

Uh oh!

zihaomu commented Aug 20, 2022

Uh oh!

asenyaev commented Aug 22, 2022

Uh oh!

asenyaev commented Aug 22, 2022

Uh oh!

zihaomu commented Aug 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

rogday left a comment

Choose a reason for hiding this comment

Uh oh!

fengyuentau Aug 24, 2022

Choose a reason for hiding this comment

Uh oh!

alalek Aug 26, 2022

Choose a reason for hiding this comment

Uh oh!

fengyuentau Aug 24, 2022

Choose a reason for hiding this comment

Uh oh!

zihaomu Aug 24, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zihaomu commented Aug 24, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zihaomu commented Aug 3, 2022 •

edited

Loading

zihaomu commented Aug 22, 2022 •

edited

Loading