System information (version)

• OpenCV => :4.2 and 4.3
• Operating System / Platform => :Windows 10 64 Bit
• Compiler => : VS 2017 and VS 2015
• GPU => : RTX 2080Ti × 2

Detailed description

When I test my model using OpenCV DNN modules on CUDA backend and OPENCV backend, I find that the inference speed of my model on GPU is much slower than that on CPU.

And my net is just like a Siamese Network, It has two inputs and four outputs. @YashasSamaga finds that one of inputs may makes the model initialize more than once. There may be some bug in initialization ligic.

Steps to reproduce

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues,
  answers.opencv.org, Stack Overflow, etc and have not found solution
• I updated to latest OpenCV version and the issue is still there

• There is reproducer code and related data files: videos, images, onnx, etc

code and .pb file.
https://drive.google.com/open?id=1Wsx4JOKvn6Xn2Rr7m0VX2HTXEvUBvWm3