Hi @Haosonn, thanks for your contribution!

Currently, our implementation (or all layers supporting Vulkan backend) runs pretty slow on discrete GPUs basically due to IO cost in function copyToHost.

Yes. Previously patch of vulkan, I just focused on the Integrated graphics. Our Vulkan backend still needs a lot of optimization. In my opinion, the first priority is supporting more layers, so that we could reduce the number of calling copyToHost. And the optimized of discrete GPUs, could be done at lower priority. There are two reasons for this: 1. we have CUDA backend for discrete GPUs, 2. fast discrete GPUs need full VkImage pipeline, more complicated than VkBuffer.

prevent copyToHost in middle layers during forwarding, (i.e keep data in GPU memory)

It's hard to do so, we can not predict if the next layer of NaryEltwiseLayer was supported by Vulkan. Some fast transfer strategy like MNN's vulkan, they have two different implementations: VkBuffer and VkImage. And the VkImage is much faster on data transfering of GPU-CPU.

Several tests failed:

• objdetect:

   [RUN      ] Objdetect_face_detection.regression
  

• video:

  [  FAILED  ] NanoTrack.accuracy_NanoTrack_V1
  [  FAILED  ] NanoTrack.accuracy_NanoTrack_V2
  

Also see https://pullrequest.opencv.org/buildbot/builders/precommit_linux64/builds/105934/steps/test_objdetect/logs/stdio, which looks like memory issues.

@Haosonn @fengyuentau please rebase and fix conflicts.

@zihaomu @fengyuentau Could you take a look again?

This patch cause FP16 test failures: #24954

I see performance degradation for this test case with 1/2/4 threads (no threading in implementation anyway) on 12700K:

To reviewers: PRs with optimization or other non-trivial implementation changes should have attached performance reports.

I see performance degradation for this test case with 1/2/4 threads (no threading in implementation anyway) on 12700K:

Name of Test base patch x-factor
NCHW_NCHW_pow::Layer_NaryEltwise::OCV/CPU 157.908 169.737 0.93
To reviewers: PRs with optimization or other non-trivial implementation changes should have attached performance reports.

Pow is not supported yet in Vulkan backend. So I guess something else happened?

There is regression on CPU, not Vulkan.

It looks weirder to me that this patch did very limited changes on the CPU implementation but yet affected the CPU performance, specifically Pow only. Let me investigate it.

Update: Oh, I see, use --perf_min_samples=100. I thought it was some kind of environment variable.

@opencv-alalek Do you know how to force opencv_perf_* running 100 samples? I found they can run 10 to 100 samples, which may lead to some mistakes.

@fengyuentau , there is TEST_CYCLE_N but it marked as deprecated (but it works for individual tests):

TEST_CYCLE_N(100)
{
…
}

Or you may use --perf_min_samples=100 --perf_force_samples=100:

--perf_min_samples (value:10)
    minimal required numer of samples
--perf_force_samples (value:100)
    force set maximum number of samples for all tests

Sorry, I missed the thing that you already found --perf_min_samples