1e-15 in that post was arbitrary. Why not simply enable FTZ in the hardware before convolving?

already_enabled = is_ftz_enabled()
enable_ftz()

// do convolution

if (!already_enabled)
    disable_ftz();

It's possible that there are no denormals in the weights (can check using std::fpclassify) but are being generated during convolution. This can still happen with clipped weights if some input to the convolution layer happens to be really small. So if an attempt is being made to avoid denormals, why not just ignore them completely instead of allowing some amount of denormals?

@YashasSamaga, that's good point but we need more experiment with it. It's hard to determine // do convolution because we have multiple CPU backends which receive fused weights as parameters (TEngine, Intel Inference Engine).

Update: there is FTZ right in the MKL-DNN source code. Ok, I'll do as you recommended, thanks!

just FYI, I gave this patch a try and my inference time is 10% slower with the patch (from 92ms to 102ms)
Before this patch, I also tried compiling with FAST_MATH and inference time was unchanged (92ms)

@JulienMaille, I just updated PR. Have you tried version with manual flushing or with intrinsics? Can you provide a reproducer?

I tried this:

            // Flush to zero (FTZ) denormal weights: https://github.com/opencv/opencv/issues/17259
            Mat mask = abs(weightsMat) <= 1e-15f;
            weightsMat.setTo(0.0f, mask & (weightsMat > 0.0f));
            weightsMat.setTo(-0.0f, mask & (weightsMat < 0.0f));

Will try the new one and report

Same 10% slow down. Do you want an onnx to reproduce?

@JulienMaille, can you share efficiency measurement approach? Here an example with min, median, mean estimation with your model (several runs):

import numpy as np
import cv2 as cv
import time
print(cv.__file__)

net = cv.dnn.readNet('model.onnx')
net.setPreferableBackend(cv.dnn.DNN_BACKEND_OPENCV)
inp = np.random.standard_normal([1, 1, 256, 256]).astype(np.float32)
net.setInput(inp)
net.forward()

speeds = []
for i in range(1000):
    start = time.time()
    net.forward()
    speeds.append((time.time() - start) * 1000)
print('%.2fms|%.2fms|%.2fms|%.2f' % (np.min(speeds), np.median(speeds), np.mean(speeds), np.std(speeds)))

I can do that, the numbers I gave were the average on 20 runs, with the first run discarded.
I'm working with the c++ api, is there a way to access "nightly" python builds with openvino? That would make it easier for testing purpose.

@JulienMaille, you don't need OpenVINO - the changes in this patch won't affect it - it's for default implementation, DNN_BACKEND_OPENCV.

Seems like the difference was due to the cpu being busier when I ran the FTZ benchs.
Even when running 100 loops I can get a 10ms difference on avg/median.
Best run for both baseline and FTZ give the same results

avg: 90.7, min: 88, median: 91, std: 2.75

Uh, sorry. Ok so for reference (100 loops):

INFERENCE_ENGINE    avg:  90.7, min:  88, median:  91, std: 2.75
BACKEND_OPENCV base avg: 137.5, min: 132, median: 137, std: 5.59
                    avg: 137.4, min: 135, median: 138, std: 5.37
                    avg: 137.2, min: 136, median: 137, std: 5.50
BACKEND_OPENCV FTZ  avg: 139.7, min: 134, median: 138, std: 7.32
                    avg: 137.9, min: 132, median: 137, std: 5.28
                    avg: 137.7, min: 134, median: 138, std: 5.41

@dkurt Do you have final decision on the patch?

@asmorkalov , yes, it's a final version.

@alalek, you mentioned that there is also a way to apply FTZ optimization for ARM CPUs, shall I add it here?

1. What is the scope of the DAZ and FTZ modes? Do they have to be set for each thread?

2. What if an exception is thrown during convolution or an early return is made (OCL is doing it)? The DAZ and FTZ modes won't be reset (and hence alter the modes in the end-users' thread that invoked net.forward()?).

An RAII based solution would solve problem 2. The RAII objects will automatically reset the FTZ and DAZ modes after an exception or before a return. It's safer and future-proof.

The following posts suggest that the FTZ and DAZ modes have to be set for each thread:

• https://stackoverflow.com/questions/11671430/flushing-denormalised-numbers-to-zero
• https://stackoverflow.com/questions/4948057/scope-of-mxcsr-control-register

I don't know how ParallelLoopBody works but if it's reusing worker threads from a global thread pool, the threads might have their own FTZ and DAZ modes.