Looks like this line clears whole "gray" to zero, so result from lines above is lost.

It is better to avoid using drawing functions (they are not optimized):

gray(Rect(0, 0, gray.cols, gray.rows)).setTo(Scalar(0));

"Looks like this line clears whole "gray" to zero" No, only 1 pixel border set to zero.

alternative to drawing function maybe like

gray.col(0).setTo(Scalar(0));
gray.col(gray.cols - 1).setTo(Scalar(0));
gray.row(0).setTo(Scalar(0));
gray.row(gray.rows - 1).setTo(Scalar(0));

rectangle is faster according the code below

1.9021sec
0.855161sec

Mat gray = Mat(1000, 1000, CV_8UC1, Scalar(255));

TickMeter tm;
tm.start();
for (int i = 0; i < 100000; i++)
{
gray.col(0).setTo(Scalar(0));
gray.col(gray.cols - 1).setTo(Scalar(0));
gray.row(0).setTo(Scalar(0));
gray.row(gray.rows - 1).setTo(Scalar(0));

}
tm.stop();
cout << tm << endl;

tm.reset();
tm.start();
for (int i = 0; i < 100000; i++)
{
rectangle(gray, Rect(0, 0, gray.cols, gray.rows), Scalar(0), 1);
}
tm.stop();
cout << tm << endl;

Right, my bad.

4 setTo() calls are good here.

Alternative way (but probably slower):

Mat gray_inner = gray(Rect(1, 1, gray.cols - 2, gray.rows - 2));
copyMakeBorder(gray_inner, gray, 1, 1, 1, 1, BORDER_ISOLATED | BORDER_CONSTANT, Scalar::all(0));