Inpaint method does not handle top and left border masked pixels correctly #25389
Description
System Information
OpenCV version: 4.9.0 (also 3.2.0)
Operating System / Platform: Windows 10
Compiler & compiler version: MSVC 19.29.30148
Detailed description
When inpainting with either Navier Stokes method or Telea algorithm, the image is not inpainted corrctly wherever neighboring zero and nonzero inpaint mask pixels both touch the left- or topmost border.
The example below demonstrates this by applying an inpaint mask to a simple 5*5 pixel image.
The image is completely grey (value = 128) and the pixels to be inpainted are black (value = 0).
The inpaint mask is nonblack where the image is black.
Therefore, the expected result should be an uniform grey image.
Instead, it gives this:
(Note how Navier Stokes' method lacks symmetry: The last pixels in 2nd column and line differ from each other, whereas Telea's algorithm produces a result symmetrical with respect to the top left to bottom right diagonal.)
The problem can be worked around by just adding a 1px border of to be inpainted pixels (mask != 0).
(For visual clarity, I made the newly added mask pixels a darker shade of grey (value = 64), to be distinguishable from the previous mask. The function will inpaint any nonzero mask pixel, so this does not alter the result.)
(Note how the algorithms both have no problem with handling inpainted pixels at the very border of the image. The bug just surfaces, when to be inpainted pixels and not to be inpainted pixels are present at the same border, and only at the top or left border.)
The result is now correct.
Obviously, this workaround is by no means an acceptable solution, since it only covers up the underlying bug and is by itself a resource hog since adding borders allocates new data and increases memory fragmentation.
Similar bug reports are to be found here and here (context).
This bugfix seemed to address the issue at first glance, but was in the end inconsequential to it. @nicheng0019 seems to have had additional fix proposals in their issue, but the issue was closed without implementing a fix.
I lack the insight to determine the bug's cause or evaluate @arnaudbrejeon's fix or @nicheng0019 proposal for completeness or formal correctness.
However, I can testify that something changed between OpenCV 3.2.0:
and OpenCV 4.9.0:
.
(Note how Navier Stokes seems identical at first glance, but some pixels differ in the exact shade of grey.
E. g. the pixel in the 4th column, 2nd row has a value of 64 in OpenCV 3.2.0 and 48 in OpenCV 4.9.0.
Steps to reproduce
This code reproduces the images shown above and saves them as a file.
Lambdas are used to ensure the usage of exactly the same processing (inpaint) for each of the input and postprocessing (scaling, labeling, painting a pixel grid, layouting) for each of the output images.
#include "opencv2\imgproc.hpp"
#include "opencv2\photo.hpp"
#include "opencv2\imgcodecs.hpp"
#include <string>
int main()
{
// Original image data.
unsigned char data[25] = { 128, 0, 128, 0, 128,
0, 0, 0, 0, 0,
128, 0, 0, 0, 128,
0, 0, 0, 0, 0,
128, 0, 128, 0, 128
};
cv::Mat image(5, 5, CV_8UC1, data); // Original image to be inpainted.
cv::Mat mask = ~(image * 2); // Inpaint mask: inpaint, wherever the original image is at least 128.
// Lambda to enhance image readability of output.
auto f_adorn = [&](cv::Mat const& src, std::string const& label = "") -> cv::Mat {
static const int scale = 30;
cv::Mat dst;
cv::resize(src, dst, cv::Size(src.cols * scale, src.rows * scale), 0.0, 0.0, cv::INTER_NEAREST);
cv::hconcat(dst({ 0, 0, 1, dst.rows }), dst, dst);
cv::vconcat(dst({ 0, 0, dst.cols, 1 }), dst, dst);
// Draw a grid.
for (int i = 0; i <= src.cols; ++i) {
cv::line(dst, { scale * i, 0 }, { scale * i, scale * src.rows - 1 }, 196);
}
for (int j = 0; j <= src.rows; ++j) {
cv::line(dst, { 0, scale * j }, { scale * src.cols - 1, scale * j }, 196);
}
if (!label.empty()) {
static const int font = cv::FONT_HERSHEY_PLAIN;
static const int font_thickness = 1;
static const double font_scale = 1.0;
int baseline;
cv::Size text_size = cv::getTextSize(label, font, font_scale, font_thickness, &baseline);
cv::Point orig((dst.cols - text_size.width) / 2, (scale + text_size.height) / 2);
cv::copyMakeBorder(dst, dst, scale, 5, 5, 5, cv::BORDER_CONSTANT, 32);
cv::putText(dst, label, orig, font, font_scale, 255, font_thickness);
}
return dst;
};
// Labda to generate human readable output of original image, inpaint mask, Navier Stokes and Telea inpainting.
// Captures current variables "image" and "mask" and derives output from there.
auto f_generate_output = [&](std::string const& file_name) {
cv::Mat inp_ns, inp_te; // Inpainted images
cv::inpaint(image, mask, inp_ns, 1, cv::INPAINT_NS);
cv::inpaint(image, mask, inp_te, 1, cv::INPAINT_TELEA);
cv::Mat tmp, out;
cv::hconcat(f_adorn(image, "Orig. image"), f_adorn(mask, "Inpaint mask"), out);
cv::hconcat(f_adorn(inp_ns, "Navier Stokes"), f_adorn(inp_te, "Telea"), tmp);
cv::vconcat(out, tmp, out);
cv::imwrite(file_name, out);
};
// Inpaint and make pretty output.
f_generate_output("OpenCV_Inpaint_Raw.png");
// Add 1px border with unkonown color (= "to be inpainted") left and top.
// Mask color intentionally darker than the one already used in the mask to be visually distinguishable from previous mask content.
cv::copyMakeBorder(image, image, 1, 0, 1, 0, cv::BORDER_CONSTANT, 0);
cv::copyMakeBorder(mask, mask, 1, 0, 1, 0, cv::BORDER_CONSTANT, 64);
// Inpaint and make pretty output again, using the newly border expanded imge and mask.
f_generate_output("OpenCV_Inpaint_WithBorder.png");
}Issue submission checklist
- I report the issue, it's not a question
- I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
- I updated to the latest OpenCV version and the issue is still there
- There is reproducer code and related data files (videos, images, onnx, etc)