Skip to content

Memory leak in fill_nearest no distance image is provided in libvips 8.16 #4352

@VivitionDeveloper

Description

@VivitionDeveloper

Bug report

Describe the bug
When calling fill_nearest() (C++ wrapper), without providing the optional distance argument, then this function call leaks memory.

To Reproduce
Use attached images, and see code piece below.
Note that out.jpg is just added as reference and is the produced result, using the code below and using in.jpg and mask.jpg.

Compile code below to e.g. test.exe and run as:
test.exe in.jpg masked_in.jpg mask.jpg

The masked_in.jpg is generated upon executing the test, and should be identical to attached out.jpg.
Because names got lost upon attaching files to this issue:

  • the input is an jpeg image of a car (6000x4000) with a visible ceiling
  • the mask is a png of 160x160 pixels and marks objects in the image to certain classes (5 == ceiling)
  • the result is an inpainted ceiling of the provided input image

When defining DOES_NOT_LEAK, then the leak disappears. I guess that some internal variable is not freed when this optional (according to the documentation) distance option is not provided.

Finally: feel free to provide feedback for the used approach of inpainting. It is laterally related to: #4155.

int main(int argc, char *argv[]) {
       if (argc < 4) {
		printf("Usage: %s <inputimage> <outputimage> <maskimage>\n", argv[0]);
		return 1;
	}

	const char* szInputImage = argv[1];
	const char* szOutputImage = argv[2];
    
	if (VIPS_INIT(<yourvipsdlldir>)) {
		vips_error_exit(nullptr);
	}

	for (int i = 0; i < 10; i++)
	{
		const char* szMaskImage = argv[3];
		auto viInput = VImage::new_from_file(szInputImage);
		const auto viMask = VImage::new_from_file(szMaskImage);
		assert(viMask.bands() == 1);
		assert(viMask.interpretation() == VipsInterpretation::VIPS_INTERPRETATION_B_W);

		const auto iImageWidth = viInput.width();
		const auto iImageHeight = viInput.height();
		constexpr int iDesiredWidthInPixelsForMask = 600;
		// take a copy of the input image and scale down as the sigma below is based on this width
		VImage viOverlay = viInput;
		const double dScaleDownFactor = (double)iImageWidth / iDesiredWidthInPixelsForMask;
#define ROUND(a) ((int)floor((a)+0.5))
		const size_t iScaleWidth = ROUND(iImageWidth / dScaleDownFactor);
		const size_t iScaleHeight = ROUND(iImageHeight / dScaleDownFactor);
		double dOverlayHorScale = iScaleWidth / (double)iImageWidth;
		double dOverlayVertScale = iScaleHeight / (double)iImageHeight;
		viOverlay = viOverlay.resize(dOverlayHorScale, VImage::option()->set("vscale", dOverlayVertScale));
		// the provided input mask is based on constants and the ceiling has a value of 5
		VImage viBW(viMask[0] == 5);
		double dBWHorScale = iScaleWidth / (double)viMask.width();
		double dBWVertScale = iScaleHeight / (double)viMask.height();
		// upscale to same aspected ratio as input image, to prevent strange transitions from image to overlay
		viBW = viBW.resize(dBWHorScale, VImage::option()->set("vscale", dBWVertScale));

		const auto doGrowBinaryMask = [](VImage& viImg, const double dSigma) {
			viImg = viImg.gaussblur(dSigma);
			// convert all non-blacks to solid white, making the white part bigger
			viImg = (viImg != 0);
			};

		// use arbitrary blur value
		const double dSigma = 5.0;
		doGrowBinaryMask(viBW, dSigma);
		auto viBWInverted = viBW.invert();
		const auto viBlackOut = VImage::bandjoin(
			{ viBWInverted, viBWInverted, viBWInverted, viBW }
		).copy(VImage::option()->set("interpretation", VIPS_INTERPRETATION_sRGB));

		// add the area to get inpainted as 'black' area in the downscaled source image
		viOverlay = viOverlay.composite(viBlackOut, VipsBlendMode::VIPS_BLEND_MODE_OVER,
			VImage::option()->set("x", 0)->set("y", 0));
		assert(viOverlay.bands() == 4);
		// Must remove alpha channel, otherwise inpainting does not work, as libvips apparently either uses the 
		// alpha channel when existent, or the other channels (the latter is wanted here).
		viOverlay = viOverlay.flatten();
		assert(viOverlay.bands() == 3);
#ifdef DOES_NOT_LEAK
		VImage viDistance;
		viOverlay = viOverlay.fill_nearest(VImage::option()->set("distance", &viDistance));
#else
		// The call below results in a memory leak when no viDistance was provided
		viOverlay = viOverlay.fill_nearest();
#endif
		// perform blur (not allowed when alpha channel exists) to remove the 'stripes' caused by inpainting
		// use a larger sigma than for the margin (see above), as otherwise the inpainting still results in 'stripes'.
		viOverlay = viOverlay.gaussblur(3 * dSigma);
		// apply additional blur on the alpha channel, to prevent visible diferences when the yaw of a car or reflection of light changes.
		doGrowBinaryMask(viBW, dSigma);
		viBW = viBW.gaussblur(dSigma);
		// and apply new alpha
		viOverlay = viOverlay.bandjoin(viBW);
		// scale extracted part back to original size
		double dImageHorScale = iImageWidth / (double)viOverlay.width();
		double dImageVertScale = iImageHeight / (double)viOverlay.height();
		viOverlay = viOverlay.resize(dImageHorScale, VImage::option()->set("vscale", dImageVertScale));

		// and apply overlay
		viInput = viInput.composite(viOverlay, VipsBlendMode::VIPS_BLEND_MODE_OVER,
			VImage::option()->set("x", 0)->set("y", 0));
		viInput.flatten();
		viInput.write_to_file((std::string(szOutputImage) + "[Q=95,optimize_coding,strip]").c_str());
	}
#endif
	
	vips_shutdown();
	return 0;
}

Expected behavior
No memory leak.

Actual behavior
When executing fill_nearest() a memory jump is found, which remains existent, even after vips_shutdown().

Environment

  • OS: Windows Server 2022, using Visual Studio 2022
  • Vips: 8.16

Image
Image
Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions