System information (version)

• OpenCV => 3.4
• Operating System / Platform => Linux 64 Bit
• Compiler => x86_64-linux-gnu-g++-8

Detailed description

For certain contours, if approxPolyDP is passed an epsilon of NaN, then it OOMs.

This may simply be a request for input validation in approxPolyDP, because I believe an epsilon of NaN is meaningless.

Steps to reproduce

Here is a fairly minimal repro. Note that this uses the same contour as in the repro of #16051. You may need to tweak the headers/namespaces.

#include "third_party/OpenCVX/include/opencv2/imgproc/imgproc.hpp"
int main(int argc, char **argv) {
  std::vector<cv::Point> contour(
      {cv::Point(2085, 1415), cv::Point(2087, 1415), cv::Point(2089, 1414),
       cv::Point(2089, 1414), cv::Point(2087, 1412)});
  float epsilon = NAN;
  std::vector<cv::Point> approx;
  cv::approxPolyDP(contour, approx, epsilon,
                    true /* closed curve */);
  // The process will OOM before getting here.
}