integer overflow in convexHull #14521
Description
System information (version)
- OpenCV => 3.4.3
- Operating System / Platform => Windows 64 Bit
- Compiler => Visual Studio 2015
Detailed description
I was testing convexHull alternative algorithms, and I had different results unsing convexHull with Point and Point2f with the same data.
I looks like there is a potential overflow in
template<typename _Tp> static int Sklansky_( Point_<_Tp>** array, int start, int end, int* stack, int nsign, int sign2 )
convhull.cpp:82
_Tp convexity = ay*bx - ax*by; // if >0 then convex angle
Steps to reproduce
std::vector<cv::Point> points;
std::vector<cv::Point2f> pointsf;
points.push_back(cv::Point(14763, 2890));
points.push_back(cv::Point(14388, 72088));
points.push_back(cv::Point(62810, 72274));
points.push_back(cv::Point(63166, 3945));
points.push_back(cv::Point(56782, 3945));
points.push_back(cv::Point(56763, 3077));
points.push_back(cv::Point(34666, 2965));
points.push_back(cv::Point(34547, 2953));
points.push_back(cv::Point(34508, 2866));
points.push_back(cv::Point(34429, 2965));
for (auto it = points.begin(); it != points.end(); ++it) pointsf.push_back(*it);
std::vector<int> hull;
std::vector<int> hullf;
cv::convexHull(points, hull, false, false);
cv::convexHull(pointsf, hullf, false, false);
if (hull != hullf) throw std::runtime_error("error!");