System information (version)

• OpenCV 4.1.2 and master branch
• Operating System: all

Detailed description

The code for Affine3DEstimatorCallback::checkSubset is bugged as it assesses colinearity of 3D points based solely on x and y coordinates. As a result, if you pass array of 3D points with coordinates [x, 0, z] and [x', 0, z'] they're always considered as colinear. CheckSubset returns false and thus EstimateAffine3D returns false as well

Steps to reproduce

Attached is a sample file where each line is made of Pt1 coordinates Pt2 coordinates
Sample code to read file and perform EstimateAffine3D transformation:

std::ifstream tis("matches.txt", std::ios_base::in);

std::vector<cv::Point3f> m1, m2;
while (tis.good())
{
    cv::Point3f p1, p2;
    tis >> p1.x >> p1.y >> p1.z >> p2.x >> p2.y >> p2.z;
    m1.push_back(p1);
    m2.push_back(p2);
}
std::cout << "m1 has " << m1.size() << " entries, m2 has " << m2.size() << std::endl;
cv::Mat m3D, inl;
int res = cv::estimateAffine3D(m1, m2, m3D, inl);
std::cout << "estimate 3D returned " << res << " with " << cv::countNonZero(inl) << " inliers;" << std::endl;

On OpenCV 2.4.11 EstimateAffine3D returns 1 with 860 inliers, with OpenCV 4.1.2 it returns 0 with 0 inliers

matches.txt