When computing the transformation between two 3D point clouds using estimateAffine3D, there are less inliers than expected (in my case, as the threshold is less than one). If I re-check all the matches, compute the error and compare it to the given threshold, then I will find more inliers.

The Affine3DEstimatorCallback, which is used by estimateAffine3D, computes the distance between the to-point and the reprojected from-point (see the square root in https://github.com/Itseez/opencv/blob/master/modules/calib3d/src/ptsetreg.cpp#l465). But RANSACPointSetRegistrator::findInliers uses the squared threshold to find inliers (see https://github.com/Itseez/opencv/blob/master/modules/calib3d/src/ptsetreg.cpp#l94 ff). Thus, the squared threshold will be used on the non-squared error.

Users of the library could fix it (for now) by using the square root of the threshold as parameter, but it would be nice if this bug could be fixed. All that would be necessary is to remove the square root from the error computation in Affine3DEstimatorCallback::computeError, so it behaves like e.g. FMEstimatorCallback or HomographyEstimatorCallback.