fixed VS warning, rebased.

@hrnr
This pull request looks like a good start for the project. A few questions:

You want to add a inlier mask as return value. This mask can be useful when you are inputing point sets as the inlier mask correspond to the set elements. However, in the test I saw you were testing the mask for estimating transform between two raw images. I don't see how inlier mask is useful in this case.

In addition, when inputing raw images to estimateRigidTransform, it uses optical flow to find correspondences. I think in your case of stitching document, feature based correspondences might be more suitable.

I am confused with the test so let me know if I misunderstand your idea:-)

When there are 2 images inliers mask has 2 purposes: calling .size() on vector gets you number of points used for transform estimation, and you can count which items are set to get number of inliers. Both can be use for probability model to measure how confident is the estimated tranformation. In this case this API is not optimal to get these 2 integers, but I think it is more important to have the same behaviour for both cases (images and points). I have chosen this API to be similar to findHomography.

Exactly. I'm not planning to use estimateRigidTransform for images, I will use it with features.

Sorry for confusion. Test is not showing how I plan to use this new functionality, it's only there to keep this function covered (tests the new overload of the estimateRigidTransform and checks values in inliers mask).

Yes, sounds nice to follow the input parameters of findHomography or findFundametalMat. Then how about following them more strictly? Just suggestion for example:

estimateRigidTransform(InputArray src, InputArray dst, bool fullAffine) // Compatibility
estimateRigidTransform(InputArray srcPoints, InputArray dstPoints, bool fullAffine,  double ransacReprojThreshold=3., double confidence=0.99, OutputArray mask=noArray()) // Maybe assert the inputs to be point sets.

Since we are using feature points, let's only make up the API we really need and test in the way it supposed to be.

I agree it would be more logical to accept only points to avoid doing too much magic in one place. But then I think we should rename the method, because it has different semantics. Something like findAffine or findAffineTransform (to be consistent with warpAffine, getAffineTransform and findHomography).

Yes. I agree that something like findAffine looks more suitable in the our context.

ok, I think this version is reasonable API for extending the current implementation of estimateRigidTransform. I have omitted confidence because it does not make sense for the implementation (it simply uses only maxIters), inlierTreshold also work differently from findHomography and others.

Another option would be to reimplement everything from stratch in calib3d and make it similar to estimateAffine3D (probably estimateAffine2D and estimatePartialAffine2D)

Tests are missing now, but I have used findAffineTransform to implement estimateRigidTransform and it passes all original tests. I will add tests once we agree on the API.

Thanks for the remind. I forgot about estimateAffine3D. Yes, it actually looks more consistent to have something like estimateAffine2D. Are you still willing to do this change after two times of the boring refactor? :-)

I will do it. It will need a complete rewrite, but I think it is worth it. There is something that looks like a nice framework for ransac in calib3d. (hope it is OK to have 2D functions in calib3d)

It's a pity that this is a blocker for other things. I have working affine matcher and affine estimator. I think I will open a PR for those too next week with transform section commented so we could discuss API and implementation.

Yes, the PointSetRegistrator framework formats RANSAC algorithm components pretty well. It should be slightly simpler to follow the callback of estimateAffine3D to get a 2D version.

superseded by #6615