In OpenCV 3.2 (and since OpenCV 3.0), solvePnPRansac does not behave as we could expect after reading the corresponding documentation.

The function prototype is: bool cv::solvePnPRansac(objectPoints, imagePoints, cameraMatrix, distCoeffs, rvec, tvec, useExtrinsicGuess = false, iterationsCount = 100, reprojectionError = 8.0, confidence = 0.99, inliers = noArray(), flags = SOLVEPNP_ITERATIVE).

The parameter useExtrinsicGuess will not be used and the choice of the PnP method can be misleading.

The RANSAC algorithm implemented currently can be summarized as (when the number of points is > 4):

• for a certain number of iterations:
  • select randomly 5 points to construct the "Minimal Sample Sets" (MSSs) and estimate the camera pose using SOLVEPNP_EPNP (see here) [Hypothesize]
  • compute the consensus set, an inlier is a 3D point with a reprojection error (using rvec and tvec estimated with MSSs) below a threshold [Test]
  • "terminates when the probability of finding a better ranked CS drops below a certain threshold" (here I think) or after reaching the maximum number of iterations
• estimate the final rvec and tvec using the inliers from the best consensus set with flags provided by the user (if SOLVEPNP_P3P it is SOLVEPNP_EPNP that is used instead), useExtrinsicGuess is ignored and sets to false (see)

The documentation should be updated to reflect this behavior (useExtrinsicGuess is ignored and about the choice of the PnP method for the two "stages") and maybe we should add a quick description of the implementation of the RANSAC algorithm.

Better, I think the implementation can be improved:

• in the MSSs step, 5 points is used but in my opinion 4 points should be used as it is the minimum number of points required to compute the camera pose, even with SOLVEPNP_EPNP (see here)

• in the MSSs step, only SOLVEPNP_EPNP is used (alternative could be SOLVEPNP_P3P, SOLVEPNP_AP3P, SOLVEPNP_ITERATIVE)

• how to deal with useExtrinsicGuess=true and the provided rvec, tvec?

  • it can be used when estimating the pose with the MSSs if SOLVEPNP_ITERATIVE is chosen
  • it can also be used for the final pose estimation with SOLVEPNP_ITERATIVE

• how to deal with useExtrinsicGuess=false and SOLVEPNP_ITERATIVE?

  • for the MSSs step, there is no problem as SOLVEPNP_ITERATIVE will call internally another function to provide an estimate of the solution
  • but for the final pose estimation, currently it will call another function that will compute the pose using all the inliers but I think we could use instead the rvec and tvec computed from the MSSs and from the best consensus set (should be tested, maybe the iterative method does not converge well if too far, that's why useExtrinsicGuess is set to false?)

• when there is only 4 points, SOLVEPNP_P3P is used for the MSSs. I don't know if the full RANSAC method will be done (at least 1 iteration? + the final pose computed with SOLVEPNP_EPNP). If this is the case, maybe we could directly call solvePnP with SOLVEPNP_P3P and return?

• how to use two differents methods for the pose computed from the MSSs and for the pose computed from the best consensus set (it is currently the case implicitly with SOLVEPNP_EPNP used for the MSSs and flags for the final pose)?

I will try to implement and see if my suppositions give better results (use 4 points, use rvec and tvec from the best consensus set). Not sure yet how to compare (generate synthetic data?, use real case data?).