Unexpected exception in solvePnPRansac caused by input points#19253
Unexpected exception in solvePnPRansac caused by input points#19253alalek merged 2 commits intoopencv:3.4from
Conversation
|
@mightbxg, thank you for the contribution. Maybe, inserting try-catch into BTW, please, keep in mind that in the upcoming OpenCV 5.0 (the "next" branch) the implementation of solvePnP, including the RANSAC flavor, will change. And both in the master branch and the next branch there is new RANSAC framework (called USAC) available that provides much better performance. You are welcome to try it out. If it works well on your dataset, then switching to this USAC is probably a better idea than trying to fix the obsolete solution that will eventually be thrown away. |
vpisarev
left a comment
There was a problem hiding this comment.
need to move try-catch outside of solvePnPGeneric into solvePnPRansac. Otherwise some user may experience undefined behaviour instead of exception in the corner cases
|
@mightbxg Friendly reminder. |
b34e39b to
7ff9825
Compare
|
@vpisarev Thanks a lot for you suggestions. I think you are right about not modifying Here is the test data that can reproduce the problem. Load points from the yaml file and call Mat rvec, tvec;
bool ret = solvePnPRansac(pts3d, pts2d, Mat::eye(3, 3, CV_64FC1), noArray(), rvec, tvec, false, 100, 4. / 460.0);I looked into the document about USAC and it's quite exciting, I will try it later. However, as the traditional |
mightbxg
left a comment
There was a problem hiding this comment.
Removed try-catch block from solvePnPGeneric. Added a point-num-checking in solvePnPRansac before it calls solvePnP. The commit is amended.
|
@mightbxg, thank you!
|
@vpisarev Sorry, I'm not sure about the purpose of adding this test. Is it for showing that the problem was there before fix (as I described before) or the problem is gone after fix? To clarify, let me describe the problem again. The role of "Exception" is to inform the user of his/her "bad usage". For instance, P3P need 4 points, and when 3 points are input, there should be an Exception to tell the user that the usage is wrong. For the former purpose (reproduce the problem), should I just create an "issue" based on the not-fixed branch 3.4? For the latter purpose, I presume this test should be added to the pull request. But as the problem is already fixed, |
modules/calib3d/src/solvepnp.cpp
Outdated
| int npoints1 = compressElems(&opoints_inliers[0], mask, 1, npoints); | ||
| compressElems(&ipoints_inliers[0], mask, 1, npoints); | ||
|
|
||
| if ( flags == SOLVEPNP_ITERATIVE && npoints1 < 6 ) |
There was a problem hiding this comment.
I believe it is a little bit more complex.
When 3D points are not planar, cvFindExtrinsicCameraParams2 uses DLT algorithm which indeed requires at least 6 points.
But with coplanar 3D points, cvFindExtrinsicCameraParams2 should work with at least 4 points.
BTW, see here: #8782 my previous comment about the behavior of solvePnPRansac and this PR: #9086
In my opinion, behavior of solvePnPRansac is still not totally clear / intuitive.
There was a problem hiding this comment.
Ok, then cvFindExtrinsicCameraParams2 may get good result with 4 points, but may throw Exception with 5 points.
I think CV_CheckGE in cvFindExtrinsicCameraParams2 is not a good idea. But since it returns void (not bool), throwing Exception may be the only option to inform that the result is bad.
As a result, try-catch seems inevitable.
There was a problem hiding this comment.
Ok, then
cvFindExtrinsicCameraParams2may get good result with 4 points, but may throw Exception with 5 points.
Not sure to follow correctly. In cvFindExtrinsicCameraParams2 it checks first if the 3D points are coplanar. For coplanar case, it should never throw exception, except theoretically for degenerate cases like all collinear points but this is not handled/detected.
And there are already a check for at least 4 points if useExtrinsicGuess is false:
opencv/modules/calib3d/src/calibration.cpp
Line 1014 in 863ecde
There was a problem hiding this comment.
cvFindExtrinsicCameraParams2 does check if the 3D points are coplanar. And if not, it may throw exception.
When someone calls cvFindExtrinsicCameraParams2 directly, he can make sure the input points are more than 4. If he knows that the points may not be coplanar, he can make sure the points are more than 6. In this cases, cvFindExtrinsicCameraParams2 won't throw exceptions, because the user calls it in the "right" way.
However, codes in solvePnPRansac does not guarantee that. When the user knows the points used to solve PnP are very likely to be noncoplanar, the only thing he can do is to input as many points as possible (6 at least). But RANSACE procedure (the intermediate calculation) may reduce the points and make them not suitable for cvFindExtrinsicCameraParams2. That's why I think Exception here is not supposed to happen. It's NOT the user's fault.
To fix this, solvePnPRansac should either check the coplanarity before calling solvePnP, or call solvePnP with try-catch. The former solution may add unnecessary calculation.
Main purpose is avoiding same/similar regressions in the future.
These cases should work too (as described - fail before, pass after the fix). |
7ff9825 to
863ecde
Compare
|
I mistakenly force-pushed branch 3.4 to my own branch (mightbxg:bugfix_PnPRansac), and github closed this pull request automatically. Sorry for that. I will reopen this later. |
I have not re-read the code carefully, but one possible other option could be to log somehow the error to warn the user that Because if I recall correctly, to compute solution for the MSS, 5 points are used and EPnP method. Consensus set is also 5 points, so it should be possible to return directly |
|
@catree You are right about the MSS step. It can calculate and return The user can know if As a computing library, I don't think OpenCV should "log something" for the user. |
The thing is that if (I don't have the solution.) ( |
I got your point. Your opinion is that if the MSS stage passed (EPNP for 5 or more points), then I agree with that in some degree. But since the user calls So, this question seems a little subjective and may need more discussion. The existing code in |
|
I see two approaches with this specific case:
or:
I have a slight preference for the first approach, but second solution is fine also for me. In any case, root case of this specific situation should be investigated, because:
|
8668ba9 to
424e633
Compare
|
@catree With this change, the user can choose to accept the result when BTW, I noticed someone else is working on this issue too #19492. His test data seems more concise. |
mightbxg
left a comment
There was a problem hiding this comment.
solvePnPRansac returns true even if the solvePnP stage failed. In this case, result from MSS stage is taken, and the exception content from solvePnP is put into log.
catree
left a comment
There was a problem hiding this comment.
Quick review.
Yes, probably we could try to fuse the two PRs.
/cc @Johnzdh
modules/calib3d/src/solvepnp.cpp
Outdated
| catch (const cv::Exception& e) | ||
| { | ||
| result = 0; | ||
| CV_UNUSED(e); |
There was a problem hiding this comment.
Is it needed since it is used in CV_LOG_DEBUG?
There was a problem hiding this comment.
I'm not quite sure about this, but I have seen same usage in calibinit.cpp. Maybe in Release build, CV_LOG_DEBUG doesn't use e, then CV_UNUSED can supress the warning?
There was a problem hiding this comment.
Yes, I think that it is to probably suppress warning in release mode.
| Mat camera_mat = Mat::eye(3, 3, CV_64FC1); | ||
| Mat rvec, tvec; | ||
| vector<uchar> inliers; | ||
| bool result = solvePnPRansac(pts3d, pts2d, Mat::eye(3, 3, CV_64FC1), noArray(), rvec, tvec, false, 100, 4. / 460.0, 0.99, inliers); |
There was a problem hiding this comment.
- / 460.0
Reprojection error is in pixel unit (default value is 8). Wondering if it explains why you only have a consensus set of 5 points?
There was a problem hiding this comment.
In my test data, the 2D points are from the normalized image coordiante system, that's why the camera matrix is Mat::eye(3, 3, CV_64FC1). So, the reprojection error here is not in pixel unit.
As you can see, the focal length of my camera is about 460.0, and the tolerance of reprojection error is set to 4 pixels. It should be fine.
There was a problem hiding this comment.
You are right. I did not see that the camera matrix is identity.
| vector<Point3f> pts3d; | ||
| vector<Point2f> pts2d; | ||
|
|
||
| pts3d = { |
There was a problem hiding this comment.
See the buildbot (https://pullrequest.opencv.org/#/summary/opencv) errors.
This requires C++11 features.
4203efa to
1b6cf69
Compare
mightbxg
left a comment
There was a problem hiding this comment.
Changes:
- Store test data with cv::Mat instead of std::vector to avoid c++11 feature (initializer list).
- Reduce test data to minimum set (6 input points)
1b6cf69 to
ba5fcfb
Compare
catree
left a comment
There was a problem hiding this comment.
Just some minor suggestions.
Thanks for your contribution. I am also an external contributor and it is great to see contributions from the community.
|
|
||
| TEST(Calib3d_SolvePnPRansac, bad_input_points) | ||
| { | ||
| Mat pts2d = (Mat_<float>(6, 2) << |
There was a problem hiding this comment.
I would add some comments before this line to explain this test:
- with this specific data
- when computing the final pose using points in the consensus set with
SOLVEPNP_ITERATIVEandsolvePnP() - an exception is thrown from
solvePnPbecause there are 5 non-coplanar 3D points and the DLT algorithm needs at least 6 non-coplanar 3D points - with PR Unexpected exception in solvePnPRansac caused by input points #19253 we choose to return
true, with the pose estimated from the MSS stage instead of throwing the exception
There was a problem hiding this comment.
Thanks, it does need some explanation.
429e74a to
08d9ca1
Compare
|
@catree @vpisarev @asmorkalov |
|
thank you, looks good to me! 👍 |
There was a problem hiding this comment.
@vpisarev @asmaloney
Need to discuss the change below.
Function result values:
| result = solvePnP | (1) existed code | (2) This PR | (3) PR 19492 | case4(no PR) |
|---|---|---|---|---|
| result > 0 | true | true | true | true |
| result<=0 | false | true (❓) | false | false |
| exception | exception | true (❓) | true (❓) | false |
| if( _inliers.needed() ) | ||
| _inliers.release(); | ||
|
|
||
| return false; |
There was a problem hiding this comment.
This part changes behavior of the case which is not covered by PR's description:
result = solvePnP()doesn't throw exception- it returns
result <= 0
Alternative #19492 preserves behavior of non-exception case.
When I call solvePnPRansac with 69 points (data I provided above), then I should get a result (even if it's false) rather than an Exception.
false code path is changed.
solvePnPRansac returns true even if the solvePnP stage failed. In this case, result from MSS stage is taken, and the exception content from solvePnP is put into log.
Only exception's case is logged, but returned value is forced "true" for other cases too.
There was a problem hiding this comment.
You are right, the logging strategy here is problematic.
There was a problem hiding this comment.
Main question is about the final returned value.
| result = solvePnP | (1) existed code | (2) This PR | (3) PR 19492 | case4(no PR) |
|---|---|---|---|---|
| result > 0 | true | true | true | true |
| result<=0 | false | true (❓) | false | false |
| exception | exception | true (❓) | true (❓) | false |
There was a problem hiding this comment.
I think you meant to @ someone else (not me)?
08d9ca1 to
914ffc0
Compare
|
@alalek In conclusion, this PR introduced changes about
The second change is what should be discussed here. Like you've said, the behavior of non-exception case in the existing OpenCV codes is to return |
|
So, what is the purpose of this Why we can't just tune |
|
|
Just did a very quick look on the new comments. Need to re-read everything later in details, and think more about the behavior changes / matrix.
Normally, the RANSAC algorithm is based upon the assumption to use the minimal number of data to compute a solution, and then check if this solution agrees with the whole data. The solution that gives the bigger consensus set is retained. So increasing the number of There are I think some workshop on this topic: |
|
We should try to fix only this specific case:
In my opinion, computing the final pose with So in this specific case, since it is not possible mathematically to compute 3D pose with 5 points and the DLT algorithm, I propose to:
And not change the behavior for the other cases:
|
|
The initial reason of this PR is: I found my SLAM program crashes frequently because of the 5-points-issue, and it cannot be avoided outside OpenCV without try-catch, which I think is unreasonable. The main problem for me is the unexpected exception, not the returned value in this case. I (and maybe most of SLAM developers who use On the other hand, as @vpisarev have said, |
|
still getting this issue in android OpenCV(4.10.0) Error: Unspecified error (> DLT algorithm needs at least 6 points for pose estimation from 3D-2D point correspondences. (expected: 'count >= 6'), where Any help ? |
resolves #17799
relates #14447
When
solvePnPRansacis called, it will do RANSAC and then callsolvePnPGeneric, in whichcvFindExtrinsicCameraParams2will be called. Sometimes the point-pairs after RANSAC are less than 6, but the DLT algorithm used incvFindExtrinsicCameraParams2(SOLVEPNP_ITERATIVE) needs at least 6 points, in this case the function will throw an error. See solvePnPGeneric, cvFindExtrinsicCameraParams2.The proper logic for
solvePnPGenericshould be returningfalse(0 or -1) instead of throwing an error to the executable, so I added try-block, just like SOLVEPNP_IPPE below.Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
Patch to opencv_extra has the same branch name.