-
-
Notifications
You must be signed in to change notification settings - Fork 56.5k
Calib3d_SolvePnP.accuracy is too unstable based on random seed #9187
Copy link
Copy link
Closed
Labels
Description
System information (version)
- OpenCV => master (7733654, 20f603a)
- Operating System / Platform => Windows 7 64 Bit Core i7 / Ubuntu 14.04
- Compiler => Visual Studio 2013 / GCC 4.8
Detailed description
- The test
Calib3d_SolvePnP.accuracyfor methodSOLVEPNP_P3Pseems too strict for some input data - I found the error on Jetson TK1
[----------] 3 tests from Calib3d_SolvePnP
[ RUN ] Calib3d_SolvePnP.accuracy
mode: 0, method: 0 -> 98.4% (err < 8.30464e-06)
mode: 0, method: 1 -> 95.6% (err < 1.34671e-05)
mode: 0, method: 2 -> 5.3% (err < 0.00592602)
mode: 0, method: 3 -> 99.9% (err < 0.000835733)
mode: 0, method: 4 -> 100% (err < 1.90072e-05)
mode: 0, method: 5 -> 98.8% (err < 0.00191663)
mode: 1, method: 0 -> 98.4% (err < 5.48124e+14)
mode: 1, method: 1 -> 96.1% (err < 0.352514)
mode: 1, method: 2 -> 5.8% (err < 0.00651514)
mode: 1, method: 3 -> 99.8% (err < 0.441405)
mode: 1, method: 4 -> 99.5% (err < 1.46951)
mode: 1, method: 5 -> 99.1% (err < 0.000364948)
/home/ubuntu/opencv-fork/modules/ts/src/ts.cpp:541: Failure
Failed
failure reason: Bad accuracy
test case #-1
seed: ffffffffffffffff
-----------------------------------
LOG:
Invalid accuracy for method 2, failed 947 tests from 1000, maximum error equals 0.005926, distortion mode equals 0
Invalid accuracy for method 2, failed 942 tests from 1000, maximum error equals 0.006515, distortion mode equals 1
-----------------------------------
[ FAILED ] Calib3d_SolvePnP.accuracy (6192 ms)
[ RUN ] Calib3d_SolvePnP.double_support
[ OK ] Calib3d_SolvePnP.double_support (2 ms)
[ RUN ] Calib3d_SolvePnP.translation
[ OK ] Calib3d_SolvePnP.translation (4 ms)
[----------] 3 tests from Calib3d_SolvePnP (6208 ms total)
- I thought this problem comes from difference of instruction set and/or compiler
- Though during the investigation I realized that this doesn't happen even on exactly same platform/OS/program
- For instance, when I pass
--gtest_filter=Calib3d_SolvePnP*to gtest program to skip all the other tests, the test actually passes.
- For instance, when I pass
[----------] 3 tests from Calib3d_SolvePnP
[ RUN ] Calib3d_SolvePnP.accuracy
mode: 0, method: 0 -> 98.6% (err < 9.75961e-06)
mode: 0, method: 1 -> 96.7% (err < 1.63235e-05)
mode: 0, method: 2 -> 89.6% (err < 0.0124706)
mode: 0, method: 3 -> 99.9% (err < 0.000706929)
mode: 0, method: 4 -> 100% (err < 1.67126e-05)
mode: 0, method: 5 -> 98.5% (err < 0.00132396)
mode: 1, method: 0 -> 98.2% (err < 7779.09)
mode: 1, method: 1 -> 96.3% (err < 0.219446)
mode: 1, method: 2 -> 88.7% (err < 0.00896903)
mode: 1, method: 3 -> 100% (err < 1.72653e-05)
mode: 1, method: 4 -> 99.4% (err < 1.62516)
mode: 1, method: 5 -> 98.8% (err < 0.00115234)
[ OK ] Calib3d_SolvePnP.accuracy (6132 ms)
[ RUN ] Calib3d_SolvePnP.double_support
[ OK ] Calib3d_SolvePnP.double_support (3 ms)
[ RUN ] Calib3d_SolvePnP.translation
[ OK ] Calib3d_SolvePnP.translation (4 ms)
[----------] 3 tests from Calib3d_SolvePnP (6148 ms total)
- I modified a bit and figured out that the error is based on input points, not the platforms
- In another word, it could happen on other platforms based on the random seed
- IMHO, this threshold should be loosen to something like
3.0e-3
Steps to reproduce
Visual Studio 2013
- I pushed a branch in my repository ( f76e226 ) and I added a test
Calib3d_SolvePnP.verifyBehaviourinopencv_test_calib3d - I hard coded a value which reproduces
bad accuracyfailure, and the value is in the range of generated random data
gcc (ARM)
- I confirmed on following platforms
- Jetson TK1, Jetson TX1, Jetson TX2, ODROID-C2, ODROID-X2, PINE64,
- As described above, it's not the problem of platform, it's the problem of run time.
- For example, I used following cmake option for Jetson TK1
cmake option:-DCPU_BASELINE=FP16 -DNEON=ON- the execution order of the test is
- cameraCalibrationTiltTest
- Calib3d_UndistortPoints
- Calib3d_InitUndistortRectifyMap
- Calib3d_Undistort
- Calib3d_StereoBM
- Calib3d_StereoSGBM
- Calib3d_StereoSGBM_HH4
- Calib3d_POSIT
- Calib3d_CalibrateCamera_CPP
- Calib3d_SolveP3P
- Calib3d_SolvePnPRansac
- Calib3d_SolvePnP
Reactions are currently unavailable