Skip to content

Calib3d_SolvePnP.accuracy is too unstable based on random seed #9187

@tomoaki0705

Description

@tomoaki0705
System information (version)
  • OpenCV => master (7733654, 20f603a)
  • Operating System / Platform => Windows 7 64 Bit Core i7 / Ubuntu 14.04
  • Compiler => Visual Studio 2013 / GCC 4.8
Detailed description
  • The test Calib3d_SolvePnP.accuracy for method SOLVEPNP_P3P seems too strict for some input data
  • I found the error on Jetson TK1
[----------] 3 tests from Calib3d_SolvePnP
[ RUN      ] Calib3d_SolvePnP.accuracy
mode: 0, method: 0 -> 98.4% (err < 8.30464e-06)
mode: 0, method: 1 -> 95.6% (err < 1.34671e-05)
mode: 0, method: 2 -> 5.3% (err < 0.00592602)
mode: 0, method: 3 -> 99.9% (err < 0.000835733)
mode: 0, method: 4 -> 100% (err < 1.90072e-05)
mode: 0, method: 5 -> 98.8% (err < 0.00191663)
mode: 1, method: 0 -> 98.4% (err < 5.48124e+14)
mode: 1, method: 1 -> 96.1% (err < 0.352514)
mode: 1, method: 2 -> 5.8% (err < 0.00651514)
mode: 1, method: 3 -> 99.8% (err < 0.441405)
mode: 1, method: 4 -> 99.5% (err < 1.46951)
mode: 1, method: 5 -> 99.1% (err < 0.000364948)
/home/ubuntu/opencv-fork/modules/ts/src/ts.cpp:541: Failure
Failed

	failure reason: Bad accuracy
	test case #-1
	seed: ffffffffffffffff
-----------------------------------
	LOG:
Invalid accuracy for method 2, failed 947 tests from 1000, maximum error equals 0.005926, distortion mode equals 0
Invalid accuracy for method 2, failed 942 tests from 1000, maximum error equals 0.006515, distortion mode equals 1

-----------------------------------
[  FAILED  ] Calib3d_SolvePnP.accuracy (6192 ms)
[ RUN      ] Calib3d_SolvePnP.double_support
[       OK ] Calib3d_SolvePnP.double_support (2 ms)
[ RUN      ] Calib3d_SolvePnP.translation
[       OK ] Calib3d_SolvePnP.translation (4 ms)
[----------] 3 tests from Calib3d_SolvePnP (6208 ms total)
  • I thought this problem comes from difference of instruction set and/or compiler
  • Though during the investigation I realized that this doesn't happen even on exactly same platform/OS/program
    • For instance, when I pass --gtest_filter=Calib3d_SolvePnP* to gtest program to skip all the other tests, the test actually passes.

[----------] 3 tests from Calib3d_SolvePnP
[ RUN      ] Calib3d_SolvePnP.accuracy
mode: 0, method: 0 -> 98.6% (err < 9.75961e-06)
mode: 0, method: 1 -> 96.7% (err < 1.63235e-05)
mode: 0, method: 2 -> 89.6% (err < 0.0124706)
mode: 0, method: 3 -> 99.9% (err < 0.000706929)
mode: 0, method: 4 -> 100% (err < 1.67126e-05)
mode: 0, method: 5 -> 98.5% (err < 0.00132396)
mode: 1, method: 0 -> 98.2% (err < 7779.09)
mode: 1, method: 1 -> 96.3% (err < 0.219446)
mode: 1, method: 2 -> 88.7% (err < 0.00896903)
mode: 1, method: 3 -> 100% (err < 1.72653e-05)
mode: 1, method: 4 -> 99.4% (err < 1.62516)
mode: 1, method: 5 -> 98.8% (err < 0.00115234)
[       OK ] Calib3d_SolvePnP.accuracy (6132 ms)
[ RUN      ] Calib3d_SolvePnP.double_support
[       OK ] Calib3d_SolvePnP.double_support (3 ms)
[ RUN      ] Calib3d_SolvePnP.translation
[       OK ] Calib3d_SolvePnP.translation (4 ms)
[----------] 3 tests from Calib3d_SolvePnP (6148 ms total)
  • I modified a bit and figured out that the error is based on input points, not the platforms
  • In another word, it could happen on other platforms based on the random seed
  • IMHO, this threshold should be loosen to something like 3.0e-3
Steps to reproduce
Visual Studio 2013
  • I pushed a branch in my repository ( f76e226 ) and I added a test Calib3d_SolvePnP.verifyBehaviour in opencv_test_calib3d
  • I hard coded a value which reproduces bad accuracy failure, and the value is in the range of generated random data
gcc (ARM)
  • I confirmed on following platforms
    • Jetson TK1, Jetson TX1, Jetson TX2, ODROID-C2, ODROID-X2, PINE64,
    • As described above, it's not the problem of platform, it's the problem of run time.
    • For example, I used following cmake option for Jetson TK1
  • cmake option : -DCPU_BASELINE=FP16 -DNEON=ON
  • the execution order of the test is
    • cameraCalibrationTiltTest
    • Calib3d_UndistortPoints
    • Calib3d_InitUndistortRectifyMap
    • Calib3d_Undistort
    • Calib3d_StereoBM
    • Calib3d_StereoSGBM
    • Calib3d_StereoSGBM_HH4
    • Calib3d_POSIT
    • Calib3d_CalibrateCamera_CPP
    • Calib3d_SolveP3P
    • Calib3d_SolvePnPRansac
    • Calib3d_SolvePnP

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions