System information (version)

• OpenCV => recent master(f6ec0cd + fde4b15)/3.4 (e4e0bb5)
• Operating System / Platform => Ubuntu 16.04 (Jetson TX2 )
• Compiler => GCC 5.4.0
• CUDA => 8.0

Detailed description

First of all, I know that this module name has the phrase "LEGACY" in it, but please let me write it down here.

When this test runs, it either causes a core dump or test failure

• when it crashes

$ ./bin/opencv_test_cudalegacy --gtest_filter=*SolvePnPRansac.Accuracy*

[----------] 1 test from CUDA_Calib3D/SolvePnPRansac
[ RUN      ] CUDA_Calib3D/SolvePnPRansac.Accuracy/0, where GetParam() = GP10B
*** Error in `./bin/opencv_test_cudalegacy': double free or corruption (top): 0x0000007f5c002be0 ***
Segmentation fault      (core dumped) 

• when it fails

$ ./bin/opencv_test_cudalegacy --gtest_filter=*SolvePnPRansac.Accuracy*

[----------] 1 test from CUDA_Calib3D/SolvePnPRansac
[ RUN      ] CUDA_Calib3D/SolvePnPRansac.Accuracy/0, where GetParam() = GP10B
unknown file: Failure
C++ exception with description "OpenCV(4.1.1-pre) /home/nvidia/opencv-fork/modules/calib3d/src/calibration.cpp:1171: error: (-2:Unspecified error) in function 'void cvFindExtrinsicCameraParams2(const CvMat*, const CvMat*, const CvMat*, const CvMat*, CvMat*, CvMat*, int)'
> DLT algorithm needs at least 6 points for pose estimation from 3D-2D point correspondences. (expected: 'count >= 6'), where
>     'count' is 4
> must be greater than or equal to
>     '6' is 6
" thrown in the test body.
[  FAILED  ] CUDA_Calib3D/SolvePnPRansac.Accuracy/0, where GetParam() = GP10B (162 ms)
[----------] 1 test from CUDA_Calib3D/SolvePnPRansac (162 ms total)

• I haven't traced for the crash version, but I assume that exception was thrown in the middle of the process and that caused double free.
• More over, the test failure seems suspicious.

1. the random sampling is relying on simple rand function which should not be used in the middle of parallel_for_
   https://github.com/opencv/opencv_contrib/blob/fde4b151c2bfabe013a7109946fdcfe7e14bec14/modules/cudalegacy/src/calib3d.cpp#L140

2. RANSAC chooses only 4 points hard coded
   https://github.com/opencv/opencv_contrib/blob/fde4b151c2bfabe013a7109946fdcfe7e14bec14/modules/cudalegacy/src/calib3d.cpp#L224

3. The input data is NOT co-plannar,
   https://github.com/opencv/opencv_contrib/blob/fde4b151c2bfabe013a7109946fdcfe7e14bec14/modules/cudalegacy/test/test_calib3d.cpp#L166

4. RANSAC tries to solve the PnP by 4 points. Thus, it always gets false at this branch
   

   opencv/modules/calib3d/src/calibration.cpp

   Line 1112 in f6ec0cd

   if( W[2]/W[1] < 1e-3)

5. and hit this assert EVERY TIME
   

   opencv/modules/calib3d/src/calibration.cpp

   Line 1171 in f6ec0cd

   CV_CheckGE(count, 6, "DLT algorithm needs at least 6 points for pose estimation from 3D-2D point correspondences.");

I'm no 100% sure, but this code existed from years ago, so I guess at the beginning, it was correct (may be trying only plannar RANSAC). But current implementation will crash (or fail).

I'm not sure but changing the hard coded points number from 4 to 6 made the test pass.
Before I send the PR, I'd like to ask which should I go.

• Shall I change the hard code value
• What should I do for the rand
• shall I keep it as is for now ? (the module has phrase "legacy" in the name)

Steps to reproduce

$ ./bin/opencv_test_cudalegacy --gtest_filter=*SolvePnPRansac.Accuracy*