solvePnPRansac with USAC returns empty inlier cv::Mat #21105
Description
System information (version)
- OpenCV => 4.5.4
- Operating System / Platform => Ubuntu 20.04
- Compiler => g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Detailed description
Used solvePnPRansac in its second definition, according to https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html#gab14667ec49eda61b4a3f14eb9704373b. The return of solvePnPRansac is true, but the inlier matrix is empty with zero rows and cols. The input 3d points, 2d points, and intrinsics matrix should be correct, because the non-USAC solvePnPRansac works perfectly, eg. cv::SOLVEPNP_EPNP.
Steps to reproduce
This is the code using cv::SOLVEPNP_EPNP.
bool pnp_result = cv::solvePnPRansac(
points_3d, points_2d, intrinsics, distortion,
r_vec, t_vec, true, 500, 2.0, 0.999, inliers, cv::SOLVEPNP_EPNP);
This is the failure code with USAC
const cv::UsacParams usac_params = cv::UsacParams();
// usac_params.confidence = 0.999;
// usac_params.maxIterations = 500;
// usac_params.threshold = 2.0;
bool pnp_result =
cv::solvePnPRansac(points_3d, points_2d, intrinsics,
distortion, r_vec, t_vec, inliers, usac_params);
Same results from the following code:
bool pnp_result = cv::solvePnPRansac(
points_3d, points_2d, intrinsics, distortion,
r_vec, t_vec, true, 500, 2.0, 0.999, inliers, cv::USAC_DEFAULT);
Besides, although the inlier set is empty, t_vec is updated, but is somehow not very accurate, compared to the ground truth.
Before upgrading to 4.5.4, I also tested on 4.5.1 with the same code. The inlier cv::Mat had the same row number as the input points, but all entries were zero.
Issue submission checklist
- I report the issue, it's not a question
- I checked the problem with documentation, FAQ, open issues,
forum.opencv.org, Stack Overflow, etc and have not found solution - I updated to latest OpenCV version and the issue is still there
- There is reproducer code and related data files: videos, images, onnx, etc
Sample data
t_vec and r_vec
cv::Mat r_vec = cv::Mat::zeros(3, 1, CV_64FC1);
cv::Mat t_vec = cv::Mat::zeros(3, 1, CV_64FC1);
point_3d
[-4.9636927, -4.9345894, 184.8967;
6.6590405, -0.030723378, 15.408057;
-7.4296608, 0.693977, 41.088154;
0.069690861, 0.084242582, 92.448349;
-15.064131, -0.067032829, 33.61758;
-0.2833156, 1.5381572, 6.8480253;
-14.256426, -0.10621057, 30.816116;
21.959194, -0.99008888, 92.448349;
6.7822266, -0.010009769, 18.48967;
1.6811881, -3.0044603, 92.448349;
-11.234019, -8.3594341, 61.632229;
-14.256425, -1.2700696, 30.816114;
-1.2964537, -0.57053161, 61.632233;
2.7111707, -7.294353, 33.61758;
3.9334979, -5.1748571, 26.413441;
10.29907, -2.9316156, 123.26447;
-126.49234, -13.629337, 369.79337;
-0.22212227, 0.86191028, 61.632225;
10.042377, -3.4280212, 20.54364;
-1.4483113, 0.28226036, 73.958679;
1.6579578, 0.86191028, 61.632233;
-47.399784, 0.97423369, 184.89668;
6.8359432, 0.66144741, 18.48967;
-1.1627278, 1.1117727, 41.088154;
20.468048, -5.5191293, 73.958672;
5.2855234, -6.3336654, 123.26447;
-0.12649794, 1.5671791, 6.9772334;
-24.976751, -3.4818075, 52.827625;
-91.576569, -13.092173, 369.79337;
11.436276, -3.7847526, 19.462391;
7.3697448, -3.4872372, 23.111639;
-20.49843, -3.6478364, 123.26446;
9.8970346, -4.199574, 18.488195;
33.373966, -3.1387517, 92.448349;
3.0241024, -3.2730434, 92.448349;
-0.46095964, 1.5570439, 6.9772344;
-13.651529, -2.0477765, 61.633373;
-27.524654, -11.111996, 184.8967;
1.8005079, -5.3647642, 36.978947;
3.685364, 1.3026766, 14.791736;
2.526715, -5.3042626, 73.958672;
7.3698778, -1.4897178, 23.112085;
0.035665292, 1.5469086, 6.9772334;
-22.421577, -11.380577, 184.89668;
-0.16047101, 1.2400403, 24.652893;
17.674788, -2.8333008, 73.958679;
9.1119413, -0.42860407, 28.445648;
-13.986185, 0.85232955, 28.445644;
3.6606677, -4.3558631, 46.223515;
-2.7116687, 1.2154616, 36.97934;
-5.8809462, 0.35599646, 36.97934;
-11.589601, 1.1828933, 28.445646;
2.4181471, -4.5840569, 33.617184;
12.502028, -4.3069787, 18.488113;
-18.858007, 1.0024983, 26.414331;
9.7843924, -3.7824104, 17.608864;
10.182422, -5.9024653, 33.616764;
-13.794711, -1.8983124, 33.618149;
-105.00571, -15.240834, 369.79337;
-8.8531494, -27.864523, 369.79709;
-13.745875, -1.3611374, 33.618145;
-14.122134, 0.92335713, 30.816118;
0.70035154, -5.3042626, 73.958679;
2.3204827, -6.8791966, 33.617203;
7.1347437, -5.0315628, 23.111654;
-6.1165895, 0.99240237, 41.088154;
-127.70605, -6.8146691, 184.89668;
10.542483, -6.708714, 30.815374;
-18.228788, -0.092170134, 46.224174;
1.5961977, 1.4940159, 7.1114111;
-12.822287, -8.7789927, 92.448357;
4.3902469, -2.2153947, 123.26447;
-2.4138522, -6.5127201, 123.26447;
-0.26518935, -6.3336654, 123.26447;
12.181621, -2.5080168, 21.75255;
31.359592, -2.7358775, 92.448349;
-1.8109517, 1.5374966, 9.4818821;
-19.705994, -2.6437073, 46.224174;
-231.31407, -12.827711, 369.91208;
3.9936106, -3.9525619, 24.652893;
-10.537328, 1.1573491, 21.752552;
-17.937126, 0.61880052, 26.41432;
10.012538, -4.4426479, 20.543646;
-13.67473, -0.6657694, 30.816622;
1.6811739, 0.15138677, 92.447365;
-25.10741, -13.797824, 184.8967;
-12.457329, 1.3068545, 28.445644;
2.2415624, -15.196732, 123.26303;
8.4992809, -0.76247853, 26.413239;
5.6240897, -2.4297578, 23.112085;
7.252336, -1.7812632, 21.752552;
1.8005232, -6.1437087, 36.979336;
9.4292078, -4.7036686, 19.462811;
5.9990287, -4.6329713, 24.652891;
-9.1215649, 1.4150907, 13.696151;
7.2192945, 0.0085482588, 46.223202;
6.5810075, 1.1751387, 26.413815;
0.2719765, -13.40625, 123.26351;
6.3943086, -3.4778998, 73.958679;
-138.87956, -11.751995, 369.87933;
6.8110976, 0.8106181, 26.413301;
7.7727528, -3.3362253, 23.112087;
-10.877041, 1.2422647, 20.544077;
4.9580164, 1.2002814, 15.408057;
-14.233967, -1.3366973, 33.61758;
-15.15284, -1.1806293, 30.81834;
-14.256424, 0.47571892, 30.816113;
0.19682343, 1.5647467, 6.6034064;
3.8729179, 1.1024122, 30.816114;
-17.936771, 1.3286146, 26.413815;
8.1594763, -3.7180388, 46.224171;
-3.6366355, 1.3569548, 15.408058;
-14.936557, 0.9349705, 28.445646;
7.3379712, 0.0027354523, 14.791478;
2.3246665, 1.2938942, 14.222823;
1.6396371, 1.4394096, 7.703733;
12.171328, -4.378449, 19.462391;
6.5921783, -0.36635503, 8.0389862;
6.2611213, 1.3265129, 8.2175322;
-18.627789, 0.65717053, 26.41433;
-13.651531, 0.19045678, 61.633377;
3.6265509, -3.6017511, 26.413446;
-9.0970716, 1.1500378, 46.22464;
-62.039413, -20.34618, 369.83447;
-13.047803, -0.82243156, 30.816118;
-13.472464, -7.6881199, 61.633339;
8.4506416, -4.4985733, 28.445038;
-23.441992, -3.3283315, 52.827629;
9.6121569, -2.5466526, 26.413815;
-14.22471, -26.521313, 369.79337;
2.4631786, 1.4015146, 7.8679442;
6.5200477, -4.3886938, 33.616936;
-114.67469, -15.240834, 369.79337;
-29.265352, -26.521317, 369.7934;
-7.7662382, 1.3671958, 10.272038;
10.478125, 0.29137868, 123.26447;
-20.6707, -27.595648, 369.7934;
9.889658, -7.8315187, 33.61758;
-20.832903, 0.81551826, 52.827625;
0.47256514, -3.2730432, 92.448349;
-0.25523296, 1.5411063, 9.019351;
-8.3174105, 1.3954192, 11.556111;
6.5360818, -2.8960834, 24.652435;
7.1565423, 0.71471095, 26.413815;
-18.707878, -11.347212, 123.26447;
-11.961657, 0.4597919, 28.446035;
9.1532602, -6.0481839, 28.445644;
3.037802, -5.8208508, 28.445286;
6.3123193, -5.3666787, 26.413342;
-15.107213, -6.1270523, 30.816654;
7.13906, 0.3428556, 19.462444;
7.0083647, -2.337733, 33.616905;
9.0069771, -5.4386082, 24.652372;
0.87374264, 1.0576484, 30.816114;
-55.725853, 1.2428167, 184.89671;
6.7783666, 0.019821785, 21.752554;
0.36275601, 1.5230988, 6.1631775;
6.312263, 0.70509899, 20.544077;
-0.69763142, 1.2937499, 24.652748;
9.5828495, -1.1410632, 123.26447;
-9.2302332, 1.4030573, 19.462811;
1.6566004, 1.3183355, 12.326447;
-9.7503719, 1.3108668, 16.078112;
-15.151987, -0.30765343, 30.816673;
-2.818423, -0.66005921, 61.632225;
6.6366587, -1.4184015, 15.408057;
6.3322215, 0.086505964, 52.826603;
2.5592558, 1.3298782, 10.271922;
4.0725718, 1.3447624, 9.9944153;
4.9024119, -2.9934289, 30.815611;
2.7111707, -5.8293562, 33.61758;
7.403317, -3.8229594, 23.111641;
-3.6971531, 1.5242498, 12.326447;
-17.574755, -3.0676425, 23.112087;
-2.2533178, 1.4626628, 16.077972;
8.23071, -3.5249889, 26.413258;
-11.98974, -6.4411888, 26.41415;
1.6930764, -7.3522606, 36.978962;
-3.8994191, 1.5282874, 14.79174;
-3.7044075, 0.17482722, 73.958679;
6.2851496, 1.2059673, 14.791491;
9.7682772, -3.1345158, 19.462406;
5.8594623, 0.4103809, 13.696052;
-12.044126, 0.89365005, 28.445644;
5.3248596, -3.4243586, 21.752554;
-11.624347, -0.64250302, 52.827629;
-11.976958, -6.5398507, 73.959839;
12.497601, -2.0972431, 21.752552;
-2.2263103, 1.5530555, 11.92882;
10.569036, -3.0852211, 18.489288;
9.4011831, -1.5855764, 28.445646;
-9.4744415, 1.4810051, 20.544077;
-9.7051449, 1.2117313, 16.80879;
7.4340334, -2.1258218, 123.26177;
-15.634634, 1.0600318, 26.413815;
7.8166203, -0.10059228, 36.978539;
18.533512, -3.7463305, 73.955643;
13.563814, -3.2180254, 26.413115;
7.1254354, -2.1918743, 21.750896;
7.6550126, -1.0405474, 36.976173;
-1.1604656, -11.705323, 123.26447;
-34.781414, -3.6594048, 61.635075;
-20.186781, 0.86191028, 61.632229;
-7.249701, 0.60455972, 73.958672;
3.7417026, -4.618577, 26.413813;
-3.8585756, 1.2899698, 19.462809;
0.092922516, -3.0211222, 123.26354;
-27.224382, 1.7154291, 33.618515;
4.869966, 1.2291124, 16.077972;
3.0791214, -4.3333321, 28.445278;
3.6345799, 1.3198718, 11.556043;
11.379731, -3.1345127, 19.462387;
13.522063, -1.1410632, 123.26446;
9.7038288, -4.5237875, 46.224174;
-17.809763, -3.2019334, 23.112087;
6.4965491, -4.5781012, 23.11043;
4.9919372, -1.3819568, 30.815603;
-24.968025, -28.132812, 369.7934;
9.4468603, -2.808722, 61.632229;
32.367134, -2.3828893, 184.88399;
-8.6807852, -0.4248423, 123.26447;
0.67218435, 1.3286146, 26.413813;
6.4041166, 1.1203386, 19.462809;
-102.87573, -9.602335, 369.8602;
-8.9329319, 1.1006689, 16.078098;
10.16637, 0.50029767, 18.48967;
-12.719324, -6.1536846, 26.415264;
-10.703639, -6.8967395, 52.828312;
7.4540219, 0.52048278, 30.816113;
-13.741061, -3.9278927, 61.633381;
34.648201, -3.9161482, 123.25587;
9.0117683, 1.0859511, 12.751496;
4.4945126, 1.2370008, 8.2176313;
6.2344851, 1.1203386, 19.462809;
6.2122512, -3.0465817, 46.224171;
9.3495483, -0.40600464, 17.608856;
12.905653, -4.4549637, 18.489279;
1.9945707, 1.3729289, 7.8678651;
-14.752146, 0.82981783, 26.413813;
5.4736485, 0.84786987, 46.224171;
11.267335, -3.6760907, 18.489285;
-11.011114, 1.1002522, 28.445644;
-15.94313, -8.0756855, 33.61758;
5.1910758, 1.3700546, 7.1114111;
-15.593006, 1.1257513, 21.752552;
8.1754694, -3.386517, 23.111616;
1.4873418, 1.5182621, 6.8480258;
6.7372355, 1.158298, 10.272038;
7.5713153, -3.8062451, 23.112087;
2.9659944, 1.1890762, 15.407866;
1.3449103, -0.79204118, 73.955727;
-1.1173919, 0.90666997, 61.631916;
3.550648, 1.3030853, 11.556044;
13.329698, -1.5022302, 46.224178;
4.3902464, 0.29137868, 123.26446]
point_2d
[562, 161;
850, 179;
463, 189;
581, 180;
286, 177;
556, 304;
276, 176;
741, 173;
813, 180;
593, 158;
459, 88;
276, 151;
567, 172;
634, 36;
677, 52;
637, 163;
346, 154;
578, 189;
893, 73;
568, 182;
599, 189;
406, 182;
815, 202;
562, 197;
767, 130;
609, 144;
569, 303;
261, 133;
410, 155;
955, 56;
786, 82;
469, 158;
920, 35;
825, 157;
603, 156;
542, 303;
432, 156;
478, 138;
613, 83;
735, 233;
604, 130;
785, 138;
582, 302;
497, 137;
576, 212;
741, 154;
789, 170;
258, 198;
633, 116;
532, 200;
477, 185;
314, 205;
629, 89;
1011, 32;
123, 203;
932, 45;
780, 65;
309, 141;
386, 151;
564, 128;
310, 152;
279, 198;
586, 130;
626, 44;
779, 39;
483, 195;
107, 153;
805, 38;
316, 176;
706, 297;
487, 115;
604, 167;
567, 143;
579, 145;
939, 106;
810, 160;
466, 274;
299, 141;
151, 153;
685, 76;
268, 212;
136, 193;
892, 42;
289, 163;
593, 181;
489, 128;
296, 208;
593, 96;
789, 160;
737, 111;
794, 126;
613, 70;
889, 26;
738, 57;
167, 241;
684, 179;
741, 208;
582, 105;
639, 147;
323, 157;
748, 199;
797, 86;
240, 217;
782, 228;
302, 152;
258, 151;
276, 188;
596, 308;
663, 202;
137, 210;
698, 125;
418, 241;
236, 200;
888, 180;
682, 235;
701, 286;
978, 37;
1051, 153;
1020, 272;
121, 194;
432, 182;
669, 90;
453, 194;
467, 141;
302, 161;
435, 94;
775, 76;
283, 136;
816, 117;
554, 130;
759, 280;
708, 93;
368, 150;
526, 130;
125, 257;
639, 182;
543, 128;
775, 27;
317, 189;
585, 155;
563, 278;
143, 251;
752, 103;
755, 197;
479, 117;
303, 189;
790, 42;
650, 44;
736, 46;
259, 43;
814, 191;
718, 134;
817, 36;
599, 202;
374, 184;
781, 180;
609, 313;
778, 201;
562, 213;
633, 173;
277, 224;
662, 244;
197, 229;
258, 171;
550, 173;
849, 120;
660, 180;
727, 255;
822, 259;
685, 116;
637, 66;
787, 73;
372, 269;
85, 89;
492, 236;
783, 93;
283, 18;
611, 48;
397, 253;
544, 182;
845, 230;
901, 77;
845, 198;
301, 199;
746, 73;
432, 171;
471, 119;
948, 118;
466, 256;
944, 75;
795, 143;
286, 224;
212, 224;
621, 168;
194, 203;
720, 178;
750, 146;
915, 100;
791, 115;
717, 161;
574, 114;
203, 138;
361, 188;
512, 182;
673, 65;
442, 227;
581, 167;
51, 211;
770, 227;
651, 79;
769, 248;
953, 79;
655, 173;
715, 110;
85, 89;
762, 51;
689, 150;
534, 128;
684, 150;
700, 171;
531, 177;
596, 212;
790, 216;
389, 161;
224, 220;
930, 196;
265, 24;
444, 91;
739, 191;
431, 137;
771, 158;
1010, 231;
899, 267;
784, 216;
670, 135;
922, 165;
1023, 27;
724, 278;
215, 199;
659, 191;
967, 54;
314, 205;
270, 22;
991, 287;
116, 211;
808, 85;
746, 345;
971, 247;
787, 73;
708, 226;
593, 175;
568, 189;
769, 248;
771, 158;
604, 181]
Intrinsics
[688.4158375072479, 1.530057988187498e-314, 581.4810475816727;
0, 688.4158375072479, 177.3726883200407;
0, 0, 1]
distortion
const cv::Mat distortion = cv::Mat::zeros(4, 1, CV_64FC1);