System information (version)

• OpenCV => 4.1.0
• Operating System / Platform => Windows 64 Bit
• Compiler => Python 2.7.16 (v2.7.16:413a49145e, Mar 4 2019, 01:37:19) [MSC v.1500 64 bit (AMD64)] on win32

Detailed description

Possible bug in calib3d calibrateCameraRO.

I'm trying to use the new more accurate camera calibration method [182] "More accurate pinhole camera calibration with imperfect planar target." as described in documentation and camera calibration tutorial. When i use calibrateCameraRO for old slyle camera calibration with iFixedPoint = -1 everything work fine. But the same with iFixedPoint = pattern_width-1 fails with an error:

cv2.error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\core\src\matmul.dispatch.cpp:337: error: (-215:Assertion failed) type == B.type() in function 'cv::gemm'

Steps to reproduce

Using samples code samples/python/calibrate.py add this line after call to calibrateCamera at line 100:

    rms, camera_matrix, dist_coefs, rvecs, tvecs, newObjPoints2  = cv.calibrateCameraRO(
                        obj_points, img_points, (w, h), int(pattern_size[0]-1),
                        camera_matrix, dist_coefs, rvecs, tvecs, None, cv.CALIB_USE_INTRINSIC_GUESS | cv.CALIB_USE_LU, 
                        (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30,  2.2204460492503131e-16)  );

Run with opencv sample data set.

Throws error:

cv2.error: OpenCV(4.1.0) C:\projects\opencv-python\opencv\modules\core\src\matmul.dispatch.cpp:337: error: (-215:Assertion failed) type == B.type() in function 'cv::gemm'