The function cv::recoverPose fails when given an inliers mask of type vector<uchar>, which is a valid input/output array. If an equal Matof type CV_U8 is given instead, the function works fine.

This is unfortunate since cv::MatchesInfouses vector<uchar> to store the inliers mask.

I assume the error is due to a bogus assertion in the corresponding method:

CV_Assert(mask.size() == mask1.size());

mask1 is always of type Mat, mask is the given input. In the case described here, cv::Size might be compared to a size_t, never returning true.