Added initInverseRectificationMap() by IanMaquignaz · Pull Request #20165 · opencv/opencv

IanMaquignaz · 2021-05-26T23:01:38Z

This pull request adds a function to Calib3D to support inverse rectification in procam systems. Essentially, it provides the inverse of initUndistortRectifyMap() such that projectors are properly handled in stereo-rectified projector-camera pairs.

Why is this function needed? Inverse rectification has been incorrectly applied in many works, including mishandling of distortion coefficients. Hopefully this proposed function will help curb this trend.

Once successfully merged, a duplicate function can be added for fisheye calibration.

Pull Request Readiness Checklist

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
[NA] There is reference to original bug report and related work
[?] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

IanMaquignaz · 2021-05-26T23:07:50Z

@alalek I need instruction on how to incorporate a performance test. This can be done with a chessboard image and check the rms error between the original and captured points.

Currently have a sloppy visual image comparison to check my implementation.

'''

// Display test image
cv::Mat img = cv::imread(img_filePath);
cv::imshow("img", img);

// Camera Matrix
cv::Mat cameraMatrix = cv::Mat::eye(3, 3, CV_64F);

// Focal Length
cameraMatrix.at<double>(0, 0) = 1.5393951443032472e+03;
cameraMatrix.at<double>(1, 1) = 1.5400748240626747e+03;

// Focal Point
cameraMatrix.at<double>(0, 2) = 6.7491727003047140e+02;
cameraMatrix.at<double>(1, 2) = 5.1226968329123963e+02;

std::cout << "Camera Matrix=\n" << cameraMatrix << std::endl;

// Distortion
cv::Mat distCoeffs = cv::Mat::zeros(5, 1, CV_64F);

// Distortion (Point Grey)
distCoeffs.at<double>(0, 0) = -3.4134571357400023e-01;
distCoeffs.at<double>(1, 0) = 2.9733267766101856e-01;
distCoeffs.at<double>(2, 0) = 3.6653586399031184e-03;
distCoeffs.at<double>(3, 0) = -3.1960714017365702e-03;
distCoeffs.at<double>(4, 0) = 0.0;

std::cout << "distCoeffs=\n" << distCoeffs << std::endl;

// Rectification
cv::Mat R = cv::Mat::eye(3, 3, CV_64F);
//double scale = 0.05;
//R.at<double>(0, 1) = scale;
//R.at<double>(1, 0) = scale;
//double scale = 1.5;
//R.at<double>(0, 0) = scale;
//R.at<double>(1, 1) = scale;
std::cout << "R=\n" << R << std::endl;

cv::Mat map1_ir, map2_ir;
cv::initInverseRectificationMap( 
    cameraMatrix, 
    distCoeffs,
    R, 
    cameraMatrix,
    img.size(), // Size
    CV_32FC1, // m1type CV_16SC2, CV_32FC1, CV_32FC2
    map1_ir, map2_ir 
);

cv::Mat map1_ur, map2_ur;
cv::initUndistortRectifyMap( 
    cameraMatrix, 
    distCoeffs,
    R.inv(), 
    cameraMatrix,
    img.size(), // Size
    CV_32FC1, // m1type CV_16SC2, CV_32FC1, CV_32FC2
    map1_ur, map2_ur 
);

std::cout << map1_ir(cv::Range(1,10), cv::Range(1,10)) << std::endl;
std::cout << map1_ur(cv::Range(1,10), cv::Range(1,10)) << std::endl;

// project
cv::Mat img_projected;
cv::remap(img, img_projected, map1_ir, map2_ir, cv::INTER_LINEAR);
cv::imshow("img_projected", img_projected);

cv::Mat img_captured;
cv::remap(img_projected, img_captured, map1_ur, map2_ur, cv::INTER_LINEAR);
cv::imshow("img_captured", img_captured);

cv::waitKey();

'''

vpisarev · 2021-05-27T13:59:20Z

@IanMaquignaz, thank you for the contribution!

Can you please consider submit the patch to master branch instead of 3.4? We will freeze 3.x branch after 5.0 release, and even now activity in 3.x is mostly about bug fixes and some compatible with 4.x optimizations, rather than a new functionality.
Can you provide some regression test?

IanMaquignaz · 2021-05-27T15:10:28Z

@vpisarev

1. Can you please consider submit the patch to master branch instead of 3.4? We will freeze 3.x branch after 5.0 release, and even now activity in 3.x is mostly about bug fixes and some compatible with 4.x optimizations, rather than a new functionality.

Yes, not a problem. Please consider updating the contribution docs as I specifically rebased from master to 3.4 per the guidelines.
https://github.com/opencv/opencv/wiki/Branches

2. Can you provide some regression test?

Gladly, just need to know how. Found \modules\calib3d\perf\perf_undistort.cpp but have not found any documentation on how it works.

The easiest validation method is take the image of a chessboard and remap it using cv::initInverseRectificationMap() and cv::initUndistortRectifyMap(). This models light passing backwards and forwards through a lens, returning a result which should match the original chessboard closely. I can do an RMS error of chessboard correspondences between before and after images.

IanMaquignaz · 2021-06-05T15:08:45Z

Hi All, I spent quite a bit of time over the last two days trying to create a fisheye version of initInverseRectificationMap(). Thus far this has been unfruitful, with my solution exhibiting greater distortion than anticipated. It appears that cv::fisheye::undistortPoints() is both greatly impacted and impacting of the camera matrix in ways I don't yet fully understand. At this point in time (or at least for my currently application) I not longer believe initInverseRectificationMap() can be viably applicable to the fisheye camera model.

cv::initInverseRectificationMap() should be good to go. The test case from cv::initUndistortRectifyMap() has been duplicated and altered to validate in the same fashion. No idea what the two failed checks are.... pullrequest.opencv.org shows all green.

IanMaquignaz · 2021-06-05T15:35:20Z

Need me to squash the commits? @vpisarev

…ionMap() Fixed trailing whitespace Update to initInverseRectificationMap documentation for clarity Added test case for initInverseRectificationMap() Updated documentation. Fixed whitespace error in docs Small update to test function Now passes success_error_level final update to inverseRectification documentation

vpisarev · 2021-06-08T01:44:26Z

@IanMaquignaz, thank you for the contribution! 👍

alalek · 2021-06-08T04:40:21Z

@vpisarev Do NOT violate existed merge rules. Again, Merging is not allowed:

during the active (pending) nightly builds. It is about 23:00 - 10:00 UTC. Merging in the middle of validation cycle breaks report on several incomplete parts.
during the opened "Merge" PRs to the same target branch (see here (4.x) Merge 3.4 #20237). You just force recreation of them from the scratch. Please do not waste my time.

alalek

Changes are required, see below.

alalek · 2021-06-08T04:45:40Z

modules/calib3d/perf/perf_undistort.cpp

+    Size size_w_h(512 + 3, 512);
+    Mat k(3, 3, CV_32FC1);
+    Mat d(1, 14, CV_64FC1);
+    Mat dst(size_w_h, CV_32FC2);
+    declare.in(k, d, WARMUP_RNG).out(dst);


Why do we have such non-standard size_w_h?
Why do we have randomized input for camera intrinsics? Prefer to measure the real cases.
Function is not a real-time, not optimized and really slow (30+ ms), so there is not enough reasons to enable it into default set of perf tests.

Test has been disabled in #20247. New unit test is being developed to replace it.

alalek · 2021-06-08T04:52:34Z

modules/calib3d/src/undistort.dispatch.cpp

+    {
+        distCoeffs.create(14, 1, CV_64F);
+        distCoeffs = 0.;
+    }


Assigning distortion matrix probably slowdowns computations below for this path.

Correct. cv::undistortPoints skips distortion entirely if the matrix is empty. Correction has been applied in #20247.

It is unfortunate the same optimization is not applied for R.

alalek · 2021-06-08T04:56:06Z

modules/calib3d/test/test_undistort.cpp


+//------------------------------------------------------
+
+class CV_InitInverseRectificationMapTest : public cvtest::ArrayTest


cvtest::ArrayTest

Adding new tests based on legacy classes should be avoided (they are deprecated since migration on Google Tests, ~10 years ago).

alalek · 2021-06-08T04:58:19Z

modules/calib3d/test/test_undistort.cpp

 TEST(Calib3d_DefaultNewCameraMatrix, accuracy) { CV_DefaultNewCameraMatrixTest test; test.safe_run(); }
 TEST(Calib3d_UndistortPoints, accuracy) { CV_UndistortPointsTest test; test.safe_run(); }
 TEST(Calib3d_InitUndistortRectifyMap, accuracy) { CV_InitUndistortRectifyMapTest test; test.safe_run(); }
+TEST(Calib3d_InitInverseRectificationMap, accuracy) { CV_InitInverseRectificationMapTest test; test.safe_run(); }


This test consumes 33% of all opencv_test_calib3d run (30 seconds from 1:30).
This is not acceptable, test must be reduced to run in 1-2sec.

This test has been disabled in #20247.

alalek · 2021-06-08T05:00:51Z

modules/ts/src/ts_func.cpp

+void initInverseRectificationMap( const Mat& _a0, const Mat& _k0, const Mat& _R0, const Mat& _new_cam0, Size sz, Mat& __mapx, Mat& __mapy, int map_type )
+{


Do NOT touch the TS module!
Module-specific functionality should be moved out of this generic module instead.

Code has been removed and integrated into test_undistort.cpp. Applied in #20247.

alalek · 2021-06-08T05:03:02Z

modules/calib3d/include/opencv2/calib3d.hpp

+CV_EXPORTS_W
+void initInverseRectificationMap( InputArray cameraMatrix, InputArray distCoeffs,
+                           InputArray R, InputArray newCameraMatrix,
+                           Size size, int m1type, OutputArray map1, OutputArray map2 );


Size size

const reference for all input parameters based on non-scalars: const Size& size

Correction has been applied in #20247

IanMaquignaz · 2021-06-08T14:33:15Z

@vpisarev
1. Can you please consider submit the patch to master branch instead of 3.4? We will freeze 3.x branch after 5.0 release, and even now activity in 3.x is mostly about bug fixes and some compatible with 4.x optimizations, rather than a new functionality.
Yes, not a problem. Please consider updating the contribution docs as I specifically rebased from master to 3.4 per the guidelines.
https://github.com/opencv/opencv/wiki/Branches
2. Can you provide some regression test?
Gladly, just need to know how. Found \modules\calib3d\perf\perf_undistort.cpp but have not found any documentation on how it works.

The easiest validation method is take the image of a chessboard and remap it using cv::initInverseRectificationMap() and cv::initUndistortRectifyMap(). This models light passing backwards and forwards through a lens, returning a result which should match the original chessboard closely. I can do an RMS error of chessboard correspondences between before and after images.

@alalek I had no idea what I was doing in developing this unit test. I have not found any documentation nor have received any guidance on how to approach creating this. As my function in essentially the inverse of InitUndistortRectify, I duplicated its test and performance tests.

IanMaquignaz · 2021-06-08T14:35:03Z

@alalek What's the correct procedure here? Can I make the requested changes and push them to this PR, or should I open a new one?

alalek · 2021-06-08T14:43:53Z

@IanMaquignaz Don't worry. This is not your fault. It is reviewer responsibility to provide guidance according to actual development rules.

This PR is merged, so please create a new PR with updates (please add link on this one in PR's description).

IanMaquignaz · 2021-06-08T17:22:31Z

@alalek what's a good unit test to copy and use as a base?

Possibly TEST_F(fisheyeTest, undistortAndDistortImage)? though I frankly don't understand how it validates cv::fisheye::initUndistortRectifyMap().
https://github.com/opencv/opencv/blob/81b897c2914b231b5baf47003dae6b702877453e/modules/calib3d/test/test_fisheye.cpp

The easiest validation method is take the image of a chessboard, and remap it using cv::initInverseRectificationMap() then cv::initUndistortRectifyMap(). This models light passing backwards and forwards through a lens, returning a result which should match the original unaltered chessboard. The chessboard correspondences can be compared for an accuracy metric.

alalek · 2021-06-08T18:26:53Z

TEST_F

Yes something like this, or even TEST(..) if you don't need parameters / class / methods.
No need to randomize multiple input cameras configurations (current test performs 500 iterations). Prefer to use one inputs set to catch regressions (which provides enough code coverage of added code).
Ideally, it should look like TEST(Calib3d_initUndistortRectifyMap, regression_14467) test.

and remap it

In general, avoid using of other OpenCV functions. Prefer to check results of tested function only.

If you want, you can add "full" variant of test (like the current one) for development/optimization purposes, but please keep it disabled (add "DISABLED_" prefix to the name).

vpisarev self-assigned this May 27, 2021

IanMaquignaz force-pushed the inverseRectification branch from cba4188 to e73979d Compare May 28, 2021 00:05

IanMaquignaz changed the base branch from 3.4 to master May 28, 2021 16:53

IanMaquignaz force-pushed the inverseRectification branch from e73979d to b556d4d Compare May 28, 2021 16:54

IanMaquignaz marked this pull request as ready for review June 5, 2021 14:57

IanMaquignaz force-pushed the inverseRectification branch from 501d9ff to 426e9ab Compare June 7, 2021 15:50

IanMaquignaz force-pushed the inverseRectification branch from acd9ad5 to b056314 Compare June 7, 2021 20:02

opencv-pushbot merged commit 1a305ca into opencv:master Jun 8, 2021

alalek reviewed Jun 8, 2021

View reviewed changes

This was referenced Jun 8, 2021

Update to initInverseRectificationMap() #20247

Merged

Added new unit test for cv::initInverseRectificationMap() #20259

Merged

alalek mentioned this pull request Jun 13, 2021

(5.x) Merge 4.x #20266

Merged

alalek mentioned this pull request Oct 15, 2021

(5.x) Merge 4.x #20886

Merged


		//------------------------------------------------------

		class CV_InitInverseRectificationMapTest : public cvtest::ArrayTest

		void initInverseRectificationMap( const Mat& _a0, const Mat& _k0, const Mat& _R0, const Mat& _new_cam0, Size sz, Mat& __mapx, Mat& __mapy, int map_type )
		{

Uh oh!

Conversation

IanMaquignaz commented May 26, 2021

Pull Request Readiness Checklist

Uh oh!

IanMaquignaz commented May 26, 2021

Uh oh!

vpisarev commented May 27, 2021

Uh oh!

IanMaquignaz commented May 27, 2021

Uh oh!

IanMaquignaz commented Jun 5, 2021

Uh oh!

IanMaquignaz commented Jun 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vpisarev commented Jun 8, 2021

Uh oh!

alalek commented Jun 8, 2021

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

IanMaquignaz commented Jun 8, 2021

Uh oh!

IanMaquignaz commented Jun 8, 2021

Uh oh!

alalek commented Jun 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IanMaquignaz commented Jun 8, 2021

Uh oh!

alalek commented Jun 8, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

IanMaquignaz commented Jun 5, 2021 •

edited

Loading

alalek commented Jun 8, 2021 •

edited

Loading