[GSoC23] Support of mixed model in stereo calibration and multi-view calibration by lpanaf · Pull Request #24052 · opencv/opencv

lpanaf · 2023-07-24T18:30:40Z

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

Summary

This is the major pull request for the project "Multi-camera calibration part 2" in GSoC 2023. This links to the final report of the project. For details of how the project was extended can be found there.

The major contribution is to add the support for a mixed model of pinhole and fisheye camera models, and the support of calibrating cameras without constraints of overlapping field of view and complete.

The documentation is also improved with debugging advice on multi-camera calibration added. This link and this correspond to the documentation for the implemented registerCameras. This links to the documentation of the multiview calibration funtion. As for the tutorial, it can be found here.

Besides, some self captured data can be found in the opencv_extra (currently in pull request 1089).

… API of stereo calibration

asmorkalov · 2023-08-02T07:08:13Z

modules/calib/src/calibration.cpp

+    {
+        int cn = rvecs.channels();
+        int depth = rvecs.depth();
+        if( (depth != CV_32F && depth != CV_64F) ||


Use CV_Assert here.

asmorkalov · 2023-08-02T07:08:51Z

modules/calib/src/calibration.cpp

+    {
+        int cn = tvecs.channels();
+        int depth = tvecs.depth();
+        if( (depth != CV_32F && depth != CV_64F) ||


asmorkalov · 2023-08-02T07:30:36Z

modules/calib/include/opencv2/calib.hpp

                                     TermCriteria criteria = TermCriteria(TermCriteria::COUNT+TermCriteria::EPS, 30, 1e-6) );

+// TODO: add documentation for this
+CV_EXPORTS_AS(stereoCalibrateExtrinsicExtended)  double stereoExtrinsicCalibrate( InputArrayOfArrays objectPoints,


Stereo is bad choice for the function name:

You cannot get real stereo if cameras have different type.

The function just registers two cameras, but does not run intrinsics tuning.

I propose to name it registerCameras.

asmorkalov · 2023-08-02T07:32:13Z

modules/calib/src/calibration.cpp

+                cv::fisheye::undistortPoints(imgpt_ik, imagePointsNormalized, cameraMatrix, distCoeffs, noArray(), cameraMatrix);
+                solvePnP(objpt_i, imagePointsNormalized, cameraMatrix, noArray(), rv, T[k], false, SOLVEPNP_ITERATIVE);


@ivashmak @lpanaf I propose to add fisheye overload for solvePnP to namespace fisheye.

Do you want to make a separate function within fisheye namespace that takes as input distorted points from fisheye camera, internally undistorts them and runs the PnP solver?

In general - yes. We have solvePnP that expects pinhole model. Let's introduce fisheye::solvePnP that expects fisheye camera model internally. The first implementation may be done with just undistort->solvePnP, if it produces reasonable result.

Yes, that makes sense. It looks like just a combination of two functions. @lpanaf will you do it since it is your pull request, or I can try to push here too?

I can add it, no problem :)

asmorkalov · 2023-08-03T13:11:07Z

modules/calib/include/opencv2/calib.hpp

+CV_EXPORTS_AS(stereoCalibrateExtrinsicExtended)  double stereoExtrinsicCalibrate( InputArrayOfArrays objectPoints,
+                                     InputArrayOfArrays imagePoints1, InputArrayOfArrays imagePoints2,
+                                     InputOutputArray cameraMatrix1, InputOutputArray distCoeffs1,
+                                     bool isFisheye1,


It makes sense to rename the parameter to cameraModel and use int/enum. It's more future proof.

This is reflected in the new code

asmorkalov · 2023-08-03T13:11:52Z

@lpanaf Please rebase the PR on top of current 5.x. Last merge introduced some conflicts.

asmorkalov · 2024-02-13T15:07:12Z

modules/calib/test/test_multiview_calib.cpp

        return cv::Mat(R_z * R_y * R_x);
    };
-    const cv::Size board_size (5,4);
+    const cv::Size board_size (9, 7);


Need to split the test on 3 cases:

Pinhole only

Fisheye only

Mixed.

asmorkalov · 2024-02-13T15:08:49Z

samples/cpp/multiview_calibration_sample.cpp

-            if (img_file.empty())
-                break;
-            std::cout << img_file << "\n";
+            if (img_file.empty()){


asmorkalov · 2024-02-13T15:11:41Z

samples/cpp/multiview_calibration_sample.cpp

            }
+            else if (pattern_type == "charuco")
+            {
+                cv::cvtColor(img, img, cv::COLOR_BGR2GRAY);


Not needed for Charuco.

asmorkalov · 2024-02-14T08:46:16Z

modules/calib/test/test_cameracalibration.cpp


 ///////////////////////////////////////////////////////////////////////////////////////////////////

+///////////////////////////////// Register Cmeras /////////////////////////////////////


asmorkalov · 2024-02-14T10:31:16Z

modules/calib/test/test_fisheye.cpp

-    cv::Rodrigues(Rs[1], theR);

-    EXPECT_MAT_NEAR(theR, R_correct, 1e-2);
+    EXPECT_MAT_NEAR(Rs[1], R_correct, 1e-2);


asmorkalov · 2024-02-14T10:32:17Z

modules/calib/test/test_cameracalibration.cpp

-    cv::Rodrigues(Rs[1], R);
+    Rs[1].copyTo(R);


modules/calib/src/fisheye.cpp

solvePnP implementation for Fisheye camera model #25028 Credits to Linfei Pan Extracted from #24052 **Warning:** The patch changes Obj-C generator behaviour and adds "fisheye_" prefix for all ObjC functions from namespace. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [ ] The feature is well documented and sample code can be built with the project CMake Co-authored-by: lpanaf <linpan@student.ethz.ch> Co-authored-by: Vadim Levin <vadim.levin@xperience.ai>

asmorkalov · 2024-02-21T09:37:44Z

modules/calib/src/calibration.cpp

+                   ((points.rows == pointsTotal[k] && points.cols*cn == 2) ||
+                   (points.rows == 1 && points.cols == pointsTotal[k] && cn == 2)));
+
+        A[k] = Matx33d::eye();


Looks like redundant. A[k] is always initialized with the following copyTo.

asmorkalov · 2024-02-21T09:38:36Z

modules/calib/src/calibration.cpp

+                solvePnP(objpt_i, imgpt_ik, A[k], tdists[k], rv, T[k], false, SOLVEPNP_ITERATIVE );
+            else if (cameraModels[k] == CALIB_MODEL_FISHEYE){
+                fisheye::solvePnP(objpt_i, imgpt_ik, A[k], tdists[k], rv, T[k], false, SOLVEPNP_ITERATIVE );
+            }


Need assert in else branch.

asmorkalov · 2024-02-21T09:39:46Z

modules/calib/src/calibration.cpp

+        if( matR.rows == 3 && matR.cols == 3 )
+            Rodrigues(matR, R);
+        else
+            matR.convertTo(R, CV_64F);


Add assert on size here.

asmorkalov · 2024-02-21T09:43:20Z

modules/calib/src/calibration.cpp

+            }
+            else if (rvecs.rows * rvecs.cols * rvecs.channels() == nimages * 3 )
+            {
+                Mat(Mat(srcR).t()).convertTo(r1d.row(i), rvecs.depth());


Sounds strange.

asmorkalov · 2024-02-21T09:43:29Z

modules/calib/src/calibration.cpp

+        if( !tvecs.empty() )
+        {
+            Vec3d srcT(param[idx + 3], param[idx + 4], param[idx + 5]);
+            Mat(Mat(srcT).t()).convertTo(t1d.row(i), tvecs.depth());


sounds strange.

modules/calib/include/opencv2/calib.hpp

RegisterCameras function for heterogenious cameras pair #25061 Credits to Linfei Pan Extracted from #24052 ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [ ] The feature is well documented and sample code can be built with the project CMake --------- Co-authored-by: lpanaf <linpan@student.ethz.ch>

asmorkalov · 2024-12-07T07:27:52Z

Remaining pieces:

Case when the cameras see different parts of the same board.

lpanaf added 18 commits June 2, 2023 13:53

minor typo fixing and missing cam_ids added

0659d80

Merge branch '5.x' of github.com:lpanaf/opencv into 5.x

ea07783

lm for multi-calib updated and the mixed model supported

7ae8e0e

mixed model of stereoCalibrate implemented. Some TODOs there

2920f9b

clean up the code of distort pinhole

83cd489

Merge remote-tracking branch 'upstream/5.x' into 5.x

5b50e9c

prepare for pr

8bb8d73

add a flag to use extrinsic guess for multi-calib

4e4a003

allow for more interfaces for stereo calibration

5f5aaf5

change to allow a vairant number of observation per image. Change the…

f8e4957

… API of stereo calibration

remove the redudant function of stereoCalibrate by undistortion

4d877ad

only perform the final optimization if extrinsic guess available

3d80429

Merge remote-tracking branch 'upstream/5.x' into 5.x

6e1503d

clean the redundant params

c510e27

remove the redudant function

f595932

remove the jacobian of weighting now

36c3d45

keep the gitignore to be the same

a9b3f76

git fix the gitignore

42e8781

asmorkalov requested review from asmorkalov and vpisarev July 27, 2023 09:29

asmorkalov added category: calib3d GSoC pr: needs test New functionality requires minimal tests set labels Jul 27, 2023

asmorkalov requested changes Aug 2, 2023

View reviewed changes

asmorkalov reviewed Aug 3, 2023

View reviewed changes

lpanaf added 4 commits August 8, 2023 15:41

minor bugs fixed

c5dff22

python example with support of charuco added. Comparison with GT added

c825cc4

minor fix

5ce92ad

incorporate suggested modifications

53b6330

lpanaf and others added 7 commits August 28, 2023 14:42

refine the python pipeline

7c78712

fix the wording error of the file

a65779a

fisheye initializatio fixed

548a5b4

code example for charuco board detection added

5a49c83

add degeneracy check for no points

74d08b6

add source reference to the test data

2a8f403

Merge branch '5.x' into mixed_model

6bc3a42

asmorkalov reviewed Feb 14, 2024

View reviewed changes

asmorkalov added 2 commits February 21, 2024 11:53

Merge branch '5.x' into mixed_model

10f6f76

Fixed double fisheye::solvePnP integration after 5.x merge.

3632187

asmorkalov mentioned this pull request Feb 21, 2024

RegisterCameras function for heterogenious cameras pair #25061

Merged

6 tasks

asmorkalov reviewed Feb 21, 2024

View reviewed changes

asmorkalov mentioned this pull request Feb 21, 2024

Ground truth check and Charuco support in multiview_calibration.py #25063

Merged

6 tasks

asmorkalov added 3 commits February 26, 2024 16:41

Merge branch '5.x' into mixed_model

9ce8d19

Code review fixes.

43c4fb8

Merge branch '5.x' into mixed_model

5dc9dd8

asmorkalov mentioned this pull request Sep 24, 2024

Multiview calibration pipeline documentation update #26183

Merged

6 tasks

Merge branch '5.x' into mixed_model

5771a30

asmorkalov force-pushed the mixed_model branch from 0b383c4 to 5771a30 Compare September 24, 2024 10:57

Restructured multi-view calibration test for further extension.

a9ec93e

This was referenced Sep 28, 2024

RegisterCameras test for heterogenious pinhone + fisheye setup. #26213

Merged

Heterogenious cameras support in multiview calibration. #26231

Merged

		cv::fisheye::undistortPoints(imgpt_ik, imagePointsNormalized, cameraMatrix, distCoeffs, noArray(), cameraMatrix);
		solvePnP(objpt_i, imagePointsNormalized, cameraMatrix, noArray(), rv, T[k], false, SOLVEPNP_ITERATIVE);


		///////////////////////////////////////////////////////////////////////////////////////////////////

		///////////////////////////////// Register Cmeras /////////////////////////////////////

Uh oh!

Conversation

lpanaf commented Jul 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asmorkalov commented Aug 3, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asmorkalov commented Dec 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lpanaf commented Jul 24, 2023 •

edited

Loading