System information (version)

• OpenCV => 4.5.1
• Operating System / Platform => Ubuntu 64 Bit
• Compiler => gcc

Detailed description

The standard deviation calculation of calibrateCamera is returning NaN for all standard deviations in all cases when provided with comparably low numbers of points.
This is due to this line, where the divisor and thus sigma2 can become negative. The variable total usually is nimages*maxPoints or less, nparams = nimages*6+18 (for standard calibration method), and nparams_nz < nparams.
For the case total < nparams_nz < nparams the resulting sigma2 is negative.
This can (and nearly always does) happen when maxPoints = 6 so total/nparams < 1.
This is likely also why it has been unnoticed so far.

Changing the total to nparams yields seemingly correct results. EDIT: Only for my case where maxPoints = 6! This might not be a general or even correct fix!

I've also opened a forum question previously, although I'm positive it is a bug.

Steps to reproduce

1. Generate a sample set of 22 6-point markers
2. Call calibrateCameraRO, requesting the standard deviations to be calculated

• One sample run yielded the following variable values: total=132 (22*6), nparams=150(22*6+18), nparams_nz=139

3. Observe the returned standard deviations to all be NaN

An example implementation can be found in this larger project (calibrateCameraRO call here).

Issue submission checklist

• [ X] I report the issue, it's not a question
• [X ] I checked the problem with documentation, FAQ, open issues,
  forum.opencv.org, Stack Overflow, etc and have not found solution
• [ X] I updated to latest OpenCV version and the issue is still there
• [ X] There is reproducer code and related data files: videos, images, onnx, etc