calibrateCamera using k1, k2, k3 should enforce that k1r^2 + k2r^4 +k3*r^6 must be either monotonically increasing or monotonically decreasing as this is a constraint of real lens, or atleast all real lenses for which this (Brown) lens model is used. A marginally harder constraint which could be used instead is that they should all have the same sign.

Current state: this not enforced by the optimizer.

The most common case during which this constraint is violated in practice is when the chessboard observations are not well spread in the image. Note that this is the source of many a failed calibration. The ring pattern, or reflections, which it creates near the edges of the sampled pixels will be familiar to many when the full space where there are pixels are computed as undistortion. In particular this type of failure will fool most people as the reprojection error will be low, and the undistortion will appear reasonable in the center of the image.

Because the optimizer does not support mixed integer inequality constraints, and because such an optimizer will not fix the problem of the poor sampling anyways I propose to add a notice to the calibrateCamera docs which describes the problem. pull requiest incoming.

Note that this problem is so common that the opencv documentation explicity states that k>0 is (typically) barrel distortion. (This comment is explicitly there because people thought failed calibrations where actually successful and wrote the error into the documentation.)

Note there are several more unfullfilled constraints on k4:6 and tangential parameters. The discussion above assumes they are set to zero. Otherwize these more complicated constraints must be fullfilled aswell.