System Information

OpeCV python version: 4.8.0
Numpy version: 1.24.3
Operating System / Platform: Ubuntu 22.04
Python version: 3.10.6

Detailed description

RQDecomp3x3 gives garbage for certain input matrices.
As an example, given the following matrix P, it fails to decompose into reconstructing matrices R and Q

import cv2 as cv
import numpy as np

P = np.array([
    [0,0,1],
    [0,1,0],
    [1,0,0]
], dtype=np.float64)
_, R, Q, *_ = cv.RQDecomp3x3(P)
print(R @ Q)
# => array([[0., 0., 0.], [0., 0., 0.], [1., 0., 0.]])

As you can see, the decomposition R and Q does not reconstruct the input matrix P.

Steps to reproduce

See description of problem

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)