This test is still failing.
What has changed in numpy/scipy?

It's this PR that changed the default behavior of cond in pinv2: scipy/scipy#10067

I changed the tol for the failing estimator, CCA. This fixed the issue locally, lets see what our CI thinks.

Hmm, I'm surprised to see something like

Max absolute difference: 0.06155424
Max relative difference: 1.

Is there a bug? Will be grateful if you can briefly explain what's happening here.

I quickly went through the code, it's strange that scaling the same data for multiple times will introduce such a big difference?

This is likely a 32-bit numerical issue, since the tests pass for the other environments.

When I set the tolerance too low, it went below np.finfo('f').eps=1e-07, which caused the 32 bit float comparisons in _nipals_twoblocks_inner_loop to fail.

Heading to bed and thanks for your hard work here :)
Honestly I'm OK to skip the test on 32-bit machines and keep the issue open. This does not seem to be a serious issue.

Quick recap of our pair debugging sesh with @thomasjpfan:

• it's not just windows or 32 bits: we can reproduce on linux and macos, using scipy 1.3
• it's definitely caused by MAINT: Fix the cutoff value inconsistency for pinv2 and pinvh scipy/scipy#10067 which changes the defualt cond param of pinv2: small eigen values (smaller than cond) would be ignored before. On 1.3, they're not ignored, causing some extremely large values in the inverse matrix Y_pinv (since pinv2 would divide by these small eigen values).

There are 2 solutions:

• Not use the default cond in our calls to pinv2
• Change the tolerance in the test. We went for this solution

@NicolasHug @thomasjpfan tests are still failing

This error stems from how _nipals_twoblocks_inner_loop calculates y_weights, when dealing with small numbers with different signs:

X = np.array([[1.0, 2.0], 
              [4.0, 3.0]])

# negative y[1,0]
y_n = np.array([[1.0e-15, 1.0],
              [-1.0e-14, 2.0]])

# positive y[1,0]
y_p = np.array([[1.0e-15, 1.0],
               [1.0e-14, 2.0]])

_nipals_twoblocks_inner_loop(X, y_n, mode="B")
# array([[-4.06091728e+12], [1.24670160e+00]])

_nipals_twoblocks_inner_loop(X, y_p, mode="B")
# array([[4.22666900e+12], [1.23841402e+00]])

The PR fixes this issue by replacing the small values with zero.

So changing cond in the call to pinv2 doesn't work?

I think we would need two conditions, one for single precision and another for double precision. (The double precision would need to be scaled up more). This was how it was done in pinv2 before scipy 1.13.0.

Does the eigenvalue conditioning issue relate to #12145?

This is related. The cond parameter in pinv2 determines if an eigenvalue is "significant". scipy 1.13.0 fixed a bug that made it more inclusive, ie smaller eigenvalues are now considered "significant".

Do the original tests now pass with the small value clipping?

Yes. The value clipping forces the eigenvalue to zero, thus forcing pinv2 to consider it "insignificant".

If the original tests pass, why have we modified them?

Thanks for the fix, @thomasjpfan