I have got some times with me (10 runs w/ 10 loops each)

@ssnl @soumith does this look good enough?

The performance speedup is great, @vishwakftw! Could you benchmark some smaller matrices? In particular, I'm reading https://groups.google.com/forum/#!topic/julia-dev/mmgO65i6-fA and from their experience it sounds like there is some value in having a second svd call that uses gesvd, but I'm curious what your benchmarking shows about that.

edit: Oh, I found the discussion in #11174 where @soumith linked to the link I sent. Please ignore my comment :)

I believe this reiterates the importance of shifting to gesdd

Yes, the LAPACK routine dgesdd uses a Divide and Conquer approach at its core, which recurses on smaller and smaller problems; at the smallest level, it uses a QR algorithm which is identical to that of dgesvd.
The only real reason to use dgesvd instead is for accuracy and workspace reasons. The former should only be important if you care about high relative accuracy for tiny singular values

I have got some more times with me (10 runs w/ 1000 loops each)

cc: @zou3519

From this, as the matrix size starts diminishing, the advantage fades away.

@vishwakftw those times look great

BTW, I would also check if the result has any NaN's. I found Intel's gesdd erroneously produced NaNs when gesvd didn't.

I feel like some of the edge cases are likely to come up on more ill-conditioned matrices. For instance here's a particular (very ill-conditioned) matrix that breaks TensorFlow's SVD routine

from scipy import linalg  # for svd
import urllib.request
import numpy as np

url="https://storage.googleapis.com/tensorflow-community-wheels/svd_in"
response = urllib.request.urlopen(url)
body = response.read()
print("Read %d bytes"%(len(body),))
assert len(body) == 15366400
open("svd_in", "wb").write(body)

dtype = np.float32
matrix0 = np.genfromtxt('svd_in',
                        delimiter= ",").astype(dtype)
assert matrix0.shape == (784, 784)
u, s, v = linalg.svd(matrix0)
print("matrix0 any NaNs: %s"% (np.isnan(matrix0).any(),))
print("u had NaNs: %s"% (np.isnan(u).any(),))

@ssnl is this good to go?

@yaroslavvb The example you have given seems to not break the current implementation in this PR. There were no NaNs or infs detected.

@pytorchbot retest this please

@zou3519 @ssnl is this good to go?