🚀 Feature

Support torch.solve on the GPU

Motivation

In pytorch/audio#768 I asked about the complex support in torch for some linear algebra functions and
@anjali411 said, I should post here my feature request.

Pitch

It would be nice, when the following code would not fail in the last line:

>>> import numpy as np, torch
>>> torch.__version__
'1.7.0.dev20200705+cu101'
>>> t = torch.tensor(np.random.randn(2, 2))
>>> _ = torch.solve(t, t)
>>> _ = torch.solve(t.to(0), t.to(0))
>>> t = torch.tensor(1j * np.random.randn(2, 2))
>>> _ = torch.solve(t, t)
>>> _ = torch.solve(t.to(0), t.to(0))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: "solve_cuda" not implemented for 'ComplexDouble'

Alternatives

Currently, I am using
https://github.com/kamo-naoyuki/pytorch_complex/tree/master/torch_complex
but a native support would be faster, consume less memory and the code would contain fewer hacks to distinguish real and complex tensors.

Additional context

cc @ezyang @anjali411 @dylanbespalko