🚀 Feature

As per title, complex numbers should be supported in torch.distributed.

Motivation

Distribute computing support for complex numbers came up in conversations with people at Argonne National Laboratory and Flatiron Institute. Currently, some of them use Uber's Horovod library for distributed computing. The operations that they commonly use are all_reduce and broadcasting operations.

Pitch

1. NCCL only defines floating and integer data types, so one way to get complex working with NCCL is by viewing complex tensors as real using torch.view_as_real(complex_tensor) which returns an equivalent R^2 floating point tensor. Since the view shares the same storage with the original complex tensor, we probably don’t need to convert it back to complex. If needed, torch.view_as_complex can be used to convert the real tensor back to complex tensor.
2. Both torch.view_as_real and torch.view_as_complex are view operations and O(1).

    >>>z=torch.randn(4, dtype=torch.cfloat) 
    >>>torch.view_as_real(z)
    tensor([[-0.4226, 0.5459],
            [ 0.9385, 1.1723],
            [-0.9454, -0.3572],
            [ 0.0624, 1.0193]])

3. Ops that should work with complex:
   1. torch.distributed.all_reduce
   2. torch.distributed.all_gather
4. Testing
   1. https://github.com/pytorch/pytorch/blob/master/test/distributed/test_c10d.py
   2. https://github.com/pytorch/pytorch/blob/master/torch/testing/_internal/distributed/distributed_test.py

cc @ezyang @anjali411 @dylanbespalko @mruberry @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar @jiayisuse @agolynski