Backward kernels are always put on the default stream #7601
Description
Issue description
Streams in THC are thread local, so in backward, since autograd creates its own threads, kernels are always put on the default streams, and don't respect stream that was set previously.
Code example
import torch
a=torch.Tensor(128,512).cuda().uniform_()
model = torch.nn.Linear(512,512).cuda()
fwd_stream = torch.cuda.Stream()
bwd_stream = torch.cuda.Stream()
for i in range(3):
with torch.cuda.stream(fwd_stream):
out = model(a).sum()
with torch.cuda.stream(bwd_stream):
out.backward() #kernels are actually on the default stream
a.detach()
Pytorch, current master