When a distributed backward pass fails on a certain worker, we should inform all other workers that the backward pass has failed and the other workers should try to cancel all the tasks it is executing for that particular backward pass.
cc @ezyang @ssnl @albanD @zou3519 @gqchen @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @aazzolini @xush6528
When a distributed backward pass fails on a certain worker, we should inform all other workers that the backward pass has failed and the other workers should try to cancel all the tasks it is executing for that particular backward pass.
cc @ezyang @ssnl @albanD @zou3519 @gqchen @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @aazzolini @xush6528