right now, process group holds a vector of share_ptrs of nccl communicators, when process group is destroyed, the nccl communicators may be still held by pending/stuck processGroup::work.
Ideally all nccl communicators should be aborted explicitly when process group is destroyed, otherwise pending nccl kernels may block any CUDA op to run even after destroying process group, this is not good for failure recovery
cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar
right now, process group holds a vector of share_ptrs of nccl communicators, when process group is destroyed, the nccl communicators may be still held by pending/stuck processGroup::work.
Ideally all nccl communicators should be aborted explicitly when process group is destroyed, otherwise pending nccl kernels may block any CUDA op to run even after destroying process group, this is not good for failure recovery
cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar