🚀 The feature, motivation and pitch

Motivation

• Saves SM
• Reduces interference with the overlapping computation (e.g. GEMM)
• Micro benchmark Add benchmark for matmul + all-gather, with CE option #172714 showed 15% end-to-end speedup when the all-gather is overlapped with GEMM, compared to non-CE case.

Recipe

• Allocate all-gather buffer from symmetric memory
• Init NCCL with zero-CTA policy

Alternatives

No response

Additional context

No response

cc @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @msaroufim @dcci @aditvenk @xmfan @zhaojuanmao @mrshenli @rohan-varma @chauhang @mori360 @ppwwyyxx