🚀 Feature

Implementing non-deterministic parallel scatter algorithms for various reduction operations. This issue is a follow-up of #33065 and a prelude to #22378.

Motivation

Read comment #22378 (comment) for more details.

Pitch

A scatter reduction API for addition, subtraction, multiplication and division that will have the following form:

input = torch.zeros(4, 4, device=device)
src = torch.ones(2, 2, device=device)
index = torch.tensor([[1], [2]], device=device, dtype=torch.long)
input.scatter_(0, index, src, reduce="sum")