🚀 Feature

Reduce the runtime cost tied to activating OSS

Motivation

Right now the speed penalty of using OSS over a couple of nodes, for a typical ClassyVision load is 2-10x.

Pitch

There are a bunch of low hanging fruits, from oss_ddp to broadcast batching which could be used to reduce this impact

Alternatives

Not using OSS I guess