I'm not so sure about the purpose of this table.

If it's for developers, it misses many key optimizations such as double buffering.

If it's for the user, do you expect users to proactively tune these configs? Many if not all of them should be by default on.

These are MLM arguments, they should definitely not be turned on by default. Default should be DDP.

These are the bare minimum arguments someone would want to turn on to use Megatron-FSDP.

Technically, very recent PR's make it so that the following are optional:

• --ckpt-format fsdp_dtensor
• --use-distributed-optimizer
• --fsdp-manual-registration

but I need to let them stew for a bit before I document this. Plus, they are informative. We should be able to soon reduce the number of configs needed to use FSDP.

Got it.

To make sure I understood you, there are three mutually exclusive ways to enable MFSDP:

1. MLM
2. FullyShardedDataParallel
3. fully_shard
   Here you are listing the flags for just the first way. Correct?

Yes, except it's more like:

• MLM / FullyShardedDataParallel (Adapter, equivalent to MCore's DDP wrapper.)
  • We might still need an adapter like this, even after the rewrite, though we simultaneously want something more modular with MiMo, so we need to code it out and see what it looks like.
• fully_shard

Both initialize MegatronFSDP.

Could the doc somehow reflect that? To a new user or developer like me, it's not immediately clear that there are two disjoint ways of using MFSDP and which configs are for which way. Maybe the doc can have an API section where we talk about FullyShardedDataParallel (including its configs) and fully_shard (including its configs) in separate subsections. Then, the doc can talk about optimizations and implementation details.

This will also clarify my other question, which might not be obvious to a new user. I think FullyShardedDataParallel is deeply integrated to MCore but fully_shard is not (in fact designed not to be).

I think I'll add another column to our tables that clarifies fully-shard vs. MCore arguments, I really want this current design where the explanation is very close to the arguments, and the arguments are at the beginning of every section.

I think we should document the following zero-copy optimizations we implement:

1. Eliminate Parameter Copies Around Communication: Avoid redundant parameter copies before and after collectives, while remaining compatible with selected DTensor features such as torch distributed. checkpointing (Add to the Advanced Bucketing section.)
2. Zero-Copy Communication via NCCL-UBR: Enable direct communication into NCCL-managed memory through NCCL User Buffer Registration (NCCL-UBR), achieving true zero-copy data movement. (Add to the Optimized Communication & SM Utilization via SHARP section.)
3. Gradient Copy Avoidance with TransformerEngine: Leverage TransformerEngine’s fuse_wgrad_accumulation to eliminate intermediate gradient copies during backward pass. (to be documented under the TransformerEngine section.)

Could we also add support for Hybrid FSDP + outer-dp operates in ZeRO-1 mode? This would provide a more flexible and efficient trade-off between memory footprint and communication overhead.

Note on Communication Overlap: AG/RS overlap with compute does not demonstrate a clear advantage over other ZeRO-DP approaches in practice. This feature may be omitted unless further differentiation is demonstrated.