Skip to content

[BUG] message Creating fp16 ZeRO stage 3 optimizer logged while the training is a bf16's one #2739

@SaulLu

Description

@SaulLu

Describe the bug

While running a bf16 training with Zero-3 I noticed in the logs the mention :

 Creating fp16 ZeRO stage 3 optimizer

Expected behavior
I would have expected the log to be:

 Creating ZeRO stage 3 optimizer

or

 Creating bf16 ZeRO stage 3 optimizer

Additional context
I think it's just a problem in the logged message but it probably needs to be confirmed

https://github.com/microsoft/DeepSpeed/blob/867da307d047c8122454ac8b0880cd23d38949e7/deepspeed/runtime/engine.py#L1594-L1595

https://github.com/microsoft/DeepSpeed/blob/867da307d047c8122454ac8b0880cd23d38949e7/deepspeed/runtime/engine.py#L1529-L1530

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingtraining

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions