Raise error and suggestion when using custom optimizer with Fairscale or Deepspeed#16786
Raise error and suggestion when using custom optimizer with Fairscale or Deepspeed#16786sgugger merged 3 commits intohuggingface:mainfrom
Conversation
|
I'm not really in favor of adding an argument to not save the optimizer as there is no point checkpointing if the optimizer is not saved. For the fairscale problem of OOM, there is an option that was detailed in #14542 to use |
|
Oh, I just saw that solution. Maybe the second part where we raise the error is enough for this PR? Do you think it is necessary if (self.sharded_ddp is not None or args.deepspeed) and (self.optimizer is not None or self.lr_scheduler is not None):
raise RuntimeError(
"Passing `optimizers` is not allowed if Fairscale or Deepspeed is enabled."
"You should subclass `Trainer` and override the `create_optimizer_and_scheduler` method."
) |
|
This change I agree with :-) If you want to remove the others, we can merge the PR. |
|
The documentation is not available anymore as the PR was closed or merged. |
|
Yeah, I will remove the others and make a check |
|
There is still one error of quality script, so you'll need to run |
|
Thanks! You have to mark the PR as ready for review, as GitHub won't let me merge a draft PR :-) |
|
Thanks. It is ready for review now. |
… or Deepspeed (huggingface#16786) * optimizer issues related to saving * remove the "optimizer saving" option * reformat using make style
What does this PR do?
1st Issues: OOM when saving the optimizer, #14542
This issue happens in consolidating the optimizer, we add an argument
save_optimizer_stateto give an option on whether we want to save it.2nd issue: Using a custom optimizer has a problem with Fairscale and Deepspeed, #15784
We simply raise an error and warn the user to proceed with different solution.
Fixes # (issue)
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
trainer: @sgugger