Add automatic best model loading to Trainer#7431
Merged
Conversation
Codecov Report
@@ Coverage Diff @@
## master #7431 +/- ##
==========================================
+ Coverage 78.17% 78.71% +0.53%
==========================================
Files 181 181
Lines 35800 35858 +58
==========================================
+ Hits 27986 28224 +238
+ Misses 7814 7634 -180
Continue to review full report at Codecov.
|
julien-c
approved these changes
Sep 28, 2020
src/transformers/trainer.py
Outdated
| checkpoints_sorted[-1], | ||
| checkpoints_sorted[best_model_index], | ||
| ) | ||
| print(checkpoints_sorted) |
Collaborator
Author
There was a problem hiding this comment.
Oopsie, leftover from debug.
LysandreJik
approved these changes
Sep 29, 2020
Member
LysandreJik
left a comment
There was a problem hiding this comment.
Nice! Cool that we allow other nn.Modules than PreTrainedModels now!
Comment on lines
+148
to
+168
| load_best_model_at_end (:obj:`bool`, `optional`, defaults to :obj:`False`): | ||
| Whether or not to load the best model found during training at the end of training. | ||
|
|
||
| .. note:: | ||
|
|
||
| When set to :obj:`True`, the parameters :obj:`save_steps` will be ignored and the model will be saved | ||
| after each evaluation. | ||
| metric_for_best_model (:obj:`str`, `optional`) | ||
| Use in conjunction with :obj:`load_best_model_at_end` to specify the metric to use to compare two different | ||
| models. Must be the name of a metric returned by the evaluation with or without the prefix :obj:`"eval_"`. | ||
| Will default to :obj:`"loss"` if unspecified and :obj:`load_best_model_at_end=True` (to use the evaluation | ||
| loss). | ||
|
|
||
| If you set this value, :obj:`greater_is_better` will defaut to :obj:`True`. Don't forget to set it to | ||
| :obj:`False` if your metric is better when lower. | ||
| greater_is_better (:obj:`bool`, `optional`) | ||
| Use in conjunction with :obj:`load_best_model_at_end` and :obj:`metric_for_best_model` to specify if better | ||
| models should have a greater metric or not. Will default to: | ||
|
|
||
| - :obj:`True` if :obj:`metric_for_best_model` is set to a value that isn't :obj:`"loss"` or | ||
| :obj:`"eval_loss"`. |
Contributor
|
IMO this closes #4186 |
Contributor
Collaborator
Author
|
The best model is not deleted with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR cleans up a bit the part that saves the training state inside
Trainerand adds an API that can track which was the best model during any of the evaluation phases to load it back at the end.When fine-tuning a model on a dataset that can easily overfit the model, it's quite common to have the last model not be the best one (in terms of metrics). This PR adds a
TrainingArgumentnamedload_best_model_at_endthat triggers the following behavior:save_stepsgets ignored and the model is saved every time there is an evaluation (determined byevaluation_strategyandeval_steps)TrainerStateof when the best model was encountered (that state is saved along the checkpoints so it can work with resuming a training)TrainingArgumentsmetric_for_best_model(defaults to the loss) andgreater_is_better(default to False for the loss, True otherwise).In passing I've added some tests of the saving API in Trainer and made sure it can handle both
PreTrainedModeland regularnn.Module(a feature asked in #6901). Both are now tested in the CI, as is the new API.Fixes #6901
Those newly introduced arguments and APIs can then be leveraged to have early stopping supported in
Trainer.