Summary

Often, we want to stop training if loss does not improve for a number of epochs. This PR adds a "patience" argument, which is a limit on the number of times we can get a non-improving eval loss before stopping training early.

It is implemented by other NLP frameworks, such as AllenNLP (see trainer.py and metric_tracker.py).

Motivation

This feature allows faster fine-tuning by breaking the training loop early and avoids users the toil of checking metrics on Tensorboard.

Caveats

Often, models are evaluated once per epoch, but run_lm_finetuning.py has an option to evaluate after a set number of model update steps (dictated by --logging_steps if --evaluate_during_training is true). Because of this, I've elected to tie patience to the number of evaluations without improvement in loss.

To-do

• Add tests
• Fix long lines