Tensorboard logger logs to same directory when trial is pruned using hp_search and optuna #16535
Description
Environment info
transformersversion: 4.11.3- Platform: Linux
- Python version: 3.8.8
Who can help
Information
I am running a hyperparameter search on various models using optuna on a modified GLUE task. Whenever optuna prunes a trial, the following trial is logged into the same logging directory and therefore doesn't show up in Tensorboard. As far as I can tell, the immediate easy fix would be to call the on_train_end callback when the trial is to be pruned. This would reset the tensorboard logger and has worked for me in testing. I'm not sure if it has any unwanted side effects, though. I've opened a Pull Request fixing the issue.