🐛 Bug

The GPU memory keep increasing when calling torch.optim.lr_scheduler.ReduceLROnPlateau multiple times. Using torch.optim.lr_scheduler.StepLR does not have such issue.

To Reproduce

Steps to reproduce the behavior:

   while True:
        net = torch.nn.Linear(5000,5000)
        net = net.cuda()
        optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)
        scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau( optimizer,
                'max', patience=args.patience, factor=0.5, verbose=True)
        # scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=30,
        #         gamma=0.1)

Expected behavior

GPU memory keeps increasing until out of memory

Environment

Collecting environment information...
PyTorch version: 1.0.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 14.04.5 LTS
GCC version: (GCC) 4.8.5
CMake version: version 3.13.4

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce GTX TITAN X
GPU 1: GeForce GTX TITAN X
GPU 2: GeForce GTX TITAN X

Nvidia driver version: 384.130
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.15.3
[pip3] torch==1.0.1.post2
[pip3] torchvision==0.1.9
[conda] magma-cuda80 2.1.0 5 soumith
[conda] mkl 2019.1 144
[conda] mkl_fft 1.0.10 py36_0 conda-forge
[conda] mkl_random 1.0.2 py36_0 conda-forge
[conda] pytorch-pretrained-bert 0.3.0 pypi_0 pypi
[conda] tensorflow-base 1.12.0 mkl_py36h3c3e929_0
[conda] torch 1.0.0 pypi_0 pypi
[conda] torchtext 0.2.3 pypi_0 pypi
[conda] torchvision 0.2.1 py36_1000 conda-forge