🐛 Bug

To Reproduce

Steps to reproduce the behavior:

1. python3.6 test/test_cuda.py TestCuda.test_caching_pinned_memory_multi_gpu

Expected behavior

The test should pass.

Current Output:
Sometimes the test fails with the following error

======================================================================
FAIL: test_caching_pinned_memory_multi_gpu (main.TestCuda)

Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1396, in wrapper
method(*args, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1396, in wrapper
method(*args, **kwargs)
File "test_cuda.py", line 1397, in test_caching_pinned_memory_multi_gpu
self.assertNotEqual(t.data_ptr(), ptr, msg='allocation re-used too soon')
File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1962, in assertNotEqual
self.assertEqual(x, y, msg, atol=atol, rtol=rtol, **kwargs)
AssertionError: AssertionError not raised : allocation re-used too soon

The test makes an assumption that a pinned memory which has been freed, will not be re-used very soon.
This is not well defined and does not seem to follow any standard time.
Sometimes the memory is freed and re-used early.

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @ngimel