💊 CircleCI build failures summary and remediations

As of commit e1b6094:

None of the build failures appear to be your fault.

• 1/1 broken upstream at merge base 8b6a898 since Feb 18

  Please rebase on the viable/strict branch (expand for instructions)

  Since your merge base is older than viable/strict, run these commands:

  git fetch origin viable/strict
  git rebase viable/strict
  

  Check out the recency history of this "viable master" tracking branch.

Detailed failure analysis

One may explore the probable reasons each build failed interactively on the Dr. CI website.

🚧 1 upstream failure recognized by patterns:

These builds matched patterns, but were probably caused by upstream breakages:

• pytorch_windows_vs2019_py36_cuda10.1_build from Feb 18 until Feb 19

Rebased on #33376 to fix conflicts. Unfortunately, that's ahead of upstream/viable/strict so the tests will fail for the time being.

The test won't trigger on CircleCI, but it would be good to add the ">4G CUDA tensor" test that would actually trigger this.

This seems to irrecoverably fail the XLA build:

Feb 19 22:14:08 ======================================================================
Feb 19 22:14:08 ERROR [4.475s]: test_minmax_large_axis_xla (__main__.TestTorchDeviceTypeXLA)
Feb 19 22:14:08 ----------------------------------------------------------------------
Feb 19 22:14:08 Traceback (most recent call last):
Feb 19 22:14:08   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_device_type.py", line 181, in instantiated_test
Feb 19 22:14:08     return test(self, device_arg)
Feb 19 22:14:08   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 382, in wrapper
Feb 19 22:14:08     fn(*args, **kwargs)
Feb 19 22:14:08   File "/var/lib/jenkins/workspace/xla/test/../../test/test_torch.py", line 8526, in test_minmax_large_axis
Feb 19 22:14:08     self.assertEqual(idx, x.shape[0] - 1)
Feb 19 22:14:08   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_device_type.py", line 336, in assertEqual
Feb 19 22:14:08     return DeviceTypeTestBase.assertEqual(self, x, y, prec, message, allow_inf)
Feb 19 22:14:08   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 784, in assertEqual
Feb 19 22:14:08     self.assertEqual(x.item(), y, prec=prec, message=message,
Feb 19 22:14:08 RuntimeError: Resource exhausted: From /job:localservice/replica:0/task:0:
Feb 19 22:14:08 Failed to allocate request for 33.00GiB (35433480192B) on device ordinal 0
Feb 19 22:14:08 	 [[{{node XRTExecute}}]]
Feb 19 22:14:08 Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Feb 19 22:14:08 

We should follow previous practice for large allocation and use the @unittest.skipIf(not TEST_LARGE_TENSOR, "not enough memory") decorator to turn this off when large memory tests are not requested.

@ezyang I can only see TEST_LARGE_TENSOR defined for test_cuda.py. Also, I can't see how a global variable would work here since the available memory depends on the device.

Would it make sense to report the apparent memory leak to XLA and just add a test skip to XLA for the time being?

Well, you can still do something similar to what the test does in the end, which is something like torch.cuda.get_device_properties(0).total_memory >= 12e9. It kind of makes sense that one size fits all as the majority of all big tensor bugs have to do with 32-bit indexing. And yes, I'd make this test run only on CUDA.

Okay, moved the test to test_cuda.py and am now guarding it with TEST_LARGE_TENSOR.

@ezyang merged this pull request in c882425.