Skip to content

[testing] -D_GLIBCXX_ASSERTIONS#75766

Closed
davidberard98 wants to merge 4 commits intogh/davidberard98/91/basefrom
gh/davidberard98/91/head
Closed

[testing] -D_GLIBCXX_ASSERTIONS#75766
davidberard98 wants to merge 4 commits intogh/davidberard98/91/basefrom
gh/davidberard98/91/head

Conversation

@davidberard98
Copy link
Contributor

@davidberard98 davidberard98 commented Apr 13, 2022 •
edited
Loading

@davidberard98
[testing] -D_GLIBCXX_ASSERTIONS
e63a89f 
curious if it'll catch anything

[ghstack-poisoned]
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Apr 13, 2022 •
edited
Loading

🔗 Helpful links

💊 CI failures summary and remediations

As of commit 449f8ba (more details on the Dr. CI page):

  • 8/8 failures introduced in this PR

🕵️ 8 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages

See GitHub Actions build periodic / linux-xenial-cuda11.3-py3.7-gcc7-debug / test (default, 2, 2, linux.4xlarge.nvidia.gpu) (1/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-04-14T21:00:50.3730182Z RuntimeError: test_jit_cuda_fuser failed! Received signal: SIGIOT 
2022-04-14T21:00:45.8292638Z   test_broadcasting_partition_logic_1 (__main__.TestCudaFuser) ... ok (0.389s)
2022-04-14T21:00:46.2359142Z   test_build_shape_expression_native_dropout (__main__.TestCudaFuser) ... ok (0.407s)
2022-04-14T21:00:47.7085000Z   test_category_rule (__main__.TestCudaFuser) ... ok (1.472s)
2022-04-14T21:00:48.9261637Z   test_channels_last_with_broadcast (__main__.TestCudaFuser) ... ok (1.218s)
2022-04-14T21:00:49.0363533Z   test_chunk (__main__.TestCudaFuser) ... /usr/include/c++/7/bits/stl_vector.h:797: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::size_type) [with _Tp = long int; _Alloc = std::allocator<long int>; std::vector<_Tp, _Alloc>::reference = long int&; std::vector<_Tp, _Alloc>::size_type = long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
2022-04-14T21:00:50.3723898Z Traceback (most recent call last):
2022-04-14T21:00:50.3724286Z   File "test/run_test.py", line 1058, in <module>
2022-04-14T21:00:50.3726873Z     main()
2022-04-14T21:00:50.3727461Z   File "test/run_test.py", line 1036, in main
2022-04-14T21:00:50.3729804Z     raise RuntimeError(err_message)
2022-04-14T21:00:50.3730182Z RuntimeError: test_jit_cuda_fuser failed! Received signal: SIGIOT
2022-04-14T21:00:50.9707340Z + cleanup
2022-04-14T21:00:50.9707674Z + retcode=1
2022-04-14T21:00:50.9707916Z + set +x
2022-04-14T21:00:50.9765361Z ##[error]Process completed with exit code 1.
2022-04-14T21:00:50.9819776Z ##[group]Run pytorch/pytorch/.github/actions/get-workflow-job-id@master
2022-04-14T21:00:50.9820122Z with:
2022-04-14T21:00:50.9821119Z   github-token: ***
2022-04-14T21:00:50.9821590Z env:
2022-04-14T21:00:50.9821859Z   IN_CI: 1
2022-04-14T21:00:50.9822107Z   IS_GHA: 1

See GitHub Actions build periodic / linux-bionic-cuda11.6-py3.7-gcc7 / test (default, 2, 2, linux.4xlarge.nvidia.gpu) (2/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-04-14T21:35:23.5347397Z AssertionError: can only test a child process 
2022-04-14T21:35:23.4904044Z   test_multi_epochs_reproducibility (__main__.TestDataLoaderPersistentWorkers) ... ok (0.060s)
2022-04-14T21:35:23.4919747Z   test_multiple_dataloaders (__main__.TestDataLoaderPersistentWorkers) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74598 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-04-14T21:35:23.5329156Z   test_multiprocessing_contexts (__main__.TestDataLoaderPersistentWorkers) ... Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7f78715829e0>
2022-04-14T21:35:23.5329738Z Traceback (most recent call last):
2022-04-14T21:35:23.5330354Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1362, in __del__
2022-04-14T21:35:23.5337709Z     self._shutdown_workers()
2022-04-14T21:35:23.5339346Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1345, in _shutdown_workers
2022-04-14T21:35:23.5344467Z     if w.is_alive():
2022-04-14T21:35:23.5345219Z   File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 151, in is_alive
2022-04-14T21:35:23.5346653Z     assert self._parent_pid == os.getpid(), 'can only test a child process'
2022-04-14T21:35:23.5347397Z AssertionError: can only test a child process
2022-04-14T21:35:26.0960588Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-04-14T21:35:26.0963515Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-04-14T21:35:26.0966473Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-04-14T21:35:28.6970195Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-04-14T21:35:28.6996391Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-04-14T21:35:28.7012398Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-04-14T21:35:33.6972836Z ok (10.205s)
2022-04-14T21:35:33.6996442Z   test_multiprocessing_iterdatapipe (__main__.TestDataLoaderPersistentWorkers) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74498 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-04-14T21:35:34.7696756Z   test_no_segfault (__main__.TestDataLoaderPersistentWorkers) ... ok (1.070s)
2022-04-14T21:35:34.7731597Z   test_numpy (__main__.TestDataLoaderPersistentWorkers) ... ok (0.003s)

See GitHub Actions build pull / linux-xenial-cuda11.3-py3.7-gcc7 / test (distributed, 1, 1, linux.8xlarge.nvidia.gpu) (3/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-04-14T21:32:25.5489376Z FAIL [6.381s]: test_scatter_stress (__main__.ProcessGroupNCCLTest) 
2022-04-14T21:32:24.3770928Z frame #2: pthread_cond_wait + 0xc0 (0x7fafc3c43360 in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-04-14T21:32:24.3771535Z frame #3: <unknown function> + 0x23a62b (0x7faf9325562b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cpp.so)
2022-04-14T21:32:24.3772176Z frame #4: <unknown function> + 0x23ac95 (0x7faf93255c95 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cuda_cpp.so)
2022-04-14T21:32:24.3772746Z frame #5: <unknown function> + 0x76ba (0x7fafc3c3d6ba in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-04-14T21:32:24.3773223Z frame #6: clone + 0x6d (0x7fafc397351d in /lib/x86_64-linux-gnu/libc.so.6)
2022-04-14T21:32:24.3773444Z 
2022-04-14T21:32:25.5488125Z FAIL (4.638s)
2022-04-14T21:32:25.5488666Z     test_scatter_stress failed - num_retries_left: 0
2022-04-14T21:32:25.5488889Z 
2022-04-14T21:32:25.5489031Z ======================================================================
2022-04-14T21:32:25.5489376Z FAIL [6.381s]: test_scatter_stress (__main__.ProcessGroupNCCLTest)
2022-04-14T21:32:25.5490118Z ----------------------------------------------------------------------
2022-04-14T21:32:25.5490502Z Traceback (most recent call last):
2022-04-14T21:32:25.5491028Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 484, in wrapper
2022-04-14T21:32:25.5491445Z     self._join_processes(fn)
2022-04-14T21:32:25.5491978Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 703, in _join_processes
2022-04-14T21:32:25.5492406Z     self._check_return_codes(elapsed_time)
2022-04-14T21:32:25.5492944Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 781, in _check_return_codes
2022-04-14T21:32:25.5493467Z     msg="Expected zero exit code but got {} for pid: {}".format(first_process.exitcode, first_process.pid)
2022-04-14T21:32:25.5494084Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py", line 2193, in assertEqual
2022-04-14T21:32:25.5494449Z     msg=msg,

See GitHub Actions build trunk / linux-bionic-cuda10.2-py3.9-gcc7 / test (multigpu, 1, 1, linux.16xlarge.nvidia.gpu) (4/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-04-14T21:18:15.0206603Z FAIL [5.065s]: test_scatter_stress (__main__.ProcessGroupNCCLTest) 
2022-04-14T21:18:14.2393695Z frame #2: pthread_cond_wait + 0x243 (0x7f7555613ad3 in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-04-14T21:18:14.2394068Z frame #3: <unknown function> + 0x31d502b (0x7f7509e0902b in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-04-14T21:18:14.2394434Z frame #4: <unknown function> + 0x31d5695 (0x7f7509e09695 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-04-14T21:18:14.2394782Z frame #5: <unknown function> + 0x76db (0x7f755560d6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-04-14T21:18:14.2395043Z frame #6: clone + 0x3f (0x7f755533661f in /lib/x86_64-linux-gnu/libc.so.6)
2022-04-14T21:18:14.2395064Z 
2022-04-14T21:18:15.0204147Z FAIL (3.342s)
2022-04-14T21:18:15.0205094Z     test_scatter_stress failed - num_retries_left: 0
2022-04-14T21:18:15.0205605Z 
2022-04-14T21:18:15.0205885Z ======================================================================
2022-04-14T21:18:15.0206603Z FAIL [5.065s]: test_scatter_stress (__main__.ProcessGroupNCCLTest)
2022-04-14T21:18:15.0207125Z ----------------------------------------------------------------------
2022-04-14T21:18:15.0207473Z Traceback (most recent call last):
2022-04-14T21:18:15.0208287Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 484, in wrapper
2022-04-14T21:18:15.0208723Z     self._join_processes(fn)
2022-04-14T21:18:15.0209247Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 703, in _join_processes
2022-04-14T21:18:15.0209680Z     self._check_return_codes(elapsed_time)
2022-04-14T21:18:15.0210234Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 778, in _check_return_codes
2022-04-14T21:18:15.0210638Z     self.assertEqual(
2022-04-14T21:18:15.0211256Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 2160, in assertEqual
2022-04-14T21:18:15.0211639Z     assert_equal(

See GitHub Actions build trunk / linux-bionic-cuda10.2-py3.9-gcc7 / test (default, 2, 2, linux.4xlarge.nvidia.gpu) (5/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-04-14T21:13:26.0239878Z RuntimeError: test_jit_cuda_fuser failed! Received signal: SIGIOT 
2022-04-14T21:13:22.0920599Z   test_broadcasting_partition_logic_1 (__main__.TestCudaFuser) ... ok (0.307s)
2022-04-14T21:13:22.4153943Z   test_build_shape_expression_native_dropout (__main__.TestCudaFuser) ... ok (0.323s)
2022-04-14T21:13:23.6064659Z   test_category_rule (__main__.TestCudaFuser) ... ok (1.191s)
2022-04-14T21:13:24.5897035Z   test_channels_last_with_broadcast (__main__.TestCudaFuser) ... ok (0.983s)
2022-04-14T21:13:24.6745683Z   test_chunk (__main__.TestCudaFuser) ... /usr/include/c++/7/bits/stl_vector.h:797: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::size_type) [with _Tp = long int; _Alloc = std::allocator<long int>; std::vector<_Tp, _Alloc>::reference = long int&; std::vector<_Tp, _Alloc>::size_type = long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
2022-04-14T21:13:26.0233004Z Traceback (most recent call last):
2022-04-14T21:13:26.0233425Z   File "/var/lib/jenkins/workspace/test/run_test.py", line 1058, in <module>
2022-04-14T21:13:26.0236049Z     main()
2022-04-14T21:13:26.0236504Z   File "/var/lib/jenkins/workspace/test/run_test.py", line 1036, in main
2022-04-14T21:13:26.0239504Z     raise RuntimeError(err_message)
2022-04-14T21:13:26.0239878Z RuntimeError: test_jit_cuda_fuser failed! Received signal: SIGIOT
2022-04-14T21:13:26.5204135Z 
2022-04-14T21:13:26.5204499Z real	23m44.273s
2022-04-14T21:13:26.5206705Z user	24m1.081s
2022-04-14T21:13:26.5208656Z sys	4m11.309s
2022-04-14T21:13:26.5209070Z + cleanup
2022-04-14T21:13:26.5209991Z + retcode=1
2022-04-14T21:13:26.5210325Z + set +x
2022-04-14T21:13:26.5269184Z ##[error]Process completed with exit code 1.
2022-04-14T21:13:26.5323424Z ##[group]Run pytorch/pytorch/.github/actions/get-workflow-job-id@master
2022-04-14T21:13:26.5323784Z with:

See GitHub Actions build pull / linux-xenial-cuda11.3-py3.7-gcc7 / test (default, 1, 2, linux.4xlarge.nvidia.gpu) (6/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-04-14T21:30:05.1674448Z AssertionError: can only test a child process 
2022-04-14T21:30:05.1151810Z   test_multi_epochs_reproducibility (__main__.TestDataLoaderPersistentWorkers) ... ok (0.071s)
2022-04-14T21:30:05.1165014Z   test_multiple_dataloaders (__main__.TestDataLoaderPersistentWorkers) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74598 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-04-14T21:30:05.1659067Z   test_multiprocessing_contexts (__main__.TestDataLoaderPersistentWorkers) ... Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7f8f9ff139e0>
2022-04-14T21:30:05.1660569Z Traceback (most recent call last):
2022-04-14T21:30:05.1661619Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1362, in __del__
2022-04-14T21:30:05.1665296Z     self._shutdown_workers()
2022-04-14T21:30:05.1666288Z   File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1345, in _shutdown_workers
2022-04-14T21:30:05.1669897Z     if w.is_alive():
2022-04-14T21:30:05.1672931Z   File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 151, in is_alive
2022-04-14T21:30:05.1673838Z     assert self._parent_pid == os.getpid(), 'can only test a child process'
2022-04-14T21:30:05.1674448Z AssertionError: can only test a child process
2022-04-14T21:30:07.8411278Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-04-14T21:30:07.8464384Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-04-14T21:30:07.8465003Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-04-14T21:30:10.5210275Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-04-14T21:30:10.5237189Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-04-14T21:30:10.5290778Z [W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
2022-04-14T21:30:15.6147786Z ok (10.498s)
2022-04-14T21:30:15.6171139Z   test_multiprocessing_iterdatapipe (__main__.TestDataLoaderPersistentWorkers) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74498 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure IN_CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-04-14T21:30:16.7288362Z   test_no_segfault (__main__.TestDataLoaderPersistentWorkers) ... ok (1.112s)
2022-04-14T21:30:16.7324573Z   test_numpy (__main__.TestDataLoaderPersistentWorkers) ... ok (0.003s)

See GitHub Actions build trunk / linux-bionic-rocm5.0-py3.7-distributed / test (distributed, 1, 1, linux.rocm.gpu) (7/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-04-14T22:16:22.5376892Z FAIL [2.271s]: test_scatter_stress (__main__.ProcessGroupNCCLTest) 
2022-04-14T22:16:22.3785885Z frame #2: pthread_cond_wait + 0x243 (0x7f3d0cad7ad3 in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-04-14T22:16:22.3786312Z frame #3: <unknown function> + 0x683f8c9 (0x7f3cbf7f58c9 in /opt/rocm/rccl/lib/librccl.so.1)
2022-04-14T22:16:22.3786731Z frame #4: <unknown function> + 0x683fdb8 (0x7f3cbf7f5db8 in /opt/rocm/rccl/lib/librccl.so.1)
2022-04-14T22:16:22.3787249Z frame #5: <unknown function> + 0x76db (0x7f3d0cad16db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-04-14T22:16:22.3787748Z frame #6: clone + 0x3f (0x7f3d0c7fa61f in /lib/x86_64-linux-gnu/libc.so.6)
2022-04-14T22:16:22.3787956Z 
2022-04-14T22:16:22.5374383Z FAIL (1.723s)
2022-04-14T22:16:22.5375463Z     test_scatter_stress failed - num_retries_left: 0
2022-04-14T22:16:22.5375903Z 
2022-04-14T22:16:22.5376172Z ======================================================================
2022-04-14T22:16:22.5376892Z FAIL [2.271s]: test_scatter_stress (__main__.ProcessGroupNCCLTest)
2022-04-14T22:16:22.5377942Z ----------------------------------------------------------------------
2022-04-14T22:16:22.5378713Z Traceback (most recent call last):
2022-04-14T22:16:22.5379904Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 484, in wrapper
2022-04-14T22:16:22.5380733Z     self._join_processes(fn)
2022-04-14T22:16:22.5382098Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 703, in _join_processes
2022-04-14T22:16:22.5382387Z     self._check_return_codes(elapsed_time)
2022-04-14T22:16:22.5383274Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 781, in _check_return_codes
2022-04-14T22:16:22.5383775Z     msg="Expected zero exit code but got {} for pid: {}".format(first_process.exitcode, first_process.pid)
2022-04-14T22:16:22.5384907Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py", line 2193, in assertEqual
2022-04-14T22:16:22.5385108Z     msg=msg,

See GitHub Actions build trunk / linux-bionic-cuda10.2-py3.9-gcc7 / test (distributed, 1, 1, linux.8xlarge.nvidia.gpu) (8/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-04-14T21:47:38.1857982Z FAIL [4.858s]: test_scatter_stress (__main__.ProcessGroupNCCLTest) 
2022-04-14T21:47:37.3587635Z frame #2: pthread_cond_wait + 0x243 (0x7fdb5cfefad3 in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-04-14T21:47:37.3588247Z frame #3: <unknown function> + 0x31d502b (0x7fdb117e502b in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-04-14T21:47:37.3588883Z frame #4: <unknown function> + 0x31d5695 (0x7fdb117e5695 in /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
2022-04-14T21:47:37.3589504Z frame #5: <unknown function> + 0x76db (0x7fdb5cfe96db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-04-14T21:47:37.3590020Z frame #6: clone + 0x3f (0x7fdb5cd1261f in /lib/x86_64-linux-gnu/libc.so.6)
2022-04-14T21:47:37.3590247Z 
2022-04-14T21:47:38.1856125Z FAIL (3.242s)
2022-04-14T21:47:38.1856654Z     test_scatter_stress failed - num_retries_left: 0
2022-04-14T21:47:38.1857173Z 
2022-04-14T21:47:38.1857330Z ======================================================================
2022-04-14T21:47:38.1857982Z FAIL [4.858s]: test_scatter_stress (__main__.ProcessGroupNCCLTest)
2022-04-14T21:47:38.1858561Z ----------------------------------------------------------------------
2022-04-14T21:47:38.1858928Z Traceback (most recent call last):
2022-04-14T21:47:38.1859466Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 484, in wrapper
2022-04-14T21:47:38.1859905Z     self._join_processes(fn)
2022-04-14T21:47:38.1860474Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 703, in _join_processes
2022-04-14T21:47:38.1860913Z     self._check_return_codes(elapsed_time)
2022-04-14T21:47:38.1861461Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_distributed.py", line 778, in _check_return_codes
2022-04-14T21:47:38.1861870Z     self.assertEqual(
2022-04-14T21:47:38.1862383Z   File "/opt/conda/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 2160, in assertEqual
2022-04-14T21:47:38.1862764Z     assert_equal(
This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@davidberard98 davidberard98 marked this pull request as draft April 13, 2022 22:49
@davidberard98 davidberard98 added the ciflow/trunk Trigger trunk jobs on your pull request label Apr 13, 2022
@davidberard98 davidberard98 mentioned this pull request Apr 13, 2022
Closed
@davidberard98
Copy link
Contributor Author

davidberard98 commented Apr 13, 2022

@davidberard98 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@davidberard98
Update on "[testing] -D_GLIBCXX_ASSERTIONS"
8e20078 
curious if it'll catch anything

Differential Revision: [D35632195](https://our.internmc.facebook.com/intern/diff/D35632195)

[ghstack-poisoned]
@davidberard98 davidberard98 mentioned this pull request Apr 14, 2022
Closed
davidberard98 added a commit that referenced this pull request Apr 14, 2022
@davidberard98
[testing] -D_GLIBCXX_ASSERTIONS
c6d66a3 
curious if it'll catch anything

ghstack-source-id: e9dc279
Pull Request resolved: #75766
@davidberard98 davidberard98 mentioned this pull request Apr 14, 2022
Closed
@davidberard98
Update on "[testing] -D_GLIBCXX_ASSERTIONS"
de58635 
curious if it'll catch anything

Differential Revision: [D35632195](https://our.internmc.facebook.com/intern/diff/D35632195)

[ghstack-poisoned]
@davidberard98
Update on "[testing] -D_GLIBCXX_ASSERTIONS"
449f8ba 
curious if it'll catch anything

Differential Revision: [D35632195](https://our.internmc.facebook.com/intern/diff/D35632195)

[ghstack-poisoned]
davidberard98 added a commit that referenced this pull request Apr 14, 2022
@davidberard98
[testing] -D_GLIBCXX_ASSERTIONS
b095a29 
curious if it'll catch anything

ghstack-source-id: ec0eb97
Pull Request resolved: #75766
@davidberard98 davidberard98 mentioned this pull request Apr 15, 2022
Open
@github-actions
Copy link
Contributor

github-actions bot commented Jun 13, 2022

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

@github-actions github-actions bot added the Stale label Jun 13, 2022
@github-actions github-actions bot closed this Jul 13, 2022
@facebook-github-bot facebook-github-bot deleted the gh/davidberard98/91/head branch August 13, 2022 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request cla signed Stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@davidberard98 @facebook-github-bot