[not for land] backward callback stream issue by rohan-varma · Pull Request #37858 · pytorch/pytorch

rohan-varma · 2020-05-05T19:23:49Z

Stack from ghstack:

[not for land] backward callback stream issue #37858 [not for land] backward callback stream issue
[WIP][DDP] Synchronize event copies that can occur on different streams. #37807 [WIP][DDP] Synchronize event copies that can occur on different streams.

backward callback stream issue

Differential Revision: D21409295

backward callback stream issue Differential Revision: [D21409295](https://our.internmc.facebook.com/intern/diff/D21409295/) [ghstack-poisoned]

backward callback stream issue Differential Revision: [D21409295](https://our.internmc.facebook.com/intern/diff/D21409295/) ghstack-source-id: 103509614 Pull Request resolved: #37858

dr-ci · 2020-05-05T19:44:47Z

💊 Build failures summary and remediations

As of commit 516bf14 (more details on the Dr. CI page):

11/11 failures possibly* introduced in this PR
- 1/11 non-CircleCI failure(s)

🕵️ 10 new failures recognized by patterns

The following build failures do not appear to be due to upstream breakages:

caffe2_onnx_main_py3_6_clang7_ubuntu16_04_build (1/10)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 05 19:37:34 /var/lib/jenkins/workspace/aten/src/ATen/cuda/ATenCUDAGeneral.h:3:10: fatal error: 'cuda.h' file not found

May 05 19:37:33 [ 98%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_schema_matching.cpp.o 
May 05 19:37:33 [ 98%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_matcher.cpp.o 
May 05 19:37:34 [ 98%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_rewriter.cpp.o 
May 05 19:37:34 [ 98%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_utils.cpp.o 
May 05 19:37:34 [ 98%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_utils.cpp.o 
May 05 19:37:34 [ 98%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/autograd/init.cpp.o 
May 05 19:37:34 [ 98%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o 
May 05 19:37:34 In file included from /var/lib/jenkins/workspace/torch/csrc/distributed/c10d/comm.cpp:6: 
May 05 19:37:34 In file included from /var/lib/jenkins/workspace/torch/csrc/distributed/c10d/reducer.h:11: 
May 05 19:37:34 In file included from /var/lib/jenkins/workspace/aten/src/ATen/cuda/CUDAEvent.h:3: 
May 05 19:37:34 /var/lib/jenkins/workspace/aten/src/ATen/cuda/ATenCUDAGeneral.h:3:10: fatal error: 'cuda.h' file not found 
May 05 19:37:34 #include <cuda.h> 
May 05 19:37:34          ^~~~~~~~ 
May 05 19:37:34 1 error generated. 
May 05 19:37:34 caffe2/torch/CMakeFiles/torch_python.dir/build.make:2918: recipe for target 'caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o' failed 
May 05 19:37:34 make[2]: *** [caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o] Error 1 
May 05 19:37:34 make[2]: *** Waiting for unfinished jobs.... 
May 05 19:37:35 make[1]: *** [caffe2/torch/CMakeFiles/torch_python.dir/all] Error 2 
May 05 19:37:35 CMakeFiles/Makefile2:13528: recipe for target 'caffe2/torch/CMakeFiles/torch_python.dir/all' failed 
May 05 19:37:35 Makefile:138: recipe for target 'all' failed 
May 05 19:37:35 make: *** [all] Error 2

pytorch_linux_bionic_py3_6_clang9_build (2/10)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 05 19:38:53 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: use of undeclared identifier \'strtod_l\'\n  return ((int*)(&strtod_l))[argc];\n                  ^\n1 error generated.\n" }

May 05 19:38:53     self.run(build_args, my_env) 
May 05 19:38:53   File "/var/lib/jenkins/workspace/tools/setup_helpers/cmake.py", line 141, in run 
May 05 19:38:53     check_call(command, cwd=self.build_dir, env=env) 
May 05 19:38:53   File "/opt/conda/lib/python3.6/subprocess.py", line 311, in check_call 
May 05 19:38:53     raise CalledProcessError(retcode, cmd) 
May 05 19:38:53 subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '3']' returned non-zero exit status 2. 
May 05 19:38:53 + cleanup 
May 05 19:38:53 + retcode=1 
May 05 19:38:53 + set +x 
May 05 19:38:53 =================== sccache compilation log =================== 
May 05 19:38:53 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: use of undeclared identifier \'strtod_l\'\n  return ((int*)(&strtod_l))[argc];\n                  ^\n1 error generated.\n" } 
May 05 19:38:53  
May 05 19:38:53 =========== If your build fails, please take a look at the log above for possible reasons =========== 
May 05 19:38:53 Compile requests               5700 
May 05 19:38:53 Compile requests executed      3410 
May 05 19:38:53 Cache hits                     3391 
May 05 19:38:53 Cache misses                      1 
May 05 19:38:53 Cache timeouts                    0 
May 05 19:38:53 Cache read errors                 0 
May 05 19:38:53 Forced recaches                   0 
May 05 19:38:53 Cache write errors                0

pytorch_linux_xenial_py3_clang5_asan_build (3/10)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 05 19:39:05 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "clang: warning: argument unused during compilation: \'-pie\' [-Wunused-command-line-argument]\n/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: use of undeclared identifier \'strtod_l\'\n  return ((int*)(&strtod_l))[argc];\n                  ^\n1 error generated.\n" }

May 05 19:39:05     self.run(build_args, my_env) 
May 05 19:39:05   File "/var/lib/jenkins/workspace/tools/setup_helpers/cmake.py", line 141, in run 
May 05 19:39:05     check_call(command, cwd=self.build_dir, env=env) 
May 05 19:39:05   File "/opt/conda/lib/python3.6/subprocess.py", line 311, in check_call 
May 05 19:39:05     raise CalledProcessError(retcode, cmd) 
May 05 19:39:05 subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '3']' returned non-zero exit status 2. 
May 05 19:39:05 + cleanup 
May 05 19:39:05 + retcode=1 
May 05 19:39:05 + set +x 
May 05 19:39:05 =================== sccache compilation log =================== 
May 05 19:39:05 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "clang: warning: argument unused during compilation: \'-pie\' [-Wunused-command-line-argument]\n/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: use of undeclared identifier \'strtod_l\'\n  return ((int*)(&strtod_l))[argc];\n                  ^\n1 error generated.\n" } 
May 05 19:39:05  
May 05 19:39:05 =========== If your build fails, please take a look at the log above for possible reasons =========== 
May 05 19:39:05 Compile requests               5192 
May 05 19:39:05 Compile requests executed      3157 
May 05 19:39:05 Cache hits                     3137 
May 05 19:39:05 Cache misses                      1 
May 05 19:39:05 Cache timeouts                    0 
May 05 19:39:05 Cache read errors                 0 
May 05 19:39:05 Forced recaches                   0 
May 05 19:39:05 Cache write errors                0

pytorch_macos_10_13_py3_build (4/10)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 05 12:39:17 FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o

May 05 12:39:16 FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o  
X -DUSE_AVX2 -DTH_HAVE_THREAD -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-write-strings -Wno-unknown-pragmas -Wno-missing-braces "-Xpreprocessor -fopenmp -I/usr/local/include" -std=gnu++14 -MD -MT caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o -MF caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o.d -o caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o -c ../torch/csrc/distributed/c10d/comm.cpp 
May 05 12:39:16 In file included from ../torch/csrc/distributed/c10d/comm.cpp:6: 
May 05 12:39:16 In file included from ../torch/csrc/distributed/c10d/reducer.h:11: 
May 05 12:39:16 In file included from ../aten/src/ATen/cuda/CUDAEvent.h:3: 
May 05 12:39:16 ../aten/src/ATen/cuda/ATenCUDAGeneral.h:3:10: fatal error: 'cuda.h' file not found 
May 05 12:39:16 #include <cuda.h> 
May 05 12:39:16          ^~~~~~~~ 
May 05 12:39:16 1 error generated. 
May 05 12:39:17 [4147/4176] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o 
May 05 12:39:17 FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o  
X -DUSE_AVX2 -DTH_HAVE_THREAD -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-write-strings -Wno-unknown-pragmas -Wno-missing-braces "-Xpreprocessor -fopenmp -I/usr/local/include" -std=gnu++14 -MD -MT caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o -MF caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o.d -o caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o -c ../torch/csrc/distributed/c10d/init.cpp 
May 05 12:39:17 In file included from ../torch/csrc/distributed/c10d/init.cpp:27: 
May 05 12:39:17 In file included from ../torch/csrc/distributed/c10d/reducer.h:11: 
May 05 12:39:17 In file included from ../aten/src/ATen/cuda/CUDAEvent.h:3: 
May 05 12:39:17 ../aten/src/ATen/cuda/ATenCUDAGeneral.h:3:10: fatal error: 'cuda.h' file not found 
May 05 12:39:17 #include <cuda.h> 
May 05 12:39:17          ^~~~~~~~ 
May 05 12:39:17 1 error generated. 
May 05 12:39:17 ninja: build stopped: subcommand failed. 
May 05 12:39:17 Traceback (most recent call last):

pytorch_linux_xenial_py3_6_gcc5_4_build (5/10)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

atus(256)), stdout: "", stderr: "/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c: In function \'main\':\n/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: \'strtod_l\' undeclared (first use in this function)\n   return ((int*)(&strtod_l))[argc];\n                   ^\n/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: note: each undeclared identifier is reported only once for each function it appears in\n" }

May 05 19:40:26     self.run(build_args, my_env) 
May 05 19:40:26   File "/var/lib/jenkins/workspace/tools/setup_helpers/cmake.py", line 141, in run 
May 05 19:40:26     check_call(command, cwd=self.build_dir, env=env) 
May 05 19:40:26   File "/opt/conda/lib/python3.6/subprocess.py", line 311, in check_call 
May 05 19:40:26     raise CalledProcessError(retcode, cmd) 
May 05 19:40:26 subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '3']' returned non-zero exit status 2. 
May 05 19:40:26 + cleanup 
May 05 19:40:26 + retcode=1 
May 05 19:40:26 + set +x 
May 05 19:40:26 =================== sccache compilation log =================== 
tus(256)), stdout: "", stderr: "/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c: In function \'main\':\n/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: \'strtod_l\' undeclared (first use in this function)\n   return ((int*)(&strtod_l))[argc];\n                   ^\n/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: note: each undeclared identifier is reported only once for each function it appears in\n" } 
May 05 19:40:26  
May 05 19:40:26 =========== If your build fails, please take a look at the log above for possible reasons =========== 
May 05 19:40:26 Compile requests               5674 
May 05 19:40:26 Compile requests executed      3397 
May 05 19:40:26 Cache hits                     3379 
May 05 19:40:26 Cache misses                      1 
May 05 19:40:26 Cache timeouts                    0 
May 05 19:40:26 Cache read errors                 0 
May 05 19:40:26 Forced recaches                   0 
May 05 19:40:26 Cache write errors                0

pytorch_xla_linux_bionic_py3_6_clang9_build (6/10)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 05 19:40:26 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: use of undeclared identifier \'strtod_l\'\n  return ((int*)(&strtod_l))[argc];\n                  ^\n1 error generated.\n" }

May 05 19:40:26     self.run(build_args, my_env) 
May 05 19:40:26   File "/var/lib/jenkins/workspace/tools/setup_helpers/cmake.py", line 141, in run 
May 05 19:40:26     check_call(command, cwd=self.build_dir, env=env) 
May 05 19:40:26   File "/opt/conda/lib/python3.6/subprocess.py", line 311, in check_call 
May 05 19:40:26     raise CalledProcessError(retcode, cmd) 
May 05 19:40:26 subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '3']' returned non-zero exit status 2. 
May 05 19:40:26 =================== sccache compilation log =================== 
May 05 19:40:26 + cleanup 
May 05 19:40:26 + retcode=1 
May 05 19:40:26 + set +x 
May 05 19:40:26 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: use of undeclared identifier \'strtod_l\'\n  return ((int*)(&strtod_l))[argc];\n                  ^\n1 error generated.\n" } 
May 05 19:40:26  
May 05 19:40:26 =========== If your build fails, please take a look at the log above for possible reasons =========== 
May 05 19:40:26 Compile requests              5700 
May 05 19:40:26 Compile requests executed     3410 
May 05 19:40:26 Cache hits                    3392 
May 05 19:40:26 Cache misses                     0 
May 05 19:40:26 Cache timeouts                   0 
May 05 19:40:26 Cache read errors                0 
May 05 19:40:26 Forced recaches                  0 
May 05 19:40:26 Cache write errors               0

binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build (7/10)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 05 19:45:54 FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o

May 05 19:45:48 [4132/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_interpreter.cpp.o 
May 05 19:45:48 [4133/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_fuser.cpp.o 
May 05 19:45:48 [4134/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_custom_class.cpp.o 
May 05 19:45:48 [4135/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_jit_type.cpp.o 
May 05 19:45:48 [4136/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_ir.cpp.o 
May 05 19:45:49 [4137/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_irparser.cpp.o 
May 05 19:45:51 [4138/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_mobile_type_parser.cpp.o 
May 05 19:45:53 [4139/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_peephole_optimize.cpp.o 
May 05 19:45:53 [4140/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_qualified_name.cpp.o 
May 05 19:45:54 [4141/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o 
May 05 19:45:54 FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o  
USE_AVX2 -DTH_HAVE_THREAD -fno-strict-aliasing -Wno-write-strings -Wno-strict-aliasing -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-missing-braces -fopenmp -std=gnu++14 -MD -MT caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o -MF caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o.d -o caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o -c ../torch/csrc/distributed/c10d/comm.cpp 
May 05 19:45:54 In file included from ../aten/src/ATen/cuda/CUDAEvent.h:3:0, 
May 05 19:45:54                  from ../torch/csrc/distributed/c10d/reducer.h:11, 
May 05 19:45:54                  from ../torch/csrc/distributed/c10d/comm.cpp:6: 
May 05 19:45:54 ../aten/src/ATen/cuda/ATenCUDAGeneral.h:3:18: fatal error: cuda.h: No such file or directory 
May 05 19:45:54 compilation terminated. 
May 05 19:45:54 [4142/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_utils.cpp.o 
May 05 19:45:55 [4143/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_rewriter.cpp.o 
May 05 19:45:55 [4144/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_interface.cpp.o 
May 05 19:45:56 [4145/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_matcher.cpp.o

binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_build (8/10)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 05 19:48:33 /pytorch/aten/src/ATen/cuda/ATenCUDAGeneral.h:3:10: fatal error: cuda.h: No such file or directory

May 05 19:48:26 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_schema_matching.cpp.o 
May 05 19:48:26 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_matcher.cpp.o 
May 05 19:48:27 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_rewriter.cpp.o 
May 05 19:48:29 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_utils.cpp.o 
May 05 19:48:32 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_utils.cpp.o 
May 05 19:48:32 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/autograd/init.cpp.o 
May 05 19:48:33 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o 
May 05 19:48:33 In file included from /pytorch/aten/src/ATen/cuda/CUDAEvent.h:3:0, 
May 05 19:48:33                  from /pytorch/torch/csrc/distributed/c10d/reducer.h:11, 
May 05 19:48:33                  from /pytorch/torch/csrc/distributed/c10d/comm.cpp:6: 
May 05 19:48:33 /pytorch/aten/src/ATen/cuda/ATenCUDAGeneral.h:3:10: fatal error: cuda.h: No such file or directory 
May 05 19:48:33  #include <cuda.h> 
May 05 19:48:33           ^~~~~~~~ 
May 05 19:48:33 compilation terminated. 
May 05 19:48:33 gmake[2]: *** [caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o] Error 1 
May 05 19:48:33 gmake[2]: *** Waiting for unfinished jobs.... 
May 05 19:48:45 gmake[1]: *** [caffe2/torch/CMakeFiles/torch_python.dir/all] Error 2 
May 05 19:48:45 gmake: *** [all] Error 2 
May 05 19:48:45 Traceback (most recent call last): 
May 05 19:48:45   File "setup.py", line 740, in <module> 
May 05 19:48:45     build_deps()

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test (9/10)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

May 05 20:38:50 RuntimeError: test_cuda failed!

May 05 20:38:50  
May 05 20:38:50 FAILED (errors=1, skipped=6) 
May 05 20:38:50  
May 05 20:38:50 Generating XML reports... 
May 05 20:38:50 Generated XML report: test-reports/python-unittest/TEST-TestCuda-20200505203621.xml 
May 05 20:38:50 Traceback (most recent call last): 
May 05 20:38:50   File "test/run_test.py", line 673, in <module> 
May 05 20:38:50     main() 
May 05 20:38:50   File "test/run_test.py", line 666, in main 
May 05 20:38:50     raise RuntimeError(message) 
May 05 20:38:50 RuntimeError: test_cuda failed! 
May 05 20:38:51 + cleanup 
May 05 20:38:51 + retcode=1 
May 05 20:38:51 + set +x 
May 05 20:38:51 =================== sccache compilation log =================== 
May 05 20:38:51 =========== If your build fails, please take a look at the log above for possible reasons =========== 
May 05 20:38:51 Compile requests               137 
May 05 20:38:51 Compile requests executed       50 
May 05 20:38:51 Cache hits                      49 
May 05 20:38:51 Cache misses                     0 
May 05 20:38:51 Cache timeouts                   0

pytorch_windows_vs2019_py36_cuda10.1_test2 (10/10)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

RuntimeError: test_cuda failed!

 
FAILED (errors=1, skipped=54) 
 
Generating XML reports... 
Generated XML report: test-reports\python-unittest\TEST-TestCuda-20200505204209.xml 
Traceback (most recent call last): 
  File "run_test.py", line 673, in <module> 
    main() 
  File "run_test.py", line 666, in main 
    raise RuntimeError(message) 
RuntimeError: test_cuda failed! 
 
(base) circleci@PACKER-5E9E037A C:\Users\circleci\project\test>if ERRORLEVEL 1 exit /b 1  
+ cleanup
+ retcode=1
+ set +x

ci.pytorch.org: 1 failed

Failed: pr/py3.6-clang7-rocmdeb-ubuntu16.04

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

See how this bot performed.

This comment has been revised 4 times.

[not for land] backward callback stream issue

516bf14

backward callback stream issue Differential Revision: [D21409295](https://our.internmc.facebook.com/intern/diff/D21409295/) [ghstack-poisoned]

rohan-varma requested review from albanD and apaszke as code owners May 5, 2020 19:23

rohan-varma mentioned this pull request May 5, 2020

[WIP][DDP] Synchronize event copies that can occur on different streams. #37807

Closed

rohan-varma mentioned this pull request May 5, 2020

autograd engine callbacks don't respect non-default cuda streams #37860

Open

rohan-varma closed this May 14, 2020

facebook-github-bot deleted the gh/rohan-varma/118/head branch June 13, 2020 14:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[not for land] backward callback stream issue#37858

[not for land] backward callback stream issue#37858
rohan-varma wants to merge 1 commit intogh/rohan-varma/118/basefrom
gh/rohan-varma/118/head

rohan-varma commented May 5, 2020 •

edited

Loading

Uh oh!

dr-ci Bot commented May 5, 2020 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

rohan-varma commented May 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr-ci Bot commented May 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 Build failures summary and remediations

🕵️ 10 new failures recognized by patterns

caffe2_onnx_main_py3_6_clang7_ubuntu16_04_build (1/10)

pytorch_linux_bionic_py3_6_clang9_build (2/10)

pytorch_linux_xenial_py3_clang5_asan_build (3/10)

pytorch_macos_10_13_py3_build (4/10)

pytorch_linux_xenial_py3_6_gcc5_4_build (5/10)

pytorch_xla_linux_bionic_py3_6_clang9_build (6/10)

binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build (7/10)

binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_build (8/10)

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test (9/10)

pytorch_windows_vs2019_py36_cuda10.1_test2 (10/10)

ci.pytorch.org: 1 failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

rohan-varma commented May 5, 2020 •

edited

Loading

dr-ci Bot commented May 5, 2020 •

edited

Loading