[WIP][DDP] Synchronize event copies that can occur on different streams. by rohan-varma · Pull Request #37807 · pytorch/pytorch

rohan-varma · 2020-05-05T03:04:23Z

Stack from ghstack:

[not for land] backward callback stream issue #37858 [not for land] backward callback stream issue
[WIP][DDP] Synchronize event copies that can occur on different streams. #37807 [WIP][DDP] Synchronize event copies that can occur on different streams.

Differential Revision: D21396403

Differential Revision: [D21396403](https://our.internmc.facebook.com/intern/diff/D21396403/) [ghstack-poisoned]

dr-ci · 2020-05-05T03:20:37Z

💊 Build failures summary and remediations

As of commit 6d2f90d (more details on the Dr. CI page):

8/8 failures introduced in this PR

🕵️ 8 new failures recognized by patterns

The following build failures do not appear to be due to upstream breakages:

caffe2_onnx_main_py3_6_clang7_ubuntu16_04_build (1/8)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 05 05:12:58 /var/lib/jenkins/workspace/aten/src/ATen/cuda/ATenCUDAGeneral.h:3:10: fatal error: 'cuda.h' file not found

May 05 05:12:57 /var/lib/jenkins/workspace/aten/src/ATen/cuda/ATenCUDAGeneral.h:3:10: fatal error: 'cuda.h' file not found 
May 05 05:12:57 #include <cuda.h> 
May 05 05:12:57          ^~~~~~~~ 
May 05 05:12:57 1 error generated. 
May 05 05:12:57 caffe2/torch/CMakeFiles/torch_python.dir/build.make:2918: recipe for target 'caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o' failed 
May 05 05:12:57 make[2]: *** [caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o] Error 1 
May 05 05:12:57 make[2]: *** Waiting for unfinished jobs.... 
May 05 05:12:58 In file included from /var/lib/jenkins/workspace/torch/csrc/distributed/c10d/init.cpp:27: 
May 05 05:12:58 In file included from /var/lib/jenkins/workspace/torch/csrc/distributed/c10d/reducer.h:11: 
May 05 05:12:58 In file included from /var/lib/jenkins/workspace/aten/src/ATen/cuda/CUDAEvent.h:3: 
May 05 05:12:58 /var/lib/jenkins/workspace/aten/src/ATen/cuda/ATenCUDAGeneral.h:3:10: fatal error: 'cuda.h' file not found 
May 05 05:12:58 #include <cuda.h> 
May 05 05:12:58          ^~~~~~~~ 
May 05 05:12:58 1 error generated. 
May 05 05:12:58 caffe2/torch/CMakeFiles/torch_python.dir/build.make:2942: recipe for target 'caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o' failed 
May 05 05:12:58 make[2]: *** [caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o] Error 1 
May 05 05:12:58 make[1]: *** [caffe2/torch/CMakeFiles/torch_python.dir/all] Error 2 
May 05 05:12:58 CMakeFiles/Makefile2:13528: recipe for target 'caffe2/torch/CMakeFiles/torch_python.dir/all' failed 
May 05 05:12:58 make: *** [all] Error 2 
May 05 05:12:58 Makefile:138: recipe for target 'all' failed 
May 05 05:12:58 -- Building version 1.6.0a0+6d2f90d

pytorch_linux_bionic_py3_6_clang9_build (2/8)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 05 05:16:15 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: use of undeclared identifier \'strtod_l\'\n  return ((int*)(&strtod_l))[argc];\n                  ^\n1 error generated.\n" }

May 05 05:16:15     self.run(build_args, my_env) 
May 05 05:16:15   File "/var/lib/jenkins/workspace/tools/setup_helpers/cmake.py", line 141, in run 
May 05 05:16:15     check_call(command, cwd=self.build_dir, env=env) 
May 05 05:16:15   File "/opt/conda/lib/python3.6/subprocess.py", line 311, in check_call 
May 05 05:16:15     raise CalledProcessError(retcode, cmd) 
May 05 05:16:15 subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '3']' returned non-zero exit status 2. 
May 05 05:16:15 + cleanup 
May 05 05:16:15 + retcode=1 
May 05 05:16:15 + set +x 
May 05 05:16:15 =================== sccache compilation log =================== 
May 05 05:16:15 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: use of undeclared identifier \'strtod_l\'\n  return ((int*)(&strtod_l))[argc];\n                  ^\n1 error generated.\n" } 
May 05 05:16:15  
May 05 05:16:15 =========== If your build fails, please take a look at the log above for possible reasons =========== 
May 05 05:16:15 Compile requests              5700 
May 05 05:16:15 Compile requests executed     3410 
May 05 05:16:15 Cache hits                    3392 
May 05 05:16:15 Cache misses                     0 
May 05 05:16:15 Cache timeouts                   0 
May 05 05:16:15 Cache read errors                0 
May 05 05:16:15 Forced recaches                  0 
May 05 05:16:15 Cache write errors               0

pytorch_linux_xenial_py3_clang5_asan_build (3/8)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 05 05:16:20 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "clang: warning: argument unused during compilation: \'-pie\' [-Wunused-command-line-argument]\n/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: use of undeclared identifier \'strtod_l\'\n  return ((int*)(&strtod_l))[argc];\n                  ^\n1 error generated.\n" }

May 05 05:16:19     self.run(build_args, my_env) 
May 05 05:16:19   File "/var/lib/jenkins/workspace/tools/setup_helpers/cmake.py", line 141, in run 
May 05 05:16:19     check_call(command, cwd=self.build_dir, env=env) 
May 05 05:16:19   File "/opt/conda/lib/python3.6/subprocess.py", line 311, in check_call 
May 05 05:16:19     raise CalledProcessError(retcode, cmd) 
May 05 05:16:19 subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '3']' returned non-zero exit status 2. 
May 05 05:16:20 + cleanup 
May 05 05:16:20 + retcode=1 
May 05 05:16:20 + set +x 
May 05 05:16:20 =================== sccache compilation log =================== 
May 05 05:16:20 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "clang: warning: argument unused during compilation: \'-pie\' [-Wunused-command-line-argument]\n/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: use of undeclared identifier \'strtod_l\'\n  return ((int*)(&strtod_l))[argc];\n                  ^\n1 error generated.\n" } 
May 05 05:16:20  
May 05 05:16:20 =========== If your build fails, please take a look at the log above for possible reasons =========== 
May 05 05:16:20 Compile requests              5190 
May 05 05:16:20 Compile requests executed     3156 
May 05 05:16:20 Cache hits                    3138 
May 05 05:16:20 Cache misses                     0 
May 05 05:16:20 Cache timeouts                   0 
May 05 05:16:20 Cache read errors                0 
May 05 05:16:20 Forced recaches                  0 
May 05 05:16:20 Cache write errors               0

pytorch_macos_10_13_py3_build (4/8)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 04 22:17:24 FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o

May 04 22:17:24 FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o  
X -DUSE_AVX2 -DTH_HAVE_THREAD -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-write-strings -Wno-unknown-pragmas -Wno-missing-braces "-Xpreprocessor -fopenmp -I/usr/local/include" -std=gnu++14 -MD -MT caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o -MF caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o.d -o caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o -c ../torch/csrc/distributed/c10d/comm.cpp 
May 04 22:17:24 In file included from ../torch/csrc/distributed/c10d/comm.cpp:6: 
May 04 22:17:24 In file included from ../torch/csrc/distributed/c10d/reducer.h:11: 
May 04 22:17:24 In file included from ../aten/src/ATen/cuda/CUDAEvent.h:3: 
May 04 22:17:24 ../aten/src/ATen/cuda/ATenCUDAGeneral.h:3:10: fatal error: 'cuda.h' file not found 
May 04 22:17:24 #include <cuda.h> 
May 04 22:17:24          ^~~~~~~~ 
May 04 22:17:24 1 error generated. 
May 04 22:17:24 [4147/4176] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o 
May 04 22:17:24 FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o  
X -DUSE_AVX2 -DTH_HAVE_THREAD -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-write-strings -Wno-unknown-pragmas -Wno-missing-braces "-Xpreprocessor -fopenmp -I/usr/local/include" -std=gnu++14 -MD -MT caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o -MF caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o.d -o caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o -c ../torch/csrc/distributed/c10d/init.cpp 
May 04 22:17:24 In file included from ../torch/csrc/distributed/c10d/init.cpp:27: 
May 04 22:17:24 In file included from ../torch/csrc/distributed/c10d/reducer.h:11: 
May 04 22:17:24 In file included from ../aten/src/ATen/cuda/CUDAEvent.h:3: 
May 04 22:17:24 ../aten/src/ATen/cuda/ATenCUDAGeneral.h:3:10: fatal error: 'cuda.h' file not found 
May 04 22:17:24 #include <cuda.h> 
May 04 22:17:24          ^~~~~~~~ 
May 04 22:17:24 1 error generated. 
May 04 22:17:24 ninja: build stopped: subcommand failed. 
May 04 22:17:24 Traceback (most recent call last):

pytorch_xla_linux_bionic_py3_6_clang9_build (5/8)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 05 05:17:36 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: use of undeclared identifier \'strtod_l\'\n  return ((int*)(&strtod_l))[argc];\n                  ^\n1 error generated.\n" }

May 05 05:17:36     self.run(build_args, my_env) 
May 05 05:17:36   File "/var/lib/jenkins/workspace/tools/setup_helpers/cmake.py", line 141, in run 
May 05 05:17:36     check_call(command, cwd=self.build_dir, env=env) 
May 05 05:17:36   File "/opt/conda/lib/python3.6/subprocess.py", line 311, in check_call 
May 05 05:17:36     raise CalledProcessError(retcode, cmd) 
May 05 05:17:36 subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '3']' returned non-zero exit status 2. 
May 05 05:17:36 =================== sccache compilation log =================== 
May 05 05:17:36 + cleanup 
May 05 05:17:36 + retcode=1 
May 05 05:17:36 + set +x 
May 05 05:17:36 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: use of undeclared identifier \'strtod_l\'\n  return ((int*)(&strtod_l))[argc];\n                  ^\n1 error generated.\n" } 
May 05 05:17:36  
May 05 05:17:36 =========== If your build fails, please take a look at the log above for possible reasons =========== 
May 05 05:17:36 Compile requests              5702 
May 05 05:17:36 Compile requests executed     3411 
May 05 05:17:36 Cache hits                    3392 
May 05 05:17:36 Cache misses                     0 
May 05 05:17:36 Cache timeouts                   0 
May 05 05:17:36 Cache read errors                0 
May 05 05:17:36 Forced recaches                  0 
May 05 05:17:36 Cache write errors               0

pytorch_linux_xenial_py3_6_gcc5_4_build (6/8)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

atus(256)), stdout: "", stderr: "/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c: In function \'main\':\n/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: \'strtod_l\' undeclared (first use in this function)\n   return ((int*)(&strtod_l))[argc];\n                   ^\n/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: note: each undeclared identifier is reported only once for each function it appears in\n" }

May 05 05:18:03     self.run(build_args, my_env) 
May 05 05:18:03   File "/var/lib/jenkins/workspace/tools/setup_helpers/cmake.py", line 141, in run 
May 05 05:18:03     check_call(command, cwd=self.build_dir, env=env) 
May 05 05:18:03   File "/opt/conda/lib/python3.6/subprocess.py", line 311, in check_call 
May 05 05:18:03     raise CalledProcessError(retcode, cmd) 
May 05 05:18:03 subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '3']' returned non-zero exit status 2. 
May 05 05:18:03 + cleanup 
May 05 05:18:03 + retcode=1 
May 05 05:18:03 + set +x 
May 05 05:18:03 =================== sccache compilation log =================== 
tus(256)), stdout: "", stderr: "/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c: In function \'main\':\n/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: error: \'strtod_l\' undeclared (first use in this function)\n   return ((int*)(&strtod_l))[argc];\n                   ^\n/var/lib/jenkins/workspace/build/CMakeFiles/CMakeTmp/CheckSymbolExists.c:8:19: note: each undeclared identifier is reported only once for each function it appears in\n" } 
May 05 05:18:03  
May 05 05:18:03 =========== If your build fails, please take a look at the log above for possible reasons =========== 
May 05 05:18:03 Compile requests              5674 
May 05 05:18:03 Compile requests executed     3397 
May 05 05:18:03 Cache hits                    3380 
May 05 05:18:03 Cache misses                     0 
May 05 05:18:03 Cache timeouts                   0 
May 05 05:18:03 Cache read errors                0 
May 05 05:18:03 Forced recaches                  0 
May 05 05:18:03 Cache write errors               0

binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build (7/8)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 05 05:22:09 FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o

May 05 05:22:09 [4141/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_utils.cpp.o 
May 05 05:22:09 [4142/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o 
May 05 05:22:09 FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o  
USE_AVX2 -DTH_HAVE_THREAD -fno-strict-aliasing -Wno-write-strings -Wno-strict-aliasing -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-missing-braces -fopenmp -std=gnu++14 -MD -MT caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o -MF caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o.d -o caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o -c ../torch/csrc/distributed/c10d/comm.cpp 
May 05 05:22:09 In file included from ../aten/src/ATen/cuda/CUDAEvent.h:3:0, 
May 05 05:22:09                  from ../torch/csrc/distributed/c10d/reducer.h:11, 
May 05 05:22:09                  from ../torch/csrc/distributed/c10d/comm.cpp:6: 
May 05 05:22:09 ../aten/src/ATen/cuda/ATenCUDAGeneral.h:3:18: fatal error: cuda.h: No such file or directory 
May 05 05:22:09 compilation terminated. 
May 05 05:22:09 [4143/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o 
May 05 05:22:09 FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o  
USE_AVX2 -DTH_HAVE_THREAD -fno-strict-aliasing -Wno-write-strings -Wno-strict-aliasing -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-missing-braces -fopenmp -std=gnu++14 -MD -MT caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o -MF caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o.d -o caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/init.cpp.o -c ../torch/csrc/distributed/c10d/init.cpp 
May 05 05:22:09 In file included from ../aten/src/ATen/cuda/CUDAEvent.h:3:0, 
May 05 05:22:09                  from ../torch/csrc/distributed/c10d/reducer.h:11, 
May 05 05:22:09                  from ../torch/csrc/distributed/c10d/init.cpp:27: 
May 05 05:22:09 ../aten/src/ATen/cuda/ATenCUDAGeneral.h:3:18: fatal error: cuda.h: No such file or directory 
May 05 05:22:09 compilation terminated. 
May 05 05:22:09 [4144/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_interface.cpp.o 
May 05 05:22:09 [4145/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_rewriter.cpp.o 
May 05 05:22:10 [4146/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_matcher.cpp.o 
May 05 05:22:10 [4147/4187] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_schema_matching.cpp.o

binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_build (8/8)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

May 05 05:22:38 /pytorch/aten/src/ATen/cuda/ATenCUDAGeneral.h:3:10: fatal error: cuda.h: No such file or directory

May 05 05:22:32 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_schema_matching.cpp.o 
May 05 05:22:33 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_matcher.cpp.o 
May 05 05:22:33 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_rewriter.cpp.o 
May 05 05:22:34 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_subgraph_utils.cpp.o 
May 05 05:22:37 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/__/test/cpp/jit/test_utils.cpp.o 
May 05 05:22:37 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/autograd/init.cpp.o 
May 05 05:22:38 [100%] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o 
May 05 05:22:38 In file included from /pytorch/aten/src/ATen/cuda/CUDAEvent.h:3:0, 
May 05 05:22:38                  from /pytorch/torch/csrc/distributed/c10d/reducer.h:11, 
May 05 05:22:38                  from /pytorch/torch/csrc/distributed/c10d/comm.cpp:6: 
May 05 05:22:38 /pytorch/aten/src/ATen/cuda/ATenCUDAGeneral.h:3:10: fatal error: cuda.h: No such file or directory 
May 05 05:22:38  #include <cuda.h> 
May 05 05:22:38           ^~~~~~~~ 
May 05 05:22:38 compilation terminated. 
May 05 05:22:38 gmake[2]: *** [caffe2/torch/CMakeFiles/torch_python.dir/csrc/distributed/c10d/comm.cpp.o] Error 1 
May 05 05:22:38 gmake[2]: *** Waiting for unfinished jobs.... 
May 05 05:22:49 gmake[1]: *** [caffe2/torch/CMakeFiles/torch_python.dir/all] Error 2 
May 05 05:22:49 gmake: *** [all] Error 2 
May 05 05:22:49 Traceback (most recent call last): 
May 05 05:22:49   File "setup.py", line 740, in <module> 
May 05 05:22:49     build_deps()

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

See how this bot performed.

This comment has been revised 12 times.

…erent streams." Differential Revision: [D21396403](https://our.internmc.facebook.com/intern/diff/D21396403/) [ghstack-poisoned]

Pull Request resolved: #37807 ghstack-source-id: 103467345 Differential Revision: [D21396403](https://our.internmc.facebook.com/intern/diff/D21396403/)

rohan-varma · 2020-05-07T02:57:52Z

not a valid fix

[WIP][DDP] Synchronize event copies that can occur on different streams.

1d60e50

Differential Revision: [D21396403](https://our.internmc.facebook.com/intern/diff/D21396403/) [ghstack-poisoned]

rohan-varma requested review from mrshenli, pietern and zhaojuanmao as code owners May 5, 2020 03:04

rohan-varma marked this pull request as draft May 5, 2020 03:09

rohan-varma marked this pull request as ready for review May 5, 2020 04:39

Update on "[WIP][DDP] Synchronize event copies that can occur on diff…

e8aa198

…erent streams." Differential Revision: [D21396403](https://our.internmc.facebook.com/intern/diff/D21396403/) [ghstack-poisoned]

rohan-varma marked this pull request as draft May 5, 2020 04:41

Update on "[WIP][DDP] Synchronize event copies that can occur on diff…

6d2f90d

…erent streams." Differential Revision: [D21396403](https://our.internmc.facebook.com/intern/diff/D21396403/) [ghstack-poisoned]

rohan-varma mentioned this pull request May 5, 2020

[not for land] backward callback stream issue #37858

Closed

rohan-varma closed this May 7, 2020

facebook-github-bot deleted the gh/rohan-varma/117/head branch June 6, 2020 14:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP][DDP] Synchronize event copies that can occur on different streams.#37807

[WIP][DDP] Synchronize event copies that can occur on different streams.#37807
rohan-varma wants to merge 3 commits intogh/rohan-varma/117/basefrom
gh/rohan-varma/117/head

rohan-varma commented May 5, 2020 •

edited

Loading

Uh oh!

dr-ci Bot commented May 5, 2020 •

edited

Loading

Uh oh!

rohan-varma commented May 7, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

rohan-varma commented May 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr-ci Bot commented May 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 Build failures summary and remediations

🕵️ 8 new failures recognized by patterns

caffe2_onnx_main_py3_6_clang7_ubuntu16_04_build (1/8)

pytorch_linux_bionic_py3_6_clang9_build (2/8)

pytorch_linux_xenial_py3_clang5_asan_build (3/8)

pytorch_macos_10_13_py3_build (4/8)

pytorch_xla_linux_bionic_py3_6_clang9_build (5/8)

pytorch_linux_xenial_py3_6_gcc5_4_build (6/8)

binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build (7/8)

binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_build (8/8)

Uh oh!

rohan-varma commented May 7, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

rohan-varma commented May 5, 2020 •

edited

Loading

dr-ci Bot commented May 5, 2020 •

edited

Loading