[WIP] Fix cudnn channels_last descriptio problem by VitalyFedyunin · Pull Request #31644 · pytorch/pytorch

VitalyFedyunin · 2019-12-27T01:24:50Z

Stack from ghstack:

[WIP] Fix cudnn channels_last descriptio problem #31644 [WIP] Fix cudnn channels_last descriptio problem
Add memory_format support to zeros, ones, full #31131 Add memory_format support to zeros, ones, full

[ghstack-poisoned]

ghstack-source-id: 9167f1a Pull Request resolved: #31644

kostmo · 2019-12-27T03:09:54Z

💊 CircleCI build failures summary and remediations

As of commit e437bca:

15/16 failures introduced in this PR
1/16 recognized as flaky ❄️
- Re-run these jobs?

Detailed failure analysis

One may explore the probable reasons each build failed interactively on the Dr. CI website.

🕵️ 15 new failures recognized by patterns

The following build failures do not appear to be due to upstream breakage:

pytorch_macos_10_13_py3_build (1/15)

Step: "Build" (full log | pattern match details)

Dec 26 19:46:00 FAILED: lib/libtorch_cpu.dylib

Dec 26 19:45:58 /Applications/Xcode-9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: lib/libCaffe2_perfkernels_avx2.a(common_avx2.cc.o) has no symbols 
Dec 26 19:45:58 [2639/2982] Linking CXX static library lib/libCaffe2_perfkernels_avx512.a 
Dec 26 19:45:58 /Applications/Xcode-9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: lib/libCaffe2_perfkernels_avx512.a(common_avx512.cc.o) has no symbols 
Dec 26 19:45:58 /Applications/Xcode-9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: lib/libCaffe2_perfkernels_avx512.a(common_avx512.cc.o) has no symbols 
Dec 26 19:45:58 warning: /Applications/Xcode-9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: warning for library: lib/libCaffe2_perfkernels_avx512.a the table of contents is empty (no object file members in the library define global symbols) 
Dec 26 19:45:59 [2640/2982] Building CXX object test_jit/CMakeFiles/test_jit.dir/gtest.cpp.o 
Dec 26 19:45:59 [2641/2982] Building CXX object test_jit/CMakeFiles/test_jit.dir/test_alias_analysis.cpp.o 
Dec 26 19:46:00 [2642/2982] Building CXX object test_jit/CMakeFiles/test_jit.dir/test_argument_spec.cpp.o 
Dec 26 19:46:00 [2643/2982] Building CXX object test_jit/CMakeFiles/test_jit.dir/test_autodiff.cpp.o 
Dec 26 19:46:00 [2644/2982] Linking CXX shared library lib/libtorch_cpu.dylib 
Dec 26 19:46:00 FAILED: lib/libtorch_cpu.dylib  
/distiller/project/build/lib/libcaffe2_protos.a lib/libclog.a lib/libasmjit.a /Users/distiller/workspace/miniconda3/lib/libuv.a -lpthread -Wl,-force_load,/Users/distiller/project/build/lib/libonnx.a lib/libonnx_proto.a lib/libprotobuf.a -Wl,-force_load,/Users/distiller/project/build/lib/libCaffe2_perfkernels_avx.a -Wl,-force_load,/Users/distiller/project/build/lib/libCaffe2_perfkernels_avx2.a lib/libc10.dylib -Wl,-force_load,/Users/distiller/project/build/lib/libCaffe2_perfkernels_avx512.a && : 
Dec 26 19:46:01 ld: warning: ignoring file caffe2/CMakeFiles/torch_cpu.dir/operators/rnn/recurrent_network_op.cc.o, file was built for unsupported file format ( 0x2E 0xEA 0xE2 0xFF 0x1F 0xDF 0x05 0x16 0x17 0x05 0x17 0xD4 0x45 0x2C 0x31 0xBC ) which is not the architecture being linked (x86_64): caffe2/CMakeFiles/torch_cpu.dir/operators/rnn/recurrent_network_op.cc.o 
Dec 26 19:46:01 ld: warning: ignoring file caffe2/CMakeFiles/torch_cpu.dir/quantization/server/batch_matmul_dnnlowp_op.cc.o, file was built for unsupported file format ( 0xB3 0xDB 0x8F 0xE4 0x8D 0xA5 0xBD 0xB6 0xCA 0xDE 0xCD 0xE1 0xBE 0x2A 0x3B 0xEB ) which is not the architecture being linked (x86_64): caffe2/CMakeFiles/torch_cpu.dir/quantization/server/batch_matmul_dnnlowp_op.cc.o 
Dec 26 19:46:01 Undefined symbols for architecture x86_64: 
Dec 26 19:46:01   "caffe2::detail::TypeMetaData const* caffe2::TypeMeta::_typeMetaDataInstance<caffe2::detail::ScratchWorkspaces>()", referenced from: 
Dec 26 19:46:01       caffe2::detail::ScratchWorkspaces const& caffe2::Blob::Get<caffe2::detail::ScratchWorkspaces>() const in recurrent_network_blob_fetcher_op.cc.o 
Dec 26 19:46:01 ld: symbol(s) not found for architecture x86_64 
Dec 26 19:46:01 clang: error: linker command failed with exit code 1 (use -v to see invocation) 
Dec 26 19:46:01 [2645/2982] Building CXX object test_jit/CMakeFiles/test_jit.dir/test_class_import.cpp.o 
Dec 26 19:46:01 ninja: build stopped: subcommand failed.

pytorch_macos_10_13_cuda10_0_cudnn7_py3_build (2/15)

Step: "Build" (full log | pattern match details)

Dec 26 19:50:08 FAILED: lib/libtorch_cpu.dylib

Dec 26 19:50:05 [2639/2982] Linking CXX static library lib/libCaffe2_perfkernels_avx2.a 
Dec 26 19:50:05 /Applications/Xcode-9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: lib/libCaffe2_perfkernels_avx2.a(common_avx2.cc.o) has no symbols 
Dec 26 19:50:05 /Applications/Xcode-9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: lib/libCaffe2_perfkernels_avx2.a(common_avx2.cc.o) has no symbols 
Dec 26 19:50:05 [2640/2982] Building CXX object test_jit/CMakeFiles/test_jit.dir/gtest.cpp.o 
Dec 26 19:50:06 [2641/2982] Building CXX object test_jit/CMakeFiles/test_jit.dir/test_alias_analysis.cpp.o 
Dec 26 19:50:06 [2642/2982] Building CXX object test_jit/CMakeFiles/test_jit.dir/test_argument_spec.cpp.o 
Dec 26 19:50:07 [2643/2982] Building CXX object test_jit/CMakeFiles/test_jit.dir/test_autodiff.cpp.o 
Dec 26 19:50:07 [2644/2982] Building CXX object test_jit/CMakeFiles/test_jit.dir/test_class_import.cpp.o 
Dec 26 19:50:08 [2645/2982] Building CXX object test_jit/CMakeFiles/test_jit.dir/test_class_parser.cpp.o 
Dec 26 19:50:08 [2646/2982] Linking CXX shared library lib/libtorch_cpu.dylib 
Dec 26 19:50:08 FAILED: lib/libtorch_cpu.dylib  
/distiller/project/build/lib/libcaffe2_protos.a lib/libclog.a lib/libasmjit.a /Users/distiller/workspace/miniconda3/lib/libuv.a -lpthread -Wl,-force_load,/Users/distiller/project/build/lib/libonnx.a lib/libonnx_proto.a lib/libprotobuf.a -Wl,-force_load,/Users/distiller/project/build/lib/libCaffe2_perfkernels_avx.a -Wl,-force_load,/Users/distiller/project/build/lib/libCaffe2_perfkernels_avx2.a lib/libc10.dylib -Wl,-force_load,/Users/distiller/project/build/lib/libCaffe2_perfkernels_avx512.a && : 
Dec 26 19:50:08 ld: warning: ignoring file caffe2/CMakeFiles/torch_cpu.dir/operators/rnn/recurrent_network_op.cc.o, file was built for unsupported file format ( 0x2E 0xEA 0xE2 0xFF 0x1F 0xDF 0x05 0x16 0x17 0x05 0x17 0xD4 0x45 0x2C 0x31 0xBC ) which is not the architecture being linked (x86_64): caffe2/CMakeFiles/torch_cpu.dir/operators/rnn/recurrent_network_op.cc.o 
Dec 26 19:50:08 ld: warning: ignoring file caffe2/CMakeFiles/torch_cpu.dir/quantization/server/batch_matmul_dnnlowp_op.cc.o, file was built for unsupported file format ( 0xB3 0xDB 0x8F 0xE4 0x8D 0xA5 0xBD 0xB6 0xCA 0xDE 0xCD 0xE1 0xBE 0x2A 0x3B 0xEB ) which is not the architecture being linked (x86_64): caffe2/CMakeFiles/torch_cpu.dir/quantization/server/batch_matmul_dnnlowp_op.cc.o 
Dec 26 19:50:08 Undefined symbols for architecture x86_64: 
Dec 26 19:50:08   "caffe2::detail::TypeMetaData const* caffe2::TypeMeta::_typeMetaDataInstance<caffe2::detail::ScratchWorkspaces>()", referenced from: 
Dec 26 19:50:08       caffe2::detail::ScratchWorkspaces const& caffe2::Blob::Get<caffe2::detail::ScratchWorkspaces>() const in recurrent_network_blob_fetcher_op.cc.o 
Dec 26 19:50:08 ld: symbol(s) not found for architecture x86_64 
Dec 26 19:50:08 clang: error: linker command failed with exit code 1 (use -v to see invocation) 
Dec 26 19:50:08 [2647/2982] Building CXX object test_jit/CMakeFiles/test_jit.dir/test_class_type.cpp.o 
Dec 26 19:50:08 ninja: build stopped: subcommand failed.

pytorch_linux_xenial_py3_6_gcc5_4_ge_config_legacy_test (3/15)

Step: "Test" (full log | pattern match details)

Dec 27 04:00:06 RuntimeError: test_jit_legacy failed!

Dec 27 04:00:06 Ran 2313 tests in 150.107s 
Dec 27 04:00:06  
Dec 27 04:00:06 FAILED (errors=2, skipped=58, expected failures=1) 
Dec 27 04:00:06  
Dec 27 04:00:06 Generating XML reports... 
Dec 27 04:00:06 Traceback (most recent call last): 
Dec 27 04:00:06   File "test/run_test.py", line 456, in <module> 
Dec 27 04:00:06     main() 
Dec 27 04:00:06   File "test/run_test.py", line 449, in main 
Dec 27 04:00:06     raise RuntimeError(message) 
Dec 27 04:00:06 RuntimeError: test_jit_legacy failed! 
Dec 27 04:00:06 + cleanup 
Dec 27 04:00:06 + retcode=1 
Dec 27 04:00:06 + set +x 
Dec 27 04:00:06 =================== sccache compilation log =================== 
Dec 27 04:00:06 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Dec 27 04:00:06 Compile requests               304 
Dec 27 04:00:06 Compile requests executed        0 
Dec 27 04:00:06 Cache hits                       0 
Dec 27 04:00:06 Cache misses                     0 
Dec 27 04:00:06 Cache timeouts                   0

pytorch_linux_xenial_py3_6_gcc5_4_ge_config_simple_test (4/15)

Step: "Test" (full log | pattern match details)

Dec 27 04:01:33 RuntimeError: test_jit_simple failed!

Dec 27 04:01:33 Ran 2313 tests in 84.177s 
Dec 27 04:01:33  
Dec 27 04:01:33 FAILED (errors=2, skipped=76, expected failures=1) 
Dec 27 04:01:33  
Dec 27 04:01:33 Generating XML reports... 
Dec 27 04:01:33 Traceback (most recent call last): 
Dec 27 04:01:33   File "test/run_test.py", line 456, in <module> 
Dec 27 04:01:33     main() 
Dec 27 04:01:33   File "test/run_test.py", line 449, in main 
Dec 27 04:01:33     raise RuntimeError(message) 
Dec 27 04:01:33 RuntimeError: test_jit_simple failed! 
Dec 27 04:01:33 + cleanup 
Dec 27 04:01:33 + retcode=1 
Dec 27 04:01:33 + set +x 
Dec 27 04:01:33 =================== sccache compilation log =================== 
Dec 27 04:01:33 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Dec 27 04:01:33 Compile requests                56 
Dec 27 04:01:33 Compile requests executed        0 
Dec 27 04:01:33 Cache hits                       0 
Dec 27 04:01:33 Cache misses                     0 
Dec 27 04:01:33 Cache timeouts                   0

pytorch_linux_xenial_py2_7_9_test (5/15)

Step: "Test" (full log | pattern match details)

Dec 27 04:06:22 RuntimeError: test_nn failed!

Dec 27 04:06:22 Ran 1477 tests in 456.333s 
Dec 27 04:06:22  
Dec 27 04:06:22 FAILED (errors=2, skipped=703, expected failures=3) 
Dec 27 04:06:22  
Dec 27 04:06:22 Generating XML reports... 
Dec 27 04:06:22 Traceback (most recent call last): 
Dec 27 04:06:22   File "test/run_test.py", line 456, in <module> 
Dec 27 04:06:22     main() 
Dec 27 04:06:22   File "test/run_test.py", line 449, in main 
Dec 27 04:06:22     raise RuntimeError(message) 
Dec 27 04:06:22 RuntimeError: test_nn failed! 
Dec 27 04:06:22 + cleanup 
Dec 27 04:06:22 + retcode=1 
Dec 27 04:06:22 + set +x 
Dec 27 04:06:22 =================== sccache compilation log =================== 
Dec 27 04:06:22 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Dec 27 04:06:22 Compile requests                 10 
Dec 27 04:06:22 Compile requests executed         8 
Dec 27 04:06:22 Cache hits                        2 
Dec 27 04:06:22 Cache misses                      6 
Dec 27 04:06:22 Cache timeouts                    0

pytorch_linux_xenial_py3_6_gcc5_4_test (6/15)

Step: "Test" (full log | pattern match details)

Dec 27 04:08:17 RuntimeError: test_nn failed!

Dec 27 04:08:17 Ran 1477 tests in 419.537s 
Dec 27 04:08:17  
Dec 27 04:08:17 FAILED (errors=2, skipped=685, expected failures=2) 
Dec 27 04:08:17  
Dec 27 04:08:17 Generating XML reports... 
Dec 27 04:08:17 Traceback (most recent call last): 
Dec 27 04:08:17   File "test/run_test.py", line 456, in <module> 
Dec 27 04:08:17     main() 
Dec 27 04:08:17   File "test/run_test.py", line 449, in main 
Dec 27 04:08:17     raise RuntimeError(message) 
Dec 27 04:08:17 RuntimeError: test_nn failed! 
Dec 27 04:08:17 + cleanup 
Dec 27 04:08:17 + retcode=1 
Dec 27 04:08:17 + set +x 
Dec 27 04:08:17 =================== sccache compilation log =================== 
Dec 27 04:08:17 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Dec 27 04:08:17 Compile requests                  7 
Dec 27 04:08:17 Compile requests executed         6 
Dec 27 04:08:17 Cache hits                        0 
Dec 27 04:08:17 Cache misses                      6 
Dec 27 04:08:17 Cache timeouts                    0

caffe2_onnx_py3_6_clang7_ubuntu16_04_test (7/15)

Step: "Test" (full log | pattern match details)

ore dumped) pytest "${args[@]}" -k 'not (TestOperators and test_full_like) and not (TestOperators and test_zeros_like) and not (TestOperators and test_ones_like) and not (TestModels and test_vgg16) and not (TestModels and test_vgg16_bn) and not (TestModels and test_vgg19) and not (TestModels and test_vgg19_bn)' --ignore "$top_dir/test/onnx/test_pytorch_onnx_onnxruntime.py" --ignore "$top_dir/test/onnx/test_custom_ops.py" --ignore "$top_dir/test/onnx/test_models_onnxruntime.py" "${test_paths[@]}"

Dec 27 04:16:35   File "/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py", line 286 in __call__ 
Dec 27 04:16:35   File "/usr/local/lib/python3.6/dist-packages/_pytest/main.py", line 237 in _main 
Dec 27 04:16:35   File "/usr/local/lib/python3.6/dist-packages/_pytest/main.py", line 193 in wrap_session 
Dec 27 04:16:35   File "/usr/local/lib/python3.6/dist-packages/_pytest/main.py", line 230 in pytest_cmdline_main 
Dec 27 04:16:35   File "/usr/local/lib/python3.6/dist-packages/pluggy/callers.py", line 187 in _multicall 
Dec 27 04:16:35   File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 86 in <lambda> 
Dec 27 04:16:35   File "/usr/local/lib/python3.6/dist-packages/pluggy/manager.py", line 92 in _hookexec 
Dec 27 04:16:35   File "/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py", line 286 in __call__ 
Dec 27 04:16:35   File "/usr/local/lib/python3.6/dist-packages/_pytest/config/__init__.py", line 90 in main 
Dec 27 04:16:35   File "/usr/local/bin/pytest", line 11 in <module> 
re dumped) pytest "${args[@]}" -k 'not (TestOperators and test_full_like) and not (TestOperators and test_zeros_like) and not (TestOperators and test_ones_like) and not (TestModels and test_vgg16) and not (TestModels and test_vgg16_bn) and not (TestModels and test_vgg19) and not (TestModels and test_vgg19_bn)' --ignore "$top_dir/test/onnx/test_pytorch_onnx_onnxruntime.py" --ignore "$top_dir/test/onnx/test_custom_ops.py" --ignore "$top_dir/test/onnx/test_models_onnxruntime.py" "${test_paths[@]}"

pytorch_linux_xenial_cuda9_cudnn7_py3_multigpu_test (8/15)

Step: "Test" (full log | pattern match details)

Dec 27 04:18:30 [ FAILED ] IntegrationTest.MNISTBatchNorm_CUDA

Dec 27 04:18:26 [       OK ] ParallelTest.DataParallelUsesAllAvailableCUDADevices_CUDA (0 ms) 
Dec 27 04:18:26 [ RUN      ] ParallelTest.DataParallelNumericalEquivalence_MultiCUDA 
Dec 27 04:18:30 [       OK ] ParallelTest.DataParallelNumericalEquivalence_MultiCUDA (3174 ms) 
Dec 27 04:18:30 [----------] 9 tests from ParallelTest (7183 ms total) 
Dec 27 04:18:30  
Dec 27 04:18:30 [----------] Global test environment tear-down 
Dec 27 04:18:30 [==========] 788 tests from 32 test cases ran. (80608 ms total) 
Dec 27 04:18:30 [  PASSED  ] 786 tests. 
Dec 27 04:18:30 [  FAILED  ] 2 tests, listed below: 
Dec 27 04:18:30 [  FAILED  ] IntegrationTest.MNIST_CUDA 
Dec 27 04:18:30 [  FAILED  ] IntegrationTest.MNISTBatchNorm_CUDA 
Dec 27 04:18:30  
Dec 27 04:18:30  2 FAILED TESTS 
Dec 27 04:18:32 + cleanup 
Dec 27 04:18:32 + retcode=1 
Dec 27 04:18:32 + set +x 
Dec 27 04:18:32 =================== sccache compilation log =================== 
Dec 27 04:18:32 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Dec 27 04:18:32 Compile requests                 0 
Dec 27 04:18:32 Compile requests executed        0 
Dec 27 04:18:32 Cache hits                       0

pytorch_linux_xenial_cuda9_cudnn7_py3_nogpu_test (9/15)

Step: "Test" (full log | pattern match details)

Dec 27 04:28:30 RuntimeError: test_nn failed!

Dec 27 04:28:30 Ran 1477 tests in 490.007s 
Dec 27 04:28:30  
Dec 27 04:28:30 FAILED (errors=2, skipped=685, expected failures=2) 
Dec 27 04:28:30  
Dec 27 04:28:30 Generating XML reports... 
Dec 27 04:28:30 Traceback (most recent call last): 
Dec 27 04:28:30   File "test/run_test.py", line 456, in <module> 
Dec 27 04:28:30     main() 
Dec 27 04:28:30   File "test/run_test.py", line 449, in main 
Dec 27 04:28:30     raise RuntimeError(message) 
Dec 27 04:28:30 RuntimeError: test_nn failed! 
Dec 27 04:28:30 + cleanup 
Dec 27 04:28:30 + retcode=1 
Dec 27 04:28:30 + set +x 
Dec 27 04:28:30 =================== sccache compilation log =================== 
Dec 27 04:28:30 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Dec 27 04:28:30 Compile requests                  7 
Dec 27 04:28:30 Compile requests executed         6 
Dec 27 04:28:30 Cache hits                        0 
Dec 27 04:28:30 Cache misses                      6 
Dec 27 04:28:30 Cache timeouts                    0

pytorch_linux_xenial_cuda9_cudnn7_py3_NO_AVX2_test (10/15)

Step: "Test" (full log | pattern match details)

Dec 27 04:30:00 RuntimeError: test_nn failed!

Dec 27 04:29:59 Ran 1573 tests in 593.549s 
Dec 27 04:29:59  
Dec 27 04:29:59 FAILED (failures=1, errors=15, skipped=65, expected failures=4) 
Dec 27 04:29:59  
Dec 27 04:29:59 Generating XML reports... 
Dec 27 04:30:00 Traceback (most recent call last): 
Dec 27 04:30:00   File "test/run_test.py", line 456, in <module> 
Dec 27 04:30:00     main() 
Dec 27 04:30:00   File "test/run_test.py", line 449, in main 
Dec 27 04:30:00     raise RuntimeError(message) 
Dec 27 04:30:00 RuntimeError: test_nn failed! 
Dec 27 04:30:00 + cleanup 
Dec 27 04:30:00 + retcode=1 
Dec 27 04:30:00 + set +x 
Dec 27 04:30:00 =================== sccache compilation log =================== 
Dec 27 04:30:00 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Dec 27 04:30:00 Compile requests                32 
Dec 27 04:30:00 Compile requests executed       11 
Dec 27 04:30:00 Cache hits                       1 
Dec 27 04:30:00 Cache misses                    10 
Dec 27 04:30:00 Cache timeouts                   0

pytorch_linux_xenial_cuda9_cudnn7_py3_NO_AVX_NO_AVX2_test (11/15)

Step: "Test" (full log | pattern match details)

Dec 27 04:30:01 RuntimeError: test_nn failed!

Dec 27 04:30:01 Ran 1573 tests in 604.326s 
Dec 27 04:30:01  
Dec 27 04:30:01 FAILED (failures=1, errors=15, skipped=65, expected failures=4) 
Dec 27 04:30:01  
Dec 27 04:30:01 Generating XML reports... 
Dec 27 04:30:01 Traceback (most recent call last): 
Dec 27 04:30:01   File "test/run_test.py", line 456, in <module> 
Dec 27 04:30:01     main() 
Dec 27 04:30:01   File "test/run_test.py", line 449, in main 
Dec 27 04:30:01     raise RuntimeError(message) 
Dec 27 04:30:01 RuntimeError: test_nn failed! 
Dec 27 04:30:02 + cleanup 
Dec 27 04:30:02 + retcode=1 
Dec 27 04:30:02 + set +x 
Dec 27 04:30:02 =================== sccache compilation log =================== 
Dec 27 04:30:02 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Dec 27 04:30:02 Compile requests                32 
Dec 27 04:30:02 Compile requests executed       11 
Dec 27 04:30:02 Cache hits                       1 
Dec 27 04:30:02 Cache misses                    10 
Dec 27 04:30:02 Cache timeouts                   0

pytorch_xla_linux_xenial_py3_6_clang7_test (12/15)

Step: "Test" (full log | pattern match details)

Dec 27 04:25:40 caused by: Connection refused (os error 111)

Dec 27 04:25:40 +++ eval 'extract_trap_cmd ' 
Dec 27 04:25:40 ++++ extract_trap_cmd 
Dec 27 04:25:40 ++++ printf '%s\n' '' 
Dec 27 04:25:40 +++ printf '%s\n' cleanup 
Dec 27 04:25:40 ++ trap -- ' 
Dec 27 04:25:40 cleanup' EXIT 
Dec 27 04:25:40 ++ which sccache 
Dec 27 04:25:40 ++ sccache --stop-server 
Dec 27 04:25:40 Stopping sccache server... 
Dec 27 04:25:40 error: couldn't connect to server 
Dec 27 04:25:40 caused by: Connection refused (os error 111) 
Dec 27 04:25:40 ++ true 
Dec 27 04:25:40 ++ rm /var/lib/jenkins/sccache_error.log 
Dec 27 04:25:40 ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 
Dec 27 04:25:40 ++ SCCACHE_IDLE_TIMEOUT=1200 
Dec 27 04:25:40 ++ RUST_LOG=sccache::server=error 
Dec 27 04:25:40 ++ sccache --start-server 
Dec 27 04:25:40 Starting sccache server... 
Dec 27 04:25:40 ++ sccache --zero-stats 
Dec 27 04:25:40 Compile requests                 0 
Dec 27 04:25:40 Compile requests executed        0

pytorch_linux_xenial_cuda9_cudnn7_py3_test (13/15)

Step: "Test" (full log | pattern match details)

Dec 27 04:31:29 RuntimeError: test_nn failed!

Dec 27 04:31:29 Ran 1573 tests in 600.530s 
Dec 27 04:31:29  
Dec 27 04:31:29 FAILED (failures=1, errors=15, skipped=65, expected failures=4) 
Dec 27 04:31:29  
Dec 27 04:31:29 Generating XML reports... 
Dec 27 04:31:29 Traceback (most recent call last): 
Dec 27 04:31:29   File "test/run_test.py", line 456, in <module> 
Dec 27 04:31:29     main() 
Dec 27 04:31:29   File "test/run_test.py", line 449, in main 
Dec 27 04:31:29     raise RuntimeError(message) 
Dec 27 04:31:29 RuntimeError: test_nn failed! 
Dec 27 04:31:30 + cleanup 
Dec 27 04:31:30 + retcode=1 
Dec 27 04:31:30 + set +x 
Dec 27 04:31:30 =================== sccache compilation log =================== 
Dec 27 04:31:30 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Dec 27 04:31:30 Compile requests                32 
Dec 27 04:31:30 Compile requests executed       11 
Dec 27 04:31:30 Cache hits                       1 
Dec 27 04:31:30 Cache misses                    10 
Dec 27 04:31:30 Cache timeouts                   0

pytorch_linux_xenial_cuda9_cudnn7_py3_slow_test (14/15)

Step: "Test" (full log | pattern match details)

Dec 27 04:42:28 [ FAILED ] JitTest.DCE

Dec 27 04:42:28 [ RUN      ] JitTest.ModuleConversion_CUDA 
Dec 27 04:42:28 [       OK ] JitTest.ModuleConversion_CUDA (1 ms) 
Dec 27 04:42:28 [ RUN      ] JitTest.Interp_CUDA 
Dec 27 04:42:28 [       OK ] JitTest.Interp_CUDA (0 ms) 
Dec 27 04:42:28 [----------] 75 tests from JitTest (3796 ms total) 
Dec 27 04:42:28  
Dec 27 04:42:28 [----------] Global test environment tear-down 
Dec 27 04:42:28 [==========] 75 tests from 1 test case ran. (3796 ms total) 
Dec 27 04:42:28 [  PASSED  ] 74 tests. 
Dec 27 04:42:28 [  FAILED  ] 1 test, listed below: 
Dec 27 04:42:28 [  FAILED  ] JitTest.DCE 
Dec 27 04:42:28  
Dec 27 04:42:28  1 FAILED TEST 
Dec 27 04:42:28 + cleanup 
Dec 27 04:42:28 + retcode=1 
Dec 27 04:42:28 + set +x 
Dec 27 04:42:28 =================== sccache compilation log =================== 
Dec 27 04:42:28 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/tmp/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/tmp/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n                       ^\n" } 
Dec 27 04:42:28  
Dec 27 04:42:28 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Dec 27 04:42:28 Compile requests               128

pytorch_linux_xenial_py3_clang5_asan_test (15/15)

Step: "Test" (full log | pattern match details)

Dec 27 04:42:38 RuntimeError: test_nn failed!

Dec 27 04:42:38 Ran 1477 tests in 1907.409s 
Dec 27 04:42:38  
Dec 27 04:42:38 FAILED (errors=2, skipped=685, expected failures=2) 
Dec 27 04:42:38  
Dec 27 04:42:38 Generating XML reports... 
Dec 27 04:42:38 Traceback (most recent call last): 
Dec 27 04:42:38   File "test/run_test.py", line 456, in <module> 
Dec 27 04:42:38     main() 
Dec 27 04:42:38   File "test/run_test.py", line 449, in main 
Dec 27 04:42:38     raise RuntimeError(message) 
Dec 27 04:42:38 RuntimeError: test_nn failed! 
Dec 27 04:42:38 + cleanup 
Dec 27 04:42:38 + retcode=1 
Dec 27 04:42:38 + set +x 
Dec 27 04:42:38 =================== sccache compilation log =================== 
Dec 27 04:42:38 =========== If your build fails, please take a look at the log above for possible reasons =========== 
Dec 27 04:42:38 Compile requests                 0 
Dec 27 04:42:38 Compile requests executed        0 
Dec 27 04:42:38 Cache hits                       0 
Dec 27 04:42:38 Cache misses                     0 
Dec 27 04:42:38 Cache timeouts                   0

❄️ 1 failure recognized as flaky

The following build failures have been detected as flaky and may not be your fault:

caffe2_onnx_py2_gcc5_ubuntu16_04_test (1/1)

Step: "Test" (full log | pattern match details) ❄️

ore dumped) pytest "${args[@]}" -k 'not (TestOperators and test_full_like) and not (TestOperators and test_zeros_like) and not (TestOperators and test_ones_like) and not (TestModels and test_vgg16) and not (TestModels and test_vgg16_bn) and not (TestModels and test_vgg19) and not (TestModels and test_vgg19_bn)' --ignore "$top_dir/test/onnx/test_pytorch_onnx_onnxruntime.py" --ignore "$top_dir/test/onnx/test_custom_ops.py" --ignore "$top_dir/test/onnx/test_models_onnxruntime.py" "${test_paths[@]}"

Dec 27 04:16:43 test/onnx/test_operators.py::TestOperators::test_empty_like PASSED       [  2%] 
Dec 27 04:16:43 test/onnx/test_operators.py::TestOperators::test_empty_like_opset7 PASSED [  2%] 
Dec 27 04:16:43 test/onnx/test_operators.py::TestOperators::test_equal PASSED            [  2%] 
Dec 27 04:16:43 test/onnx/test_operators.py::TestOperators::test_erf PASSED              [  2%] 
Dec 27 04:16:43 test/onnx/test_operators.py::TestOperators::test_exp PASSED              [  2%] 
Dec 27 04:16:43 test/onnx/test_operators.py::TestOperators::test_expand PASSED           [  2%] 
Dec 27 04:16:43 test/onnx/test_operators.py::TestOperators::test_flatten PASSED          [  2%] 
Dec 27 04:16:43 test/onnx/test_operators.py::TestOperators::test_flatten2D PASSED        [  2%] 
Dec 27 04:16:43 test/onnx/test_operators.py::TestOperators::test_fmod PASSED             [  2%] 
Dec 27 04:16:43 test/onnx/test_operators.py::TestOperators::test_frobenius_norm PASSED   [  2%] 
re dumped) pytest "${args[@]}" -k 'not (TestOperators and test_full_like) and not (TestOperators and test_zeros_like) and not (TestOperators and test_ones_like) and not (TestModels and test_vgg16) and not (TestModels and test_vgg16_bn) and not (TestModels and test_vgg19) and not (TestModels and test_vgg19_bn)' --ignore "$top_dir/test/onnx/test_pytorch_onnx_onnxruntime.py" --ignore "$top_dir/test/onnx/test_custom_ops.py" --ignore "$top_dir/test/onnx/test_models_onnxruntime.py" "${test_paths[@]}"

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

This comment has been revised 15 times.

[ghstack-poisoned]

ghstack-source-id: e34ac60 Pull Request resolved: #31644

[WIP] Fix cudnn channels_last descriptio problem

1391e7f

[ghstack-poisoned]

VitalyFedyunin mentioned this pull request Dec 27, 2019

Add memory_format support to zeros, ones, full #31131

Closed

VitalyFedyunin added a commit that referenced this pull request Dec 27, 2019

[WIP] Fix cudnn channels_last descriptio problem

2a49b24

ghstack-source-id: 9167f1a Pull Request resolved: #31644

experiemnts on "[WIP] Fix cudnn channels_last descriptio problem"

e437bca

[ghstack-poisoned]

VitalyFedyunin added a commit that referenced this pull request Dec 27, 2019

[WIP] Fix cudnn channels_last descriptio problem

7a5d740

ghstack-source-id: e34ac60 Pull Request resolved: #31644

VitalyFedyunin closed this Dec 30, 2019

facebook-github-bot deleted the gh/VitalyFedyunin/107/head branch January 30, 2020 15:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Fix cudnn channels_last descriptio problem#31644

[WIP] Fix cudnn channels_last descriptio problem#31644
VitalyFedyunin wants to merge 2 commits intogh/VitalyFedyunin/107/basefrom
gh/VitalyFedyunin/107/head

VitalyFedyunin commented Dec 27, 2019 •

edited

Loading

Uh oh!

kostmo commented Dec 27, 2019 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

VitalyFedyunin commented Dec 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kostmo commented Dec 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CircleCI build failures summary and remediations

Detailed failure analysis

🕵️ 15 new failures recognized by patterns

pytorch_macos_10_13_py3_build (1/15)

pytorch_macos_10_13_cuda10_0_cudnn7_py3_build (2/15)

pytorch_linux_xenial_py3_6_gcc5_4_ge_config_legacy_test (3/15)

pytorch_linux_xenial_py3_6_gcc5_4_ge_config_simple_test (4/15)

pytorch_linux_xenial_py2_7_9_test (5/15)

pytorch_linux_xenial_py3_6_gcc5_4_test (6/15)

caffe2_onnx_py3_6_clang7_ubuntu16_04_test (7/15)

pytorch_linux_xenial_cuda9_cudnn7_py3_multigpu_test (8/15)

pytorch_linux_xenial_cuda9_cudnn7_py3_nogpu_test (9/15)

pytorch_linux_xenial_cuda9_cudnn7_py3_NO_AVX2_test (10/15)

pytorch_linux_xenial_cuda9_cudnn7_py3_NO_AVX_NO_AVX2_test (11/15)

pytorch_xla_linux_xenial_py3_6_clang7_test (12/15)

pytorch_linux_xenial_cuda9_cudnn7_py3_test (13/15)

pytorch_linux_xenial_cuda9_cudnn7_py3_slow_test (14/15)

pytorch_linux_xenial_py3_clang5_asan_test (15/15)

❄️ 1 failure recognized as flaky

caffe2_onnx_py2_gcc5_ubuntu16_04_test (1/1)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

VitalyFedyunin commented Dec 27, 2019 •

edited

Loading

kostmo commented Dec 27, 2019 •

edited

Loading