Skip to content

[quant] Histogram Observer support List[Tensor]#38436

Closed
jerryzh168 wants to merge 21 commits intogh/jerryzh168/318/basefrom
gh/jerryzh168/318/head
Closed

[quant] Histogram Observer support List[Tensor]#38436
jerryzh168 wants to merge 21 commits intogh/jerryzh168/318/basefrom
gh/jerryzh168/318/head

Conversation

@jerryzh168
Copy link
Copy Markdown
Contributor

@jerryzh168 jerryzh168 commented May 13, 2020

Stack from ghstack:

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
@dr-ci
Copy link
Copy Markdown

dr-ci Bot commented May 13, 2020

💊 CI failures summary and remediations

As of commit 16df832 (more details on the Dr. CI page):



🕵️ 8 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See CircleCI build pytorch_linux_bionic_py3_6_clang9_test (1/8)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

May 21 00:04:12 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n ^\n" }
May 21 00:04:12     raise RuntimeError(message) 
May 21 00:04:12 RuntimeError: test_quantization failed! 
May 21 00:04:12  
May 21 00:04:12 real	15m50.809s 
May 21 00:04:12 user	18m7.894s 
May 21 00:04:12 sys	1m56.338s 
May 21 00:04:12 + cleanup 
May 21 00:04:12 + retcode=1 
May 21 00:04:12 + set +x 
May 21 00:04:12 =================== sccache compilation log =================== 
May 21 00:04:12 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n                       ^\n" } 
May 21 00:04:12  
May 21 00:04:12 =========== If your build fails, please take a look at the log above for possible reasons =========== 
May 21 00:04:12 Compile requests                 64 
May 21 00:04:12 Compile requests executed        35 
May 21 00:04:12 Cache hits                       27 
May 21 00:04:12 Cache misses                      7 
May 21 00:04:12 Cache timeouts                    0 
May 21 00:04:12 Cache read errors                 0 
May 21 00:04:12 Forced recaches                   0 
May 21 00:04:12 Cache write errors                0 

See CircleCI build pytorch_windows_vs2019_py36_cpu_test2 (2/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

RuntimeError: test_quantization failed!
Generated XML report: test-reports\python-unittest\TEST-quantization.test_quantized_tensor.TestQuantizedTensor-20200521000139.xml 
Generated XML report: test-reports\python-unittest\TEST-quantization.test_workflow_module.TestRecordHistogramObserver-20200521000139.xml 
Generated XML report: test-reports\python-unittest\TEST-quantization.test_backward_compatibility.TestSerialization-20200521000139.xml 
Generated XML report: test-reports\python-unittest\TEST-quantization.test_quantized_module.TestStaticQuantizedModule-20200521000139.xml 
Generated XML report: test-reports\python-unittest\TEST-quantization.test_quantized_op.TestQNNPackOps-20200521000139.xml 
Traceback (most recent call last): 
  File "run_test.py", line 691, in <module> 
    main() 
  File "run_test.py", line 684, in main 
    raise RuntimeError(message) 
RuntimeError: test_quantization failed! 
 
(base) circleci@PACKER-5E9D8C4B C:\Users\circleci\project\test>if ERRORLEVEL 1 exit /b 1  
+ cleanup
+ retcode=1
+ set +x

See CircleCI build pytorch_linux_xenial_py3_6_gcc5_4_ge_config_simple_test (3/8)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

May 21 00:09:46 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n ^\n" }
May 21 00:09:45 Traceback (most recent call last): 
May 21 00:09:45   File "test/run_test.py", line 691, in <module> 
May 21 00:09:45     main() 
May 21 00:09:45   File "test/run_test.py", line 684, in main 
May 21 00:09:45     raise RuntimeError(message) 
May 21 00:09:45 RuntimeError: test_quantization failed! 
May 21 00:09:46 + cleanup 
May 21 00:09:46 + retcode=1 
May 21 00:09:46 + set +x 
May 21 00:09:46 =================== sccache compilation log =================== 
May 21 00:09:46 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n                       ^\n" } 
May 21 00:09:46  
May 21 00:09:46 =========== If your build fails, please take a look at the log above for possible reasons =========== 
May 21 00:09:46 Compile requests                 64 
May 21 00:09:46 Compile requests executed        35 
May 21 00:09:46 Cache hits                       27 
May 21 00:09:46 Cache misses                      7 
May 21 00:09:46 Cache timeouts                    0 
May 21 00:09:46 Cache read errors                 0 
May 21 00:09:46 Forced recaches                   0 
May 21 00:09:46 Cache write errors                0 

See CircleCI build pytorch_linux_xenial_py3_6_gcc5_4_test (4/8)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

May 21 00:10:10 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n ^\n" }
May 21 00:10:10 Traceback (most recent call last): 
May 21 00:10:10   File "test/run_test.py", line 691, in <module> 
May 21 00:10:10     main() 
May 21 00:10:10   File "test/run_test.py", line 684, in main 
May 21 00:10:10     raise RuntimeError(message) 
May 21 00:10:10 RuntimeError: test_quantization failed! 
May 21 00:10:10 + cleanup 
May 21 00:10:10 + retcode=1 
May 21 00:10:10 + set +x 
May 21 00:10:10 =================== sccache compilation log =================== 
May 21 00:10:10 ERROR:sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function \'int main()\':\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected \';\' before \'}\' token\n int main() { return 0 }\n                       ^\n" } 
May 21 00:10:10  
May 21 00:10:10 =========== If your build fails, please take a look at the log above for possible reasons =========== 
May 21 00:10:10 Compile requests                 64 
May 21 00:10:10 Compile requests executed        35 
May 21 00:10:10 Cache hits                       27 
May 21 00:10:10 Cache misses                      7 
May 21 00:10:10 Cache timeouts                    0 
May 21 00:10:10 Cache read errors                 0 
May 21 00:10:10 Forced recaches                   0 
May 21 00:10:10 Cache write errors                0 

See CircleCI build pytorch_macos_10_13_py3_test (5/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

May 20 17:14:26 RuntimeError: test_quantization failed!
May 20 17:14:26 Generated XML report: test-reports/dist-gloo/TEST-quantization.test_quantized_module.TestStaticQuantizedModule-20200520171034.xml 
May 20 17:14:26 Generated XML report: test-reports/dist-gloo/TEST-quantization.test_quantized_module.TestDynamicQuantizedModule-20200520171034.xml 
May 20 17:14:26 Generated XML report: test-reports/dist-gloo/TEST-quantization.test_quantize.TestFusion-20200520171034.xml 
May 20 17:14:26 Generated XML report: test-reports/dist-gloo/TEST-quantization.test_quantize.TestPostTrainingDynamic-20200520171034.xml 
May 20 17:14:26 Generated XML report: test-reports/dist-gloo/TEST-quantization.test_quantized_op.TestQNNPackOps-20200520171034.xml 
May 20 17:14:26 Traceback (most recent call last): 
May 20 17:14:26   File "test/run_test.py", line 691, in <module> 
May 20 17:14:26     main() 
May 20 17:14:26   File "test/run_test.py", line 684, in main 
May 20 17:14:26     raise RuntimeError(message) 
May 20 17:14:26 RuntimeError: test_quantization failed! 
May 20 17:14:27 + cleanup 
May 20 17:14:27 + retcode=1 
May 20 17:14:27 + set +x 

See CircleCI build pytorch_windows_vs2019_py36_cuda10.1_on_cpu_test2 (6/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

RuntimeError: test_quantization failed!
Generated XML report: test-reports\python-unittest\TEST-quantization.test_quantized_tensor.TestQuantizedTensor-20200521001328.xml 
Generated XML report: test-reports\python-unittest\TEST-quantization.test_workflow_module.TestRecordHistogramObserver-20200521001328.xml 
Generated XML report: test-reports\python-unittest\TEST-quantization.test_backward_compatibility.TestSerialization-20200521001328.xml 
Generated XML report: test-reports\python-unittest\TEST-quantization.test_quantized_module.TestStaticQuantizedModule-20200521001328.xml 
Generated XML report: test-reports\python-unittest\TEST-quantization.test_quantized_op.TestQNNPackOps-20200521001328.xml 
Traceback (most recent call last): 
  File "run_test.py", line 691, in <module> 
    main() 
  File "run_test.py", line 684, in main 
    raise RuntimeError(message) 
RuntimeError: test_quantization failed! 
 
(base) circleci@PACKER-5E9D8C4B C:\Users\circleci\project\test>if ERRORLEVEL 1 exit /b 1  
+ cleanup
+ retcode=1
+ set +x

See CircleCI build pytorch_windows_vs2019_py36_cuda10.1_test2 (7/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

RuntimeError: test_quantization failed!
Generated XML report: test-reports\python-unittest\TEST-quantization.test_quantized_tensor.TestQuantizedTensor-20200521002941.xml 
Generated XML report: test-reports\python-unittest\TEST-quantization.test_workflow_module.TestRecordHistogramObserver-20200521002941.xml 
Generated XML report: test-reports\python-unittest\TEST-quantization.test_backward_compatibility.TestSerialization-20200521002941.xml 
Generated XML report: test-reports\python-unittest\TEST-quantization.test_quantized_module.TestStaticQuantizedModule-20200521002941.xml 
Generated XML report: test-reports\python-unittest\TEST-quantization.test_quantized_op.TestQNNPackOps-20200521002941.xml 
Traceback (most recent call last): 
  File "run_test.py", line 691, in <module> 
    main() 
  File "run_test.py", line 684, in main 
    raise RuntimeError(message) 
RuntimeError: test_quantization failed! 
 
(base) circleci@PACKER-5E9E037A C:\Users\circleci\project\test>if ERRORLEVEL 1 exit /b 1  
+ cleanup
+ retcode=1
+ set +x

See CircleCI build pytorch_linux_xenial_py3_clang5_asan_test (8/8)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

May 20 23:54:24 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /var/lib/jenkins/workspace/aten/src/ATen/Utils.cpp:11:3 in
May 20 23:54:23     #7 0x55907f0db74b in PyEval_EvalCode /tmp/build/80754af9/python_1585002248360/work/Python/ceval.c:731 
May 20 23:54:23     #8 0x55907f15b633 in run_mod /tmp/build/80754af9/python_1585002248360/work/Python/pythonrun.c:1025 
May 20 23:54:23     #9 0x55907f15b6cc in PyRun_StringFlags /tmp/build/80754af9/python_1585002248360/work/Python/pythonrun.c:949 
May 20 23:54:23     #10 0x55907f15b72e in PyRun_SimpleStringFlags /tmp/build/80754af9/python_1585002248360/work/Python/pythonrun.c:445 
May 20 23:54:23     #11 0x55907f15f532 in run_command /tmp/build/80754af9/python_1585002248360/work/Modules/main.c:301 
May 20 23:54:23     #12 0x55907f15f532 in Py_Main /tmp/build/80754af9/python_1585002248360/work/Modules/main.c:749 
May 20 23:54:23     #13 0x55907f02a1fd in main /tmp/build/80754af9/python_1585002248360/work/Programs/python.c:69 
May 20 23:54:24     #14 0x7fbfab94682f in __libc_start_main /build/glibc-LK5gWL/glibc-2.23/csu/../csu/libc-start.c:291 
May 20 23:54:24     #15 0x55907f108c29 in _start /home/rdonnelly/mc/conda-bld/compilers_linux-64_1534865402226/work/.build/src/glibc-2.12.2/csu/../sysdeps/x86_64/elf/start.S:103 
May 20 23:54:24  
May 20 23:54:24 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /var/lib/jenkins/workspace/aten/src/ATen/Utils.cpp:11:3 in  
May 20 23:54:24 + retcode=1 
May 20 23:54:24 + set -e 
May 20 23:54:24 + return 1 
May 20 23:54:24 + [[ pytorch-linux-xenial-py3-clang5-asan-test == *-NO_AVX-* ]] 
May 20 23:54:24 + [[ pytorch-linux-xenial-py3-clang5-asan-test == *-NO_AVX2-* ]] 
May 20 23:54:24 + '[' -n https://github.com/pytorch/pytorch/pull/38436 ']' 
May 20 23:54:24 ++ mktemp 
May 20 23:54:24 + DETERMINE_FROM=/tmp/tmp.6d0yLf30RJ 
May 20 23:54:24 + file_diff_from_base /tmp/tmp.6d0yLf30RJ 
May 20 23:54:24 + set +e 

❄️ 1 failure tentatively classified as flaky

but reruns have not yet been triggered to confirm:

See CircleCI build pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun) ❄️

May 21 00:44:12 ConnectionResetError: [Errno 104] Connection reset by peer
May 21 00:44:12   File "/opt/conda/lib/python3.6/multiprocessing/connection.py", line 455, in accept 
May 21 00:44:12     deliver_challenge(c, self._authkey) 
May 21 00:44:12   File "/opt/conda/lib/python3.6/multiprocessing/connection.py", line 722, in deliver_challenge 
May 21 00:44:12     response = connection.recv_bytes(256)        # reject large message 
May 21 00:44:12   File "/opt/conda/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes 
May 21 00:44:12     buf = self._recv_bytes(maxlength) 
May 21 00:44:12   File "/opt/conda/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes 
May 21 00:44:12     buf = self._recv(4) 
May 21 00:44:12   File "/opt/conda/lib/python3.6/multiprocessing/connection.py", line 379, in _recv 
May 21 00:44:12     chunk = read(handle, remaining) 
May 21 00:44:12 ConnectionResetError: [Errno 104] Connection reset by peer 
May 21 00:44:13 /opt/conda/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 14 leaked semaphores to clean up at shutdown 
May 21 00:44:13   len(cache)) 
May 21 00:44:15 Process ErrorTrackingProcess-126: 
May 21 00:44:15 Traceback (most recent call last): 
May 21 00:44:15   File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap 
May 21 00:44:15     self.run() 
May 21 00:44:15   File "/var/lib/jenkins/workspace/test/test_dataloader.py", line 362, in run 
May 21 00:44:15     super(ErrorTrackingProcess, self).run() 
May 21 00:44:15   File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, in run 
May 21 00:44:15     self._target(*self._args, **self._kwargs) 

Extra GitHub checks: 1 failed


ci.pytorch.org: 1 failed


This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 127 times.

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
jerryzh168 added a commit that referenced this pull request May 14, 2020
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

ghstack-source-id: 35121bd
Pull Request resolved: #38436
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
@jerryzh168 jerryzh168 requested a review from eellison May 14, 2020 18:20
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
jerryzh168 added a commit that referenced this pull request May 14, 2020
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

ghstack-source-id: 225f955
Pull Request resolved: #38436
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants