Skip to content

[Do not commit] Re-introduce bug to test pass bisection mechanism.#50519

Closed
ZolotukhinM wants to merge 1 commit intogh/ZolotukhinM/393/basefrom
gh/ZolotukhinM/393/head
Closed

[Do not commit] Re-introduce bug to test pass bisection mechanism.#50519
ZolotukhinM wants to merge 1 commit intogh/ZolotukhinM/393/basefrom
gh/ZolotukhinM/393/head

Conversation

@ZolotukhinM
Copy link
Copy Markdown

@ZolotukhinM ZolotukhinM commented Jan 14, 2021 •
edited
Loading

Stack from ghstack:

Just to reproduce the bug and make sure that the bisection works.

Differential Revision: D25908596

[Do not commit] Re-introduce bug to test pass bisection mechanism.
5a0216b 
Just to reproduce the bug and make sure that the bisection works.

[ghstack-poisoned]
@ZolotukhinM ZolotukhinM mentioned this pull request Jan 14, 2021
Closed
ZolotukhinM pushed a commit that referenced this pull request Jan 14, 2021
[Do not commit] Re-introduce bug to test pass bisection mechanism.
e93c46c 
Just to reproduce the bug and make sure that the bisection works.

ghstack-source-id: 071151b
Pull Request resolved: #50519
@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Jan 14, 2021 •
edited
Loading

💊 CI failures summary and remediations

As of commit 5a0216b (more details on the Dr. CI page):

🕵️ 2 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See CircleCI build pytorch_windows_vs2019_py36_cuda10.1_test2 (1/2)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

AssertionError: False is not true : Tensors failed to compare as equal!With rtol=1.3e-06 and atol=1e-05, found 368 element(s) (out of 400) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 4.5334222310064783e+18 (1.9716532230377197 vs. 4.5334222310064783e+18), which occurred at index (7, 1). 
======================================================================
FAIL: test_multiple_outputs (__main__.TestTensorExprFuser)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_tensorexpr.py", line 1650, in test_multiple_outputs
    self.assertEqual(ref, exp)
  File "C:\Users\circleci\project\build\win_tmp\build\torch\testing\_internal\common_utils.py", line 1214, in assertEqual
    exact_dtype=exact_dtype, exact_device=exact_device)
  File "C:\Users\circleci\project\build\win_tmp\build\torch\testing\_internal\common_utils.py", line 1180, in assertEqual
    super().assertTrue(result, msg=self._get_assert_msg(msg, debug_msg=debug_msg))
AssertionError: False is not true : Tensors failed to compare as equal!With rtol=1.3e-06 and atol=1e-05, found 368 element(s) (out of 400) whose difference(s) exceeded the margin of error (including 0 nan comparisons). The greatest difference was 4.5334222310064783e+18 (1.9716532230377197 vs. 4.5334222310064783e+18), which occurred at index (7, 1).

----------------------------------------------------------------------
Ran 72 tests in 34.566s

FAILED (failures=1, skipped=11)
Traceback (most recent call last):
  File "run_test.py", line 910, in <module>
    main()
  File "run_test.py", line 889, in main
    raise RuntimeError(err_message)

See CircleCI build pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test1 (2/2)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jan 14 04:08:29 sccache: error: couldn't connect to server 
Jan 14 04:08:29 +++ eval 'extract_trap_cmd '
Jan 14 04:08:29 ++++ extract_trap_cmd
Jan 14 04:08:29 ++++ printf '%s\n' ''
Jan 14 04:08:29 +++ printf '%s\n' cleanup
Jan 14 04:08:29 ++ trap -- '
Jan 14 04:08:29 cleanup' EXIT
Jan 14 04:08:29 ++ [[ pytorch-linux-xenial-cuda10.2-cudnn7-py3-gcc7-test1 != *pytorch-win-* ]]
Jan 14 04:08:29 ++ which sccache
Jan 14 04:08:29 ++ sccache --stop-server
Jan 14 04:08:29 Stopping sccache server...
Jan 14 04:08:29 sccache: error: couldn't connect to server
Jan 14 04:08:29 sccache: caused by: Connection refused (os error 111)
Jan 14 04:08:29 ++ true
Jan 14 04:08:29 ++ rm /var/lib/jenkins/sccache_error.log
Jan 14 04:08:29 ++ [[ pytorch-linux-xenial-cuda10.2-cudnn7-py3-gcc7-test1 == *rocm* ]]
Jan 14 04:08:29 ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
Jan 14 04:08:29 ++ SCCACHE_IDLE_TIMEOUT=1200
Jan 14 04:08:29 ++ RUST_LOG=sccache::server=error
Jan 14 04:08:29 ++ sccache --start-server
Jan 14 04:08:29 sccache: Starting the server...
Jan 14 04:08:29 ++ sccache --zero-stats

1 job timed out:

  • pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test1

❄️ 5 failures tentatively classified as flaky

but reruns have not yet been triggered to confirm:

See CircleCI build pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test2 (1/5)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun) ❄️

Jan 14 06:23:27 unknown file: Failure 
Jan 14 06:23:27 [       OK ] LoopNest.ScheduleSplitInlineSimplify (0 ms)
Jan 14 06:23:27 [ RUN      ] LoopNest.ScheduleInlineThreeMixedOnce
Jan 14 06:23:27 [       OK ] LoopNest.ScheduleInlineThreeMixedOnce (0 ms)
Jan 14 06:23:27 [ RUN      ] LoopNest.ScheduleInlineThreeMixedTwice
Jan 14 06:23:27 [       OK ] LoopNest.ScheduleInlineThreeMixedTwice (0 ms)
Jan 14 06:23:27 [ RUN      ] LoopNest.ScheduleInlineThreeMixedInner
Jan 14 06:23:27 [       OK ] LoopNest.ScheduleInlineThreeMixedInner (1 ms)
Jan 14 06:23:27 [ RUN      ] LoopNest.ScheduleInlineThreeMixedSplit
Jan 14 06:23:27 [       OK ] LoopNest.ScheduleInlineThreeMixedSplit (0 ms)
Jan 14 06:23:27 [ RUN      ] LoopNest.ScheduleInlineOutputTensors
Jan 14 06:23:27 unknown file: Failure
Jan 14 06:23:27 C++ exception with description "Expected to find "      y[m2, n2, k2] = (n2 * m2) * k2 + m2;" but did not find it
Jan 14 06:23:27 Searched string:
Jan 14 06:23:27   for (int m2 = 0; m2 < 4; m2++) {
Jan 14 06:23:27     for (int n2 = 0; n2 < 5; n2++) {
Jan 14 06:23:27       for (int k2 = 0; k2 < 6; k2++) {
Jan 14 06:23:27         y[m2, n2, k2] = (x(m2, n2, k2)) + m2;
Jan 14 06:23:27       }
Jan 14 06:23:27     }
Jan 14 06:23:27   }
Jan 14 06:23:27 }

See CircleCI build pytorch_linux_bionic_py3_8_gcc9_coverage_test2 (2/5)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun) ❄️

Jan 14 05:12:33 unknown file: Failure 
Jan 14 05:12:33 [       OK ] LoopNest.ScheduleSplitInlineSimplify (0 ms)
Jan 14 05:12:33 [ RUN      ] LoopNest.ScheduleInlineThreeMixedOnce
Jan 14 05:12:33 [       OK ] LoopNest.ScheduleInlineThreeMixedOnce (1 ms)
Jan 14 05:12:33 [ RUN      ] LoopNest.ScheduleInlineThreeMixedTwice
Jan 14 05:12:33 [       OK ] LoopNest.ScheduleInlineThreeMixedTwice (0 ms)
Jan 14 05:12:33 [ RUN      ] LoopNest.ScheduleInlineThreeMixedInner
Jan 14 05:12:33 [       OK ] LoopNest.ScheduleInlineThreeMixedInner (1 ms)
Jan 14 05:12:33 [ RUN      ] LoopNest.ScheduleInlineThreeMixedSplit
Jan 14 05:12:33 [       OK ] LoopNest.ScheduleInlineThreeMixedSplit (0 ms)
Jan 14 05:12:33 [ RUN      ] LoopNest.ScheduleInlineOutputTensors
Jan 14 05:12:33 unknown file: Failure
Jan 14 05:12:33 C++ exception with description "Expected to find "      y[m2, n2, k2] = (n2 * m2) * k2 + m2;" but did not find it
Jan 14 05:12:33 Searched string:
Jan 14 05:12:33   for (int m2 = 0; m2 < 4; m2++) {
Jan 14 05:12:33     for (int n2 = 0; n2 < 5; n2++) {
Jan 14 05:12:33       for (int k2 = 0; k2 < 6; k2++) {
Jan 14 05:12:33         y[m2, n2, k2] = (x(m2, n2, k2)) + m2;
Jan 14 05:12:33       }
Jan 14 05:12:33     }
Jan 14 05:12:33   }
Jan 14 05:12:33 }

See CircleCI build pytorch_linux_bionic_py3_6_clang9_test (3/5)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun) ❄️

Jan 14 05:19:02 unknown file: Failure 
Jan 14 05:19:02 [       OK ] LoopNest.ScheduleSplitInlineSimplify (0 ms)
Jan 14 05:19:02 [ RUN      ] LoopNest.ScheduleInlineThreeMixedOnce
Jan 14 05:19:02 [       OK ] LoopNest.ScheduleInlineThreeMixedOnce (0 ms)
Jan 14 05:19:02 [ RUN      ] LoopNest.ScheduleInlineThreeMixedTwice
Jan 14 05:19:02 [       OK ] LoopNest.ScheduleInlineThreeMixedTwice (0 ms)
Jan 14 05:19:02 [ RUN      ] LoopNest.ScheduleInlineThreeMixedInner
Jan 14 05:19:02 [       OK ] LoopNest.ScheduleInlineThreeMixedInner (1 ms)
Jan 14 05:19:02 [ RUN      ] LoopNest.ScheduleInlineThreeMixedSplit
Jan 14 05:19:02 [       OK ] LoopNest.ScheduleInlineThreeMixedSplit (0 ms)
Jan 14 05:19:02 [ RUN      ] LoopNest.ScheduleInlineOutputTensors
Jan 14 05:19:02 unknown file: Failure
Jan 14 05:19:02 C++ exception with description "Expected to find "      y[m2, n2, k2] = (n2 * m2) * k2 + m2;" but did not find it
Jan 14 05:19:02 Searched string:
Jan 14 05:19:02   for (int m2 = 0; m2 < 4; m2++) {
Jan 14 05:19:02     for (int n2 = 0; n2 < 5; n2++) {
Jan 14 05:19:02       for (int k2 = 0; k2 < 6; k2++) {
Jan 14 05:19:02         y[m2, n2, k2] = (x(m2, n2, k2)) + m2;
Jan 14 05:19:02       }
Jan 14 05:19:02     }
Jan 14 05:19:02   }
Jan 14 05:19:02 }

See CircleCI build pytorch_linux_xenial_py3_clang5_asan_test2 (4/5)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun) ❄️

Jan 14 05:44:23 unknown file: Failure 
Jan 14 05:44:23 [       OK ] LoopNest.ScheduleSplitInlineSimplify (0 ms)
Jan 14 05:44:23 [ RUN      ] LoopNest.ScheduleInlineThreeMixedOnce
Jan 14 05:44:23 [       OK ] LoopNest.ScheduleInlineThreeMixedOnce (3 ms)
Jan 14 05:44:23 [ RUN      ] LoopNest.ScheduleInlineThreeMixedTwice
Jan 14 05:44:23 [       OK ] LoopNest.ScheduleInlineThreeMixedTwice (2 ms)
Jan 14 05:44:23 [ RUN      ] LoopNest.ScheduleInlineThreeMixedInner
Jan 14 05:44:23 [       OK ] LoopNest.ScheduleInlineThreeMixedInner (2 ms)
Jan 14 05:44:23 [ RUN      ] LoopNest.ScheduleInlineThreeMixedSplit
Jan 14 05:44:23 [       OK ] LoopNest.ScheduleInlineThreeMixedSplit (1 ms)
Jan 14 05:44:23 [ RUN      ] LoopNest.ScheduleInlineOutputTensors
Jan 14 05:44:23 unknown file: Failure
Jan 14 05:44:23 C++ exception with description "Expected to find "      y[m2, n2, k2] = (n2 * m2) * k2 + m2;" but did not find it
Jan 14 05:44:23 Searched string:
Jan 14 05:44:23   for (int m2 = 0; m2 < 4; m2++) {
Jan 14 05:44:23     for (int n2 = 0; n2 < 5; n2++) {
Jan 14 05:44:23       for (int k2 = 0; k2 < 6; k2++) {
Jan 14 05:44:23         y[m2, n2, k2] = (x(m2, n2, k2)) + m2;
Jan 14 05:44:23       }
Jan 14 05:44:23     }
Jan 14 05:44:23   }
Jan 14 05:44:23 }

See CircleCI build pytorch_linux_xenial_py3_6_gcc5_4_test (5/5)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun) ❄️

Jan 14 05:24:09 unknown file: Failure 
Jan 14 05:24:09 [       OK ] LoopNest.ScheduleSplitInlineSimplify (1 ms)
Jan 14 05:24:09 [ RUN      ] LoopNest.ScheduleInlineThreeMixedOnce
Jan 14 05:24:09 [       OK ] LoopNest.ScheduleInlineThreeMixedOnce (0 ms)
Jan 14 05:24:09 [ RUN      ] LoopNest.ScheduleInlineThreeMixedTwice
Jan 14 05:24:09 [       OK ] LoopNest.ScheduleInlineThreeMixedTwice (0 ms)
Jan 14 05:24:09 [ RUN      ] LoopNest.ScheduleInlineThreeMixedInner
Jan 14 05:24:09 [       OK ] LoopNest.ScheduleInlineThreeMixedInner (1 ms)
Jan 14 05:24:09 [ RUN      ] LoopNest.ScheduleInlineThreeMixedSplit
Jan 14 05:24:09 [       OK ] LoopNest.ScheduleInlineThreeMixedSplit (0 ms)
Jan 14 05:24:09 [ RUN      ] LoopNest.ScheduleInlineOutputTensors
Jan 14 05:24:09 unknown file: Failure
Jan 14 05:24:09 C++ exception with description "Expected to find "      y[m2, n2, k2] = (n2 * m2) * k2 + m2;" but did not find it
Jan 14 05:24:09 Searched string:
Jan 14 05:24:09   for (int m2 = 0; m2 < 4; m2++) {
Jan 14 05:24:09     for (int n2 = 0; n2 < 5; n2++) {
Jan 14 05:24:09       for (int k2 = 0; k2 < 6; k2++) {
Jan 14 05:24:09         y[m2, n2, k2] = (x(m2, n2, k2)) + m2;
Jan 14 05:24:09       }
Jan 14 05:24:09     }
Jan 14 05:24:09   }
Jan 14 05:24:09 }

Extra GitHub checks: 1 failed

ci.pytorch.org: 1 failed

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

@facebook-github-bot facebook-github-bot added the oncall: jit Add this issue/PR to JIT oncall triage queue label Jan 14, 2021
@facebook-github-bot facebook-github-bot deleted the gh/ZolotukhinM/393/head branch February 28, 2021 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

cla signed oncall: jit Add this issue/PR to JIT oncall triage queue

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ZolotukhinM @facebook-github-bot