[WIP] Support broadcasting via slow path by izdeby · Pull Request #52448 · pytorch/pytorch

izdeby · 2021-02-18T19:33:52Z

Stack from ghstack:

wip #52450 wip
[wip] Replace optimizers in torch.optim with the ones from torch.optim._multi_tensor #49039 [wip] Replace optimizers in torch.optim with the ones from torch.optim._multi_tensor
[WIP] Add type promotion to a fast path for foreach APIs #52449 [WIP] Add type promotion to a fast path for foreach APIs
[WIP] Support broadcasting via slow path #52448 [WIP] Support broadcasting via slow path
[wip] Pushed unsupported scenarios to slow path #48224 [wip] Pushed unsupported scenarios to slow path
Updated foreach pointwise ops to support complex and bloat16 on CUDA #51170 Updated foreach pointwise ops to support complex and bloat16 on CUDA
Update foreach max/min #49714 Update foreach max/min
Refactor foreach binary ops tests with tensor lists to use OpInfo #51059 Refactor foreach binary ops tests with tensor lists to use OpInfo
Refactor foreach binary ops tests with scalars to use OpInfo #51058 Refactor foreach binary ops tests with scalars to use OpInfo

Differential Revision: D26520922

[ghstack-poisoned]

facebook-github-bot · 2021-02-18T19:34:02Z

💊 CI failures summary and remediations

As of commit d370e27 (more details on the Dr. CI page):

11/11 failures possibly* introduced in this PR
- 2/11 non-scanned failure(s)

🕵️ 7 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_macos_10_13_py3_test (1/7)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

Apr 01 19:35:45 RuntimeError: test_foreach failed!

Apr 01 19:35:43 Executing ['/Users/distiller/workspace/miniconda3/bin/python', 'test_foreach.py', '-v'] ... [2021-04-01 19:35:43.765230]
Apr 01 19:35:45 Traceback (most recent call last):
Apr 01 19:35:45   File "test_foreach.py", line 8, in <module>
Apr 01 19:35:45     from torch.testing._internal.common_methods_invocations import \
Apr 01 19:35:45 ImportError: cannot import name 'foreach_binary_op_tensor_list_db' from 'torch.testing._internal.common_methods_invocations' (/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/common_methods_invocations.py)
Apr 01 19:35:45 Traceback (most recent call last):
Apr 01 19:35:45   File "test/run_test.py", line 1094, in <module>
Apr 01 19:35:45     main()
Apr 01 19:35:45   File "test/run_test.py", line 1073, in main
Apr 01 19:35:45     raise RuntimeError(err_message)
Apr 01 19:35:45 RuntimeError: test_foreach failed!
Apr 01 19:35:45 + cleanup
Apr 01 19:35:45 + retcode=1
Apr 01 19:35:45 + set +x


Exited with code exit status 1

pytorch_linux_bionic_py3_8_gcc9_coverage_test1 (2/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Apr 01 19:43:26 [E request_callback_no_python.cpp:656] Received error while processing request type 256: The following operation failed in the TorchScript interpreter.

Apr 01 19:43:26 
Apr 01 19:43:26 [E request_callback_no_python.cpp:656] Received error while processing request type 256: The following operation failed in the TorchScript interpreter.
Apr 01 19:43:26 Traceback of TorchScript (most recent call last):
Apr 01 19:43:26   File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 334, in raise_func_script
Apr 01 19:43:26 @torch.jit.script
Apr 01 19:43:26 def raise_func_script(expected_err: str) -> torch.Tensor:
Apr 01 19:43:26     raise ValueError(expected_err)
Apr 01 19:43:26     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
Apr 01 19:43:26 RuntimeError: Expected error
Apr 01 19:43:26 
Apr 01 19:43:26 [E request_callback_no_python.cpp:656] Received error while processing request type 256: The following operation failed in the TorchScript interpreter.
Apr 01 19:43:26 Traceback of TorchScript (most recent call last):
Apr 01 19:43:26   File "/opt/conda/lib/python3.8/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 334, in raise_func_script
Apr 01 19:43:26 @torch.jit.script
Apr 01 19:43:26 def raise_func_script(expected_err: str) -> torch.Tensor:
Apr 01 19:43:26     raise ValueError(expected_err)
Apr 01 19:43:26     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
Apr 01 19:43:26 RuntimeError: Expected error
Apr 01 19:43:26 
Apr 01 19:43:26 ok (2.756s)
Apr 01 19:43:28   test_wait_all_multiple_call (__main__.ProcessGroupRpcTestWithSpawn) ... RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend.

pytorch_linux_xenial_py3_clang5_asan_test2 (3/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Apr 01 20:37:06 RuntimeError: test_foreach failed!

Apr 01 20:37:03 Executing ['/opt/conda/bin/python', 'test_foreach.py', '-v'] ... [2021-04-01 20:37:03.797448]
Apr 01 20:37:06 Traceback (most recent call last):
Apr 01 20:37:06   File "test_foreach.py", line 8, in <module>
Apr 01 20:37:06     from torch.testing._internal.common_methods_invocations import \
Apr 01 20:37:06 ImportError: cannot import name 'foreach_binary_op_tensor_list_db'
Apr 01 20:37:06 Traceback (most recent call last):
Apr 01 20:37:06   File "test/run_test.py", line 1094, in <module>
Apr 01 20:37:06     main()
Apr 01 20:37:06   File "test/run_test.py", line 1073, in main
Apr 01 20:37:06     raise RuntimeError(err_message)
Apr 01 20:37:06 RuntimeError: test_foreach failed!
Apr 01 20:37:07 =================== sccache compilation log ===================
Apr 01 20:37:07 + cleanup
Apr 01 20:37:07 + retcode=1
Apr 01 20:37:07 + set +x
Apr 01 20:37:07 =========== If your build fails, please take a look at the log above for possible reasons ===========
Apr 01 20:37:07 Compile requests                      4
Apr 01 20:37:07 Compile requests executed             2
Apr 01 20:37:07 Cache hits                            2
Apr 01 20:37:07 Cache hits (C/C++)                    2
Apr 01 20:37:07 Cache misses                          0

pytorch_linux_bionic_py3_6_clang9_noarch_test (4/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Apr 01 18:56:44 RuntimeError: test_foreach failed!

Apr 01 18:56:42 Executing ['/opt/conda/bin/python', 'test_foreach.py', '-v'] ... [2021-04-01 18:56:42.763283]
Apr 01 18:56:43 Traceback (most recent call last):
Apr 01 18:56:43   File "test_foreach.py", line 8, in <module>
Apr 01 18:56:43     from torch.testing._internal.common_methods_invocations import \
Apr 01 18:56:43 ImportError: cannot import name 'foreach_binary_op_tensor_list_db'
Apr 01 18:56:44 Traceback (most recent call last):
Apr 01 18:56:44   File "test/run_test.py", line 1094, in <module>
Apr 01 18:56:44     main()
Apr 01 18:56:44   File "test/run_test.py", line 1073, in main
Apr 01 18:56:44     raise RuntimeError(err_message)
Apr 01 18:56:44 RuntimeError: test_foreach failed!
Apr 01 18:56:44 
Apr 01 18:56:44 real	21m47.759s
Apr 01 18:56:44 user	27m43.002s
Apr 01 18:56:44 sys	4m25.188s
Apr 01 18:56:44 + cleanup
Apr 01 18:56:44 + retcode=1
Apr 01 18:56:44 + set +x
Apr 01 18:56:44 =================== sccache compilation log ===================
Apr 01 18:56:44 ERROR 2021-04-01T18:45:30Z: sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function ‘int main()’:\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected ‘;’ before ‘}’ token\n int main() { return 0 }\n                       ^\n" }
Apr 01 18:56:44

pytorch_linux_xenial_py3_6_gcc5_4_test (5/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Apr 01 18:47:54 RuntimeError: test_foreach failed!

Apr 01 18:47:53 Executing ['/opt/conda/bin/python', 'test_foreach.py', '-v'] ... [2021-04-01 18:47:53.142649]
Apr 01 18:47:54 Traceback (most recent call last):
Apr 01 18:47:54   File "test_foreach.py", line 8, in <module>
Apr 01 18:47:54     from torch.testing._internal.common_methods_invocations import \
Apr 01 18:47:54 ImportError: cannot import name 'foreach_binary_op_tensor_list_db'
Apr 01 18:47:54 Traceback (most recent call last):
Apr 01 18:47:54   File "test/run_test.py", line 1094, in <module>
Apr 01 18:47:54     main()
Apr 01 18:47:54   File "test/run_test.py", line 1073, in main
Apr 01 18:47:54     raise RuntimeError(err_message)
Apr 01 18:47:54 RuntimeError: test_foreach failed!
Apr 01 18:47:54 + cleanup
Apr 01 18:47:54 + retcode=1
Apr 01 18:47:54 + set +x
Apr 01 18:47:54 =================== sccache compilation log ===================
Apr 01 18:47:54 ERROR 2021-04-01T18:37:15Z: sccache::server: Compilation failed: Output { status: ExitStatus(ExitStatus(256)), stdout: "", stderr: "/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp: In function ‘int main()’:\n/var/lib/jenkins/.cache/torch_extensions/test_compilation_error_formatting/main.cpp:2:23: error: expected ‘;’ before ‘}’ token\n int main() { return 0 }\n                       ^\n" }
Apr 01 18:47:54 
Apr 01 18:47:54 =========== If your build fails, please take a look at the log above for possible reasons ===========
Apr 01 18:47:54 Compile requests                     83
Apr 01 18:47:54 Compile requests executed            54
Apr 01 18:47:54 Cache hits                           29

pytorch_windows_vs2019_py36_cuda10.1_test1 (6/7)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

AssertionError: RuntimeError not raised

  test_scatter_gpu_neg_dim (__main__.TestCudaComm) ... skip (0.000s)
  test_scatter_gpu_sizes (__main__.TestCudaComm) ... skip (0.000s)
  test_scatter_namedtuple (__main__.TestCudaComm) ... skip (0.003s)

======================================================================
FAIL [0.019s]: test_grad_scaling_unscale (__main__.TestCuda)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_cuda.py", line 1917, in test_grad_scaling_unscale
    inv_scale)
AssertionError: RuntimeError not raised

----------------------------------------------------------------------
Ran 158 tests in 111.507s

FAILED (failures=1, skipped=65)

Generating XML reports...
Generated XML report: test-reports\dist-gloo\test_cuda\TEST-TestCuda-20210401194456.xml
Generated XML report: test-reports\dist-gloo\test_cuda\TEST-TestCudaComm-20210401194456.xml
Traceback (most recent call last):

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test2 (7/7)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Apr 01 20:41:21 AssertionError: RuntimeError not raised

Apr 01 20:41:21 ======================================================================
Apr 01 20:41:21 FAIL [0.031s]: test_grad_scaling_unscale (__main__.TestCuda)
Apr 01 20:41:21 ----------------------------------------------------------------------
Apr 01 20:41:21 Traceback (most recent call last):
Apr 01 20:41:21   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 979, in wrapper
Apr 01 20:41:21     method(*args, **kwargs)
Apr 01 20:41:21   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 979, in wrapper
Apr 01 20:41:21     method(*args, **kwargs)
Apr 01 20:41:21   File "test_cuda.py", line 1917, in test_grad_scaling_unscale
Apr 01 20:41:21     inv_scale)
Apr 01 20:41:21 AssertionError: RuntimeError not raised
Apr 01 20:41:21 
Apr 01 20:41:21 ----------------------------------------------------------------------
Apr 01 20:41:21 Ran 158 tests in 166.602s
Apr 01 20:41:21 
Apr 01 20:41:21 FAILED (failures=1, skipped=12)
Apr 01 20:41:21 
Apr 01 20:41:21 Generating XML reports...
Apr 01 20:41:21 Generated XML report: test-reports/dist-gloo/test_cuda/TEST-TestCuda-20210401203834.xml
Apr 01 20:41:21 Generated XML report: test-reports/dist-gloo/test_cuda/TEST-TestCudaComm-20210401203834.xml
Apr 01 20:41:21 Traceback (most recent call last):

2 failures not recognized by patterns:

Job	Step	Action
^flake8-py3	^{Fail if there were any warnings}	🔁 rerun
^quick-checks	^{Ensure no trailing spaces}	🔁 rerun

ci.pytorch.org: 1 failed

Failed: pr/pytorch-linux-bionic-rocm4.0.1-py3.6

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

facebook-github-bot · 2021-04-09T09:45:48Z

Hi @izdeby!

Thank you for your pull request.

We require contributors to sign our Contributor License Agreement, and yours needs attention.

You currently have a record in our system, but the CLA is no longer valid, and will need to be resubmitted.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

Summary: This is based on #48224. To make `foreach` more flexible, this PR pushes unsupported cases to slow path. Also, this adds some tests to verify that - `foreach` functions work with tensors of different dtypes and/or memory layouts in 7bd4b2c - `foreach` functions work with tensors on different devices in a list, but are on the same device if the indices are the same: def4b9b Future plans: 1. Improve the coverage of unittests using `ops` decorator & updating `foreach_unary_op_db` and creating `foreach_(binary|pointwise|minmax)_db`. 2. Support broadcasting in slow path. Ref: #52448 3. Support type promotion in fast path. Ref #52449 CC: ngimel mcarilli ptrblck Pull Request resolved: #56993 Reviewed By: zou3519 Differential Revision: D28630580 Pulled By: ngimel fbshipit-source-id: e26ee74a39a591025e18c1ead48948cb7ec53c19

Summary: This is based on pytorch#48224. To make `foreach` more flexible, this PR pushes unsupported cases to slow path. Also, this adds some tests to verify that - `foreach` functions work with tensors of different dtypes and/or memory layouts in pytorch@7bd4b2c - `foreach` functions work with tensors on different devices in a list, but are on the same device if the indices are the same: pytorch@def4b9b Future plans: 1. Improve the coverage of unittests using `ops` decorator & updating `foreach_unary_op_db` and creating `foreach_(binary|pointwise|minmax)_db`. 2. Support broadcasting in slow path. Ref: pytorch#52448 3. Support type promotion in fast path. Ref pytorch#52449 CC: ngimel mcarilli ptrblck Pull Request resolved: pytorch#56993 Reviewed By: zou3519 Differential Revision: D28630580 Pulled By: ngimel fbshipit-source-id: e26ee74a39a591025e18c1ead48948cb7ec53c19

pytorchbot · 2022-04-12T05:05:09Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
Stale pull requests will automatically be closed 30 days after being marked Stale

Support broadcasting via slow path

a6e867a

[ghstack-poisoned]

facebook-github-bot added the cla signed label Feb 18, 2021

Iurii Zdebskyi and others added 12 commits February 18, 2021 12:16

Update on "Support broadcasting via slow path"

1665ef1

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "Support broadcasting via slow path"

7e3a300

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "Support broadcasting via slow path"

17da955

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "Support broadcasting via slow path"

16a39f9

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "Support broadcasting via slow path"

a1f19a7

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "Support broadcasting via slow path"

477a2eb

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "Support broadcasting via slow path"

814c7af

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "Support broadcasting via slow path"

575697a

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "Support broadcasting via slow path"

7557d51

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "Support broadcasting via slow path"

89c0c08

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "Support broadcasting via slow path"

d7175e3

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "Support broadcasting via slow path"

5362552

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "Support broadcasting via slow path"

45cbd12

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

izdeby changed the title ~~Support broadcasting via slow path~~ [WIP] Support broadcasting via slow path Mar 16, 2021

izdeby and others added 4 commits March 18, 2021 15:16

Update on "[WIP] Support broadcasting via slow path"

3493ded

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "[WIP] Support broadcasting via slow path"

f170a5f

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "[WIP] Support broadcasting via slow path"

7edd673

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

Update on "[WIP] Support broadcasting via slow path"

d370e27

Differential Revision: [D26520922](https://our.internmc.facebook.com/intern/diff/D26520922) [ghstack-poisoned]

pytorchbot added the open source label Apr 9, 2021

crcrpar mentioned this pull request Apr 27, 2021

[PyTorch] Reduce errors of foreach functions #56993

Closed

pytorchbot added the Stale label Apr 12, 2022

github-actions bot closed this May 12, 2022

facebook-github-bot deleted the gh/izdeby/87/head branch June 11, 2022 14:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Support broadcasting via slow path#52448

[WIP] Support broadcasting via slow path#52448
izdeby wants to merge 18 commits intogh/izdeby/87/basefrom
gh/izdeby/87/head

izdeby commented Feb 18, 2021 •

edited

Loading

Uh oh!

facebook-github-bot commented Feb 18, 2021 •

edited

Loading

Uh oh!

facebook-github-bot commented Apr 9, 2021

Uh oh!

pytorchbot commented Apr 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

izdeby commented Feb 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Feb 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

🕵️ 7 new failures recognized by patterns

pytorch_macos_10_13_py3_test (1/7)

pytorch_linux_bionic_py3_8_gcc9_coverage_test1 (2/7)

pytorch_linux_xenial_py3_clang5_asan_test2 (3/7)

pytorch_linux_bionic_py3_6_clang9_noarch_test (4/7)

pytorch_linux_xenial_py3_6_gcc5_4_test (5/7)

pytorch_windows_vs2019_py36_cuda10.1_test1 (6/7)

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test2 (7/7)

2 failures not recognized by patterns:

ci.pytorch.org: 1 failed

Uh oh!

facebook-github-bot commented Apr 9, 2021

Process

Uh oh!

pytorchbot commented Apr 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

izdeby commented Feb 18, 2021 •

edited

Loading

facebook-github-bot commented Feb 18, 2021 •

edited

Loading