[primTorch] Adds random operations by mruberry · Pull Request #78026 · pytorch/pytorch

mruberry · 2022-05-20T23:31:52Z

This PR...

Issues Found

Testing

disables stride consistency checks in test_ops and test_meta pending resolution of RFC: [primTorch] Stride-agnostic Operator Semantics #78050
skips chalf in reference tests (addressing [primTorch] many primTorch test xfails are due to chalf #78054)
splits test test_python_reference_consistency in one test for the ctx where torch.foo is torch.foo, and another for when torch.foo is refs.foo
updates test names to be more natural and consistent:
- test_python_reference_errors -> test_python_ref_errors
- test_python_reference_consistency -> test_python_ref and test_python_ref_torch_fallback
- test_python_reference_meta_functions -> test_python_ref_meta
- test_reference_testing -> test_numpy_ref
updates test_python_ref and test_python_ref_torch_fallback to check that the reference is more accurate than the torch op if the reference and torch op results are not close, a warning is raised when this occurs (addressing PrimTorch's test_ops.py reference_consistency testing is worse than test_decomps.py testing #77687)
adds reference inputs for broadcast_tensors
Updates the "fill_" OpInfo to "fill", adding a NumPy reference and making it an elementwise unary operator
Adds 1D no element sample inputs to the cat OpInfo and updates the NumPy reference to handle them and type promotion correctly
Adds reference inputs for elementwise ternary operations, like clamp
Adds a NumPy reference for clamp
Adds reference inputs to where's OpInfo
Makes softplus an elementwise unary OpInfo
Removes the great majority of Python reference OpInfo skips and xfails due to the above test changes
Adds Python reference OpInfos for fill, dropout, clamp, broadcast_tensors, and where

Prims

adds the fill, empty_strided, and uniform prims
removes the empty, empty_like, full, and full_like prims -- these are now references that use empty_strided and fill
renames the "concatenate" and "select" prims to "cat" and "where", respectively, to be consistent with PyTorch
extends the _elementwise_meta operation to accepts tensors that don't participate in type promotion, like the cond tensor in where
fixes a bug in the stride propagation of broadcast_in_dim
moves some error checks from prims.cat to prims.where to refs.cat and refs.where, respectively, consistent with our new policy of doing as much error checking in the ref as possible

Utils

adds the canoicalize_device, extract_shape, and extract_shape_from_varargs helpers
adds the elementwise_unary_scalar_wrapper -- this allows elementwise unary operators to take and return scalar values (ex. refs.sin(1) will return .84...)

Refs

adds the fill, broadcast_tensors, clamp, empty_strided, ones, zeros, and uniform references
adds the nn.functional.dropout reference
fixes refs.cat to handle 1D tensors with no inputs consistent with eager mode

facebook-github-bot · 2022-05-20T23:31:57Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/78026
📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓Need help or want to give feedback on the CI? Visit our office hours
↩️ [fb-only] Re-run with SSH instructions

❌ 1 New Failures

As of commit 0516660 (more details on the Dr. CI page):

Expand to see more

1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages

pull / pytorch-xla-linux-bionic-py3.7-clang8 / test (xla, 1, 1, linux.2xlarge) (1/1)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-05-22T23:52:26.5593369Z RuntimeError: tens...OK() (UNKNOWN: Could not start gRPC server vs. OK)

2022-05-22T23:52:26.5585933Z   File "/opt/conda/lib/python3.7/site-packages/torch_xla-1.12-py3.7-linux-x86_64.egg/torch_xla/distributed/xla_multiprocessing.py", line 315, in _setup_replication
2022-05-22T23:52:26.5586600Z     device = xm.xla_device()
2022-05-22T23:52:26.5587370Z   File "/opt/conda/lib/python3.7/site-packages/torch_xla-1.12-py3.7-linux-x86_64.egg/torch_xla/core/xla_model.py", line 232, in xla_device
2022-05-22T23:52:26.5588004Z     devkind=devkind if devkind is not None else None)
2022-05-22T23:52:26.5588868Z   File "/opt/conda/lib/python3.7/site-packages/torch_xla-1.12-py3.7-linux-x86_64.egg/torch_xla/core/xla_model.py", line 137, in get_xla_supported_devices
2022-05-22T23:52:26.5589461Z     xla_devices = _DEVICES.value
2022-05-22T23:52:26.5590236Z   File "/opt/conda/lib/python3.7/site-packages/torch_xla-1.12-py3.7-linux-x86_64.egg/torch_xla/utils/utils.py", line 32, in value
2022-05-22T23:52:26.5590835Z     self._value = self._gen_fn()
2022-05-22T23:52:26.5591815Z   File "/opt/conda/lib/python3.7/site-packages/torch_xla-1.12-py3.7-linux-x86_64.egg/torch_xla/core/xla_model.py", line 19, in <lambda>
2022-05-22T23:52:26.5592504Z     _DEVICES = xu.LazyProperty(lambda: torch_xla._XLAC._xla_get_devices())
2022-05-22T23:52:26.5593369Z RuntimeError: tensorflow/compiler/xla/xla_client/xrt_local_service.cc:56 : Check failed: tensorflow::NewServer(server_def, &server_) == ::tensorflow::Status::OK() (UNKNOWN: Could not start gRPC server vs. OK)
2022-05-22T23:52:26.8021092Z Traceback (most recent call last):
2022-05-22T23:52:26.8021873Z   File "/var/lib/jenkins/workspace/xla/test/test_mp_save.py", line 63, in <module>
2022-05-22T23:52:26.8022218Z     xmp.spawn(_mp_fn, args=(temp_file,))
2022-05-22T23:52:26.8022914Z   File "/opt/conda/lib/python3.7/site-packages/torch_xla-1.12-py3.7-linux-x86_64.egg/torch_xla/distributed/xla_multiprocessing.py", line 395, in spawn
2022-05-22T23:52:26.8023252Z     start_method=start_method)
2022-05-22T23:52:26.8023642Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
2022-05-22T23:52:26.8024237Z     while not context.join():
2022-05-22T23:52:26.8024590Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 154, in join
2022-05-22T23:52:26.8024864Z     exit_code=exitcode
2022-05-22T23:52:26.8025186Z torch.multiprocessing.spawn.ProcessExitedException: process 1 terminated with exit code 17

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

ngimel · 2022-05-22T03:19:55Z

test/test_ops.py

+
+            msg = f"Reference result was farther ({ref_distance}) from the precise \
+                    computation than the torch result was ({torch_distance})!"
+            self.assertTrue(ref_distance <= torch_distance, msg=msg)


I don't think ref_distance is always weakly less than torch_distance, it's ok to be larger with some tolerance?

I was also thinking about this but wasn't happy with any immediate ideas I had -- cool if I add a TODO comment?

ngimel · 2022-05-22T03:21:04Z

test/test_ops.py

+        # Reports numerical accuracy discrepancies
+        if ex is not None:
+            msg = "Test passed because the reference was more accurate than the torch operator."
+            print(msg)


should this be a warning? Pytest hides stdout of passing tests, so it will be hard to see for people using pytest

Sure -- warning it is

ngimel · 2022-05-22T03:39:32Z

torch/_prims/__init__.py

    scalar_tensor = None
    number = None
-    for arg in args:
+    for arg in args_:


this would potentially set tensor to something with wrong dtype (from args_with_different_types)

Great catch - fixed

ngimel · 2022-05-22T03:39:51Z

torch/_prims/__init__.py

-    *args, type_promotion: ELEMENTWISE_PRIM_TYPE_PROMOTION_KIND
+    *args,
+    type_promotion: ELEMENTWISE_PRIM_TYPE_PROMOTION_KIND,
+    args_with_different_dtypes: Tuple[TensorLikeType, ...] = None,


args_with_fixed_types?

Yeah that's way better -- fixed

ngimel · 2022-05-22T03:48:10Z

torch/_prims/__init__.py



-def _select_aten(pred: Tensor, a: Tensor, b: Tensor) -> Tensor:
+def _where_aten(pred: Tensor, a: Tensor, b: Tensor) -> Tensor:


out of curiosity, why do we need this helper, and not just use torch.where in make_prim?

Great point -- I was just on automatic mode -- fixed!

ngimel · 2022-05-22T03:49:54Z

torch/_prims/__init__.py


-def _empty_like_aten(
-    a: Tensor, *, dtype: torch.dtype, device: torch.device, requires_grad: bool
+def _empty_strided_aten(


same question here?

Yep -- fixed

ngimel

Great

ngimel · 2022-05-22T04:22:20Z

torch/testing/_internal/common_methods_invocations.py

+    else:
+        value = 3
+
+    return ({'value': value}, {'value': value})


why 2 tuple elements?

This is super weird and we could sugar over this, but it's because we sometimes pass different arguments to the NumPy op, so we have "torch kwargs" and "NumPy kwargs" here

mruberry · 2022-05-22T10:01:57Z

@pytorchbot merge this please

github-actions · 2022-05-22T10:07:12Z

Hey @mruberry.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

suo · 2022-05-22T18:08:03Z

@pytorchbot revert -m "This broke trunk: https://hud.pytorch.org/pytorch/pytorch/commit/043cf1f9c746b4dda2c404ba6c76c6ccad5e2cbe" -c landrace

suo · 2022-05-22T18:09:30Z

Actually this looks like the proper classification is "nosignal"--only slow tests broke.

This reverts commit 043cf1f. Reverted #78026 on behalf of https://github.com/suo due to This broke trunk: https://hud.pytorch.org/pytorch/pytorch/commit/043cf1f9c746b4dda2c404ba6c76c6ccad5e2cbe

mruberry · 2022-05-23T01:51:53Z

@pytorchbot merge this please

github-actions · 2022-05-23T01:57:03Z

Hey @mruberry.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

ezyang · 2022-05-23T14:25:17Z

torch/_refs/special/__init__.py

-    self = torch.clamp(self, lo, hi)
-    return (self / (1 - self)).log()
+    self = refs.clamp(self, lo, hi)
+    return refs.log(refs.true_divide(self, refs.sub(1, self)))


@mruberry Given that the context manager exists now, we should prefer using the torch API calls as this ensures that the decomposition in question is using the limited API supported by torch and not the expanded API from refs. Is this just to work around the local problem that ref consistency tests don't work? I'd much rather we dupe the tests in that case.

It's that the meta tests weren't working; I did duplicate the consistency tests

Summary: This PR... **Issues Found** - #78058 - #78054 - #78053 - #78050 - #77932 **Testing** - disables stride consistency checks in test_ops and test_meta pending resolution of #78050 - skips chalf in reference tests (addressing #78054) - splits test test_python_reference_consistency in one test for the ctx where torch.foo is torch.foo, and another for when torch.foo is refs.foo - updates test names to be more natural and consistent: - test_python_reference_errors -> test_python_ref_errors - test_python_reference_consistency -> test_python_ref and test_python_ref_torch_fallback - test_python_reference_meta_functions -> test_python_ref_meta - test_reference_testing -> test_numpy_ref - updates test_python_ref and test_python_ref_torch_fallback to check that the reference is more accurate than the torch op if the reference and torch op results are not close, a warning is raised when this occurs (addressing #77687) - adds reference inputs for broadcast_tensors - Updates the "fill_" OpInfo to "fill", adding a NumPy reference and making it an elementwise unary operator - Adds 1D no element sample inputs to the cat OpInfo and updates the NumPy reference to handle them and type promotion correctly - Adds reference inputs for elementwise ternary operations, like clamp - Adds a NumPy reference for clamp - Adds reference inputs to where's OpInfo - Makes softplus an elementwise unary OpInfo - Removes the great majority of Python reference OpInfo skips and xfails due to the above test changes - Adds Python reference OpInfos for fill, dropout, clamp, broadcast_tensors, and where **Prims** - adds the fill, empty_strided, and uniform prims - removes the empty, empty_like, full, and full_like prims -- these are now references that use empty_strided and fill - renames the "concatenate" and "select" prims to "cat" and "where", respectively, to be consistent with PyTorch - extends the `_elementwise_meta` operation to accepts tensors that don't participate in type promotion, like the `cond` tensor in `where` - fixes a bug in the stride propagation of broadcast_in_dim - moves some error checks from prims.cat to prims.where to refs.cat and refs.where, respectively, consistent with our new policy of doing as much error checking in the ref as possible **Utils** - adds the canoicalize_device, extract_shape, and extract_shape_from_varargs helpers - adds the elementwise_unary_scalar_wrapper -- this allows elementwise unary operators to take and return scalar values (ex. refs.sin(1) will return .84...) **Refs** - adds the fill, broadcast_tensors, clamp, empty_strided, ones, zeros, and uniform references - adds the nn.functional.dropout reference - fixes refs.cat to handle 1D tensors with no inputs consistent with eager mode Pull Request resolved: #78026 Approved by: https://github.com/ngimel Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/d4345ed0a6c06b1e489e41c219f94d26d3014ce6 Reviewed By: seemethere Differential Revision: D36610393 Pulled By: mruberry fbshipit-source-id: 415e532ab647ab8425f9064796704f6c44115f0e

This PR... **Issues Found** - #78058 - #78054 - #78053 - #78050 - #77932 **Testing** - disables stride consistency checks in test_ops and test_meta pending resolution of #78050 - skips chalf in reference tests (addressing #78054) - splits test test_python_reference_consistency in one test for the ctx where torch.foo is torch.foo, and another for when torch.foo is refs.foo - updates test names to be more natural and consistent: - test_python_reference_errors -> test_python_ref_errors - test_python_reference_consistency -> test_python_ref and test_python_ref_torch_fallback - test_python_reference_meta_functions -> test_python_ref_meta - test_reference_testing -> test_numpy_ref - updates test_python_ref and test_python_ref_torch_fallback to check that the reference is more accurate than the torch op if the reference and torch op results are not close, a warning is raised when this occurs (addressing #77687) - adds reference inputs for broadcast_tensors - Updates the "fill_" OpInfo to "fill", adding a NumPy reference and making it an elementwise unary operator - Adds 1D no element sample inputs to the cat OpInfo and updates the NumPy reference to handle them and type promotion correctly - Adds reference inputs for elementwise ternary operations, like clamp - Adds a NumPy reference for clamp - Adds reference inputs to where's OpInfo - Makes softplus an elementwise unary OpInfo - Removes the great majority of Python reference OpInfo skips and xfails due to the above test changes - Adds Python reference OpInfos for fill, dropout, clamp, broadcast_tensors, and where **Prims** - adds the fill, empty_strided, and uniform prims - removes the empty, empty_like, full, and full_like prims -- these are now references that use empty_strided and fill - renames the "concatenate" and "select" prims to "cat" and "where", respectively, to be consistent with PyTorch - extends the `_elementwise_meta` operation to accepts tensors that don't participate in type promotion, like the `cond` tensor in `where` - fixes a bug in the stride propagation of broadcast_in_dim - moves some error checks from prims.cat to prims.where to refs.cat and refs.where, respectively, consistent with our new policy of doing as much error checking in the ref as possible **Utils** - adds the canoicalize_device, extract_shape, and extract_shape_from_varargs helpers - adds the elementwise_unary_scalar_wrapper -- this allows elementwise unary operators to take and return scalar values (ex. refs.sin(1) will return .84...) **Refs** - adds the fill, broadcast_tensors, clamp, empty_strided, ones, zeros, and uniform references - adds the nn.functional.dropout reference - fixes refs.cat to handle 1D tensors with no inputs consistent with eager mode Pull Request resolved: #78026 Approved by: https://github.com/ngimel

This reverts commit 043cf1f. Reverted #78026 on behalf of https://github.com/suo due to This broke trunk: https://hud.pytorch.org/pytorch/pytorch/commit/043cf1f9c746b4dda2c404ba6c76c6ccad5e2cbe

This PR... **Issues Found** - #78058 - #78054 - #78053 - #78050 - #77932 **Testing** - disables stride consistency checks in test_ops and test_meta pending resolution of #78050 - skips chalf in reference tests (addressing #78054) - splits test test_python_reference_consistency in one test for the ctx where torch.foo is torch.foo, and another for when torch.foo is refs.foo - updates test names to be more natural and consistent: - test_python_reference_errors -> test_python_ref_errors - test_python_reference_consistency -> test_python_ref and test_python_ref_torch_fallback - test_python_reference_meta_functions -> test_python_ref_meta - test_reference_testing -> test_numpy_ref - updates test_python_ref and test_python_ref_torch_fallback to check that the reference is more accurate than the torch op if the reference and torch op results are not close, a warning is raised when this occurs (addressing #77687) - adds reference inputs for broadcast_tensors - Updates the "fill_" OpInfo to "fill", adding a NumPy reference and making it an elementwise unary operator - Adds 1D no element sample inputs to the cat OpInfo and updates the NumPy reference to handle them and type promotion correctly - Adds reference inputs for elementwise ternary operations, like clamp - Adds a NumPy reference for clamp - Adds reference inputs to where's OpInfo - Makes softplus an elementwise unary OpInfo - Removes the great majority of Python reference OpInfo skips and xfails due to the above test changes - Adds Python reference OpInfos for fill, dropout, clamp, broadcast_tensors, and where **Prims** - adds the fill, empty_strided, and uniform prims - removes the empty, empty_like, full, and full_like prims -- these are now references that use empty_strided and fill - renames the "concatenate" and "select" prims to "cat" and "where", respectively, to be consistent with PyTorch - extends the `_elementwise_meta` operation to accepts tensors that don't participate in type promotion, like the `cond` tensor in `where` - fixes a bug in the stride propagation of broadcast_in_dim - moves some error checks from prims.cat to prims.where to refs.cat and refs.where, respectively, consistent with our new policy of doing as much error checking in the ref as possible **Utils** - adds the canoicalize_device, extract_shape, and extract_shape_from_varargs helpers - adds the elementwise_unary_scalar_wrapper -- this allows elementwise unary operators to take and return scalar values (ex. refs.sin(1) will return .84...) **Refs** - adds the fill, broadcast_tensors, clamp, empty_strided, ones, zeros, and uniform references - adds the nn.functional.dropout reference - fixes refs.cat to handle 1D tensors with no inputs consistent with eager mode Pull Request resolved: #78026 Approved by: https://github.com/ngimel

Ref: #69991 Probably started working since : #78026 Pull Request resolved: #80277 Approved by: https://github.com/zou3519

Summary: Ref: #69991 Probably started working since : #78026 Pull Request resolved: #80277 Approved by: https://github.com/zou3519 Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/1b18c2e93cb5ae96314247e97f3040fda36b6356 Reviewed By: b0noI Differential Revision: D37495906 fbshipit-source-id: 25dfcb5f8bbe61e5ff2da1c59810a6ebed1850c3

stashes

5381974

mruberry requested a review from ngimel as a code owner May 20, 2022 23:31

facebook-github-bot added the cla signed label May 20, 2022

Mike Ruberry added 3 commits May 20, 2022 20:48

merges

2fd32bb

updates per review

fc424f6

precision dtype update

1bc5a90

ngimel reviewed May 22, 2022

View reviewed changes

merges

a428d56

ngimel approved these changes May 22, 2022

View reviewed changes

Mike Ruberry added 2 commits May 21, 2022 21:24

stashes

a5123f8

merges

3795780

mruberry mentioned this pull request May 22, 2022

[primTorch] refs: margin_ranking_loss, hinge_embedding_loss #78057

Closed

Mike Ruberry added 4 commits May 21, 2022 21:38

updates per review

0141e79

lint

1b8905b

lint

daffdab

backcompat

805a25c

mruberry mentioned this pull request May 22, 2022

PrimTorch's test_ops.py reference_consistency testing is worse than test_decomps.py testing #77687

Closed

Mike Ruberry added 3 commits May 21, 2022 23:16

test fix

ee2a361

test fix

0cae349

typo

f1c91e6

pytorchmergebot added the Merged label May 22, 2022

pytorchmergebot closed this in 043cf1f May 22, 2022

pytorchmergebot added the Reverted label May 22, 2022

mruberry reopened this May 22, 2022

mruberry added the ciflow/all label May 22, 2022

test fix

0516660

pytorchmergebot closed this in d4345ed May 23, 2022

This was referenced May 23, 2022

[primTorch] many primTorch test xfails are due to chalf #78054

Closed

PrimTorch decomps for random functions #77659

Closed

ezyang reviewed May 23, 2022

View reviewed changes

mruberry mentioned this pull request May 23, 2022

addr ref #78014

Closed

kshitij12345 mentioned this pull request Jun 26, 2022

[composite compliance] fill #80277

Closed

pytorchmergebot pushed a commit that referenced this pull request Jun 27, 2022

[composite compliance] fill (#80277)

1b18c2e

Ref: #69991 Probably started working since : #78026 Pull Request resolved: #80277 Approved by: https://github.com/zou3519

github-actions bot deleted the primtorch_random branch February 16, 2024 01:56



		def _select_aten(pred: Tensor, a: Tensor, b: Tensor) -> Tensor:
		def _where_aten(pred: Tensor, a: Tensor, b: Tensor) -> Tensor:

Conversation

mruberry commented May 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented May 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

❌ 1 New Failures

🕵️ 1 new failure recognized by patterns

pull / pytorch-xla-linux-bionic-py3.7-clang8 / test (xla, 1, 1, linux.2xlarge) (1/1)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngimel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mruberry commented May 22, 2022

Uh oh!

github-actions bot commented May 22, 2022

Uh oh!

suo commented May 22, 2022

Uh oh!

suo commented May 22, 2022

Uh oh!

mruberry commented May 23, 2022

Uh oh!

github-actions bot commented May 23, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mruberry commented May 20, 2022 •

edited

Loading

facebook-github-bot commented May 20, 2022 •

edited

Loading