Add memory_format support `to` and `type` operators by VitalyFedyunin · Pull Request #27107 · pytorch/pytorch

VitalyFedyunin · 2019-09-30T22:25:44Z

Stack from ghstack:

Add memory format support to resize_as_ operator #27979 Add memory format support to resize_as_ operator
Cleanup testing of _like operators #27891 Cleanup testing of _like operators
Add memory format support to randn_like operator #27890 Add memory format support to randn_like operator
Add memory format support to randint_like operator #27889 Add memory format support to randint_like operator
Add memory format support to zeros_like operator #27562 Add memory format support to zeros_like operator
Add memory format support to rand_like operator #27561 Add memory format support to rand_like operator
Add memory format support to ones_like operator #27270 Add memory format support to ones_like operator
Add memory format support to full_like operator #27262 Add memory format support to full_like operator
Add memory format support to empty_like operator #27244 Add memory format support to empty_like operator
Add memory format support to typecasting shortcuts byte,char,double,bool,half,int,long,short,float,bfloat16 #27228 Add memory format support to typecasting shortcuts byte,char,double,bool,half,int,long,short,float,bfloat16
Add memory format support to cpu and cuda operators #27223 Add memory format support to cpu and cuda operators
Add memory_format support to and type operators #27107 Add memory_format support to and type operators

Adds memory_format keyword argument (positional for cpp).

'Preserve' behavior now follows next rules:

If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor.
If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format.
Output tensor is going to be contiguous in all other cases.

Dense tensor is the tensor that store values in a contiguous block of memory.
Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory.

Differential Revision: D17931062

[ghstack-poisoned]

ghstack-source-id: c5c9d56 Pull Request resolved: #27107

[ghstack-poisoned]

Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. [ghstack-poisoned]

ezyang · 2019-10-04T15:22:49Z

If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format.

Define what it means to be in 'channels last' format? :)

ezyang · 2019-10-04T15:23:49Z

aten/src/ATen/core/TensorBody.h

-  Tensor to(Device device, ScalarType dtype, bool non_blocking=false, bool copy=false) const;
-  Tensor to(ScalarType dtype, bool non_blocking=false, bool copy=false) const;
-  Tensor to(const Tensor & other, bool non_blocking=false, bool copy=false) const;
+  Tensor to(const TensorOptions & options, bool non_blocking=false, bool copy=false, c10::optional<MemoryFormat> memory_format=c10::nullopt) const;


I told you this before, and I'm going to complain about it again here: MemoryFormat should be a parameter in TensorOptions, not a freestanding positional argument as it is here. The state of positional arguments here is basically unusable for actual users of this API.

Going to be merged into TensorOptions as soon as TensorOptions codegen rewrite finished ( CC @izdeby )

ezyang · 2019-10-04T15:28:14Z

torch/csrc/autograd/utils/python_arg_parsing.h

-    "to(Tensor tensor, bool non_blocking=False, bool copy=False)",
+    "to(Device device=None, ScalarType dtype=None, bool non_blocking=False, bool copy=False, *, MemoryFormat? memory_format=None)",
+    "to(ScalarType dtype, bool non_blocking=False, bool copy=False, *, MemoryFormat? memory_format=None)",
+    "to(Tensor tensor, bool non_blocking=False, bool copy=False, *, MemoryFormat? memory_format=None)",


Ugh, we should have made non_blocking and copy keyword only too XD

Going to be separate PR.

ezyang · 2019-10-04T15:29:05Z

native_functions_test_metadata.yaml

@@ -0,0 +1,678 @@
+- func: __and__


What's this?

Not supposed to be part of PR. Removing.

ezyang · 2019-10-14T15:42:51Z

Please rerequest review when this is ready

Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. [ghstack-poisoned]

VitalyFedyunin · 2019-10-14T18:19:10Z

Define what it means to be in 'channels last' format? :)

pytorch/c10/core/TensorImpl.h

Line 1643 in d70f8dd

// Tensor is stored in the channels last memory format, when dimensions

Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. [ghstack-poisoned]

…ators" Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. [ghstack-poisoned]

…operators" Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. [ghstack-poisoned]

ezyang · 2019-10-15T14:42:46Z

aten/src/ATen/native/TensorConversions.cpp

+  auto memory_format =
+      optional_memory_format.value_or(MemoryFormat::Contiguous);
+
+  if (self.options() == options && !copy &&


Don't compare options. See #25478 and #27572 (btw, bikeshedding on TensorAxes welcome)

Result of in person conversation: Let's add something like "match TensorOptions with Tensor". This tests if a Tensor is "compatible" with a TensorOptions. (This is not an equality test, because options can be left unspecified, and in that case you always match.)

ezyang · 2019-10-15T15:08:29Z

aten/src/ATen/native/TensorConversions.cpp

+    if (self.is_non_overlapping_and_dense()) {
+      // Copy all strides
+      auto r = at::empty_strided(self.sizes(), self.strides(), options);
+      r.copy_(self);


I'm pleasantly surprised to learn copy_ does the right thing in this case :>

ezyang · 2019-10-15T15:12:06Z

aten/src/ATen/native/TensorConversions.cpp

-  return to_impl(self, self.options().device(device).dtype(dtype), non_blocking);
+  return to_impl(
+      self,
+      self.options().device(device).dtype(dtype),


I know this code originally existed, but I want to point out that it is pretty weird that we have to call self.options() here, because the point of TensorOptions is that it has optional fields, so to_impl should be the method that actually handles the population in the end. No change necessary.

ezyang

I did not carefully check the Python arg parser changes

ezyang · 2019-10-15T15:13:31Z

To clarify on TensorOptions equality testing: I see in other parts of the code that this was a preexisting pattern. So I won't require it to be fixed in this PR.

VitalyFedyunin · 2019-10-15T16:01:11Z

@ailzhang breaks XLA (as almost every PR in this stack)

Summary: Pull Request resolved: pytorch/pytorch#27107 Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Test Plan: Imported from OSS Differential Revision: D17931062 Pulled By: VitalyFedyunin fbshipit-source-id: 2c5dd3dd05bf58a9a29f25562cd45190b009c3f9

facebook-github-bot · 2019-10-15T23:46:53Z

@VitalyFedyunin merged this pull request in d39ab03.

ghstack-source-id: 46697e1 Pull Request resolved: pytorch/pytorch#27107

Summary: Pull Request resolved: pytorch#27107 Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Test Plan: Imported from OSS Differential Revision: D17931062 Pulled By: VitalyFedyunin fbshipit-source-id: 2c5dd3dd05bf58a9a29f25562cd45190b009c3f9

[WIP] Add memory_format support to and type operators

1681e75

[ghstack-poisoned]

VitalyFedyunin requested a review from apaszke as a code owner September 30, 2019 22:25

pytorchbot added module: autograd Related to torch.autograd, and the autograd engine in general module: internals Related to internal abstractions in c10 and ATen module: operators module: pybind Related to our Python bindings / interactions with other Python libraries labels Sep 30, 2019

VitalyFedyunin mentioned this pull request Sep 30, 2019

Add memory format argument to the clone operator #27106

Closed

VitalyFedyunin added a commit that referenced this pull request Sep 30, 2019

[WIP] Add memory_format support to and type operators

e44ef56

ghstack-source-id: c5c9d56 Pull Request resolved: #27107

Update on "[WIP] Add memory_format support to and type operators"

885d6bf

[ghstack-poisoned]

Update on "[WIP] Add memory_format support to and type operators"

488d8c1

[ghstack-poisoned]

This was referenced Oct 2, 2019

Add memory format support to cpu and cuda operators #27223

Closed

Add memory format support to typecasting shortcuts byte,char,double,bool,half,int,long,short,float,bfloat16 #27228

Closed

Update on "[WIP] Add memory_format support to and type operators"

1669942

[ghstack-poisoned]

VitalyFedyunin mentioned this pull request Oct 2, 2019

Add memory format support to empty_like operator #27244

Closed

Update on "Add memory_format support to and type operators"

bab2998

[ghstack-poisoned]

VitalyFedyunin changed the title ~~[WIP] Add memory_format support to and type operators~~ Add memory_format support to and type operators Oct 2, 2019

VitalyFedyunin mentioned this pull request Oct 3, 2019

Add memory format support to full_like operator #27262

Closed

VitalyFedyunin mentioned this pull request Oct 3, 2019

Add memory format support to ones_like operator #27270

Closed

pytorchbot added the module: onnx Related to torch.onnx label Oct 3, 2019

VitalyFedyunin requested review from ezyang, gchanan, ifedan and zou3519 October 3, 2019 22:10

ezyang reviewed Oct 4, 2019

View reviewed changes

This was referenced Oct 8, 2019

Add memory format support to rand_like operator #27561

Closed

Add memory format support to zeros_like operator #27562

Closed

VitalyFedyunin mentioned this pull request Oct 14, 2019

Draft PR. #27885

Closed

VitalyFedyunin requested a review from ezyang October 14, 2019 18:16

This was referenced Oct 14, 2019

Add memory format support to randint_like operator #27889

Closed

Add memory format support to randn_like operator #27890

Closed

Cleanup testing of _like operators #27891

Closed

VitalyFedyunin added 2 commits October 14, 2019 12:46

VitalyFedyunin mentioned this pull request Oct 15, 2019

Add memory format support to resize_as_ operator #27979

Closed

ezyang reviewed Oct 15, 2019

View reviewed changes

ezyang approved these changes Oct 15, 2019

View reviewed changes

facebook-github-bot closed this in d39ab03 Oct 15, 2019

ailzhang mentioned this pull request Oct 15, 2019

Add memory_format to to overrides. pytorch/xla#1200

Merged

facebook-github-bot added the merged label Oct 15, 2019

facebook-github-bot deleted the gh/VitalyFedyunin/2/head branch October 28, 2019 22:07

xxtEchjovs44 pushed a commit to xxtEchjovs44/pytorch that referenced this pull request Jan 29, 2020

Add memory_format support to and type operators

a83f45e

ghstack-source-id: 46697e1 Pull Request resolved: pytorch/pytorch#27107

mruberry added the Merged label Oct 28, 2020

Conversation

VitalyFedyunin commented Sep 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ezyang commented Oct 4, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ezyang commented Oct 14, 2019

Uh oh!

VitalyFedyunin commented Oct 14, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ezyang left a comment

Choose a reason for hiding this comment

Uh oh!

ezyang commented Oct 15, 2019

Uh oh!

VitalyFedyunin commented Oct 15, 2019

Uh oh!

facebook-github-bot commented Oct 15, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

VitalyFedyunin commented Sep 30, 2019 •

edited

Loading