we silently enabled type promotion for kron :-)

This will be worse than std::vector for 3+d tensors, because it will have to grow original 5-element vector

changed to c10::SmallVector<int64_t, 10>

style nit: parens after the equals sign for readability

decltype i to maxdim's type to be consistent with auto usage

Does this need to use resize_output? Actually, in this case things seem trickier than that.

What kind of tensor can a use pass to out= here without this complaining? If a user passes a tensor with shape result_reshape then can mul_out yell that it received a tensor of the incorrect size? If a user passes a tensor of the shape that mul_out likes then can it be resized without warning? That'd be bad if out= was a view, for example.

Is out=view a defined behavior?

Yes, typically it uses resize_ on out, so if out is correct size, it is directly used and view relationship persists, if it is not then anything could happen, so in that sense it's not defined. Here though preserving this behavior would require first re-viewing out with expected size of mul if possible (for reductions with similar problems I inserted unsqueezes and not cared about actual sizes, if they are wrong the resize_ would handle it), and then re-viewing it back.

Looks like the current approach preserve storage, but will erase discontiguity:

>>> import torch

>>> a = torch.randn(4,6,8,10).contiguous(memory_format=torch.channels_last)
>>> a.stride()
(480, 1, 60, 6)
>>> a.view(2,2,2,3,2,4,2,5).stride()
(960, 480, 3, 1, 240, 60, 30, 6)
>>> a.resize_(2,2,2,3,2,4,2,5).stride()
(960, 480, 240, 80, 40, 10, 5, 1)

>>> a = torch.randn(4,6,8,10).contiguous(memory_format=torch.channels_last)
>>> a.view(2,2,2,3,2,4,2,5).resize_(2,2,2,3,2,4,2,5).stride()
(960, 480, 3, 1, 240, 60, 30, 6)

>>> a = torch.randn(4,6,8,10).contiguous(memory_format=torch.channels_last)
>>> a.storage().data_ptr()
94728832336256
>>> a.resize_(2,2,2,3,2,4,2,5).storage().data_ptr()
94728832336256

Should we do

result.resize_(result_reshape, self.suggest_memory_format());

instead of resize_output?

Sorry, why? We want to be consistent with PyTorch, where the out= semantics are that the operation is performed and then its output is "safe copied" to the tensor passed to out=. If the tensor has to be altered then a warning is thrown, unless the tensor had no elements.

No, we should preserve any permutation, like we automatically do now, not just channels last.
For a warning, we can do result.view(...with inserted 1 dimensions...) and rely on mul_out to throw a warning if sizes are wrong, or do as Xiang suggested a few comments up, and use resize_output there instead of resize_, it doesn't really matter.

Or maybe add an argument c10::optional<MemoryFormat> optional_memory_format to resize_output, and inside resize_output, check for both shape and memory format.

Please no, use of memory_format argument is wrong in most cases outside of torch.nn.

We should use this pattern more often.

No, Sebastian wants to disallow it.