I like this idea in general. I am wondering if it is too much noise, but in general it seems a good idea if we (or those that could make use of it like astropy ;)), add that keyword to other functions as well.

@seberg, yes, it would be useful in other functions as well, the various stack operations being prime examples. But for those subclasses might in principle need to check that they are consistent, so I thought I would start with a case where I could not see any reason not to allow it. See also the discussion in #4164.

Not sure about that ;)

In [9]: class a(object):
    def __array__(self):
        return np.arange(10)
    def __array_wrap__(self, arr):
        print "array wrap"
        return self
    def __array_finalize__(self):
        print "finalize"
   ...:         

In [10]: np.reshape(a(), (5, 2))
array wrap
Out[10]: <__main__.a at 0x3f5be50>

Am much amused by your example! According to the documentation, __array_wrap__ is only called at the end of every ufunc, and a quick test shows that it is not called when the class is an ndarray subclass. But clearly there is more to it...
I guess I partially see what you mean with the class: for as_strided we can not necessarily be sure that x is actually an array, just that it has an __array_interface__. I rewrote to make it safer.

Not sure, but the default __array_wrap__ will itself call __array_finalize__ I believe, so that might add to the confusion. Array wrap is there for ufuncs, but it is called elsewhere (though I don't know exactly where, might be pretty much everywhere).

@seberg - carrying on in the main thread so we don't loose the discussion once I push the next installment...
view indeed calls __array_finalize__, but it will be passed array as the object, which does not carry information that the subclass might need. Probably we should mimic what happens with np.reshape. Looking at fromnumeric.py, if the input is an ndarray subclass, it goes to the reshape method, which is defined in PyArray_NewShape (in multiarray/shape.c), where at the end a call is made to PyArray_NewFromDescr (in ctors.c), which goes to PyArray_NewFromDescr_int (ibid), which does call __array_finalize__. If np.reshape encounters something that is not an ndarray, it uses _wrapit, which does check for the existence of __array_wrap__. The idea here seems to be able to deal with other classes that are array-like.

My sense, though, is that we're making it slightly too hard: subok should probably just be what it is in np.array: indicate that it passes through ndarray subclasses, not that it tries to behave for everything that has an __array_wrap__. (And for the specific, most common, case of np.broadcast_arrays, we are already guaranteed to get an ndarray or, with this PR, a subclass.)

p.s. The travis error seems spurious (a build hanging).

Updated a comment; travis now passes.

@seberg - did you have further comments on this?

Well, still not sure if I don't prefer wrap. We have precedence of wrapping for example in insert and delete, though these are old and it is probably OK to rethink... If we see this more as array creation functions, I think not wrapping is better. Otherwise I think wrapping is better.

@seberg - looking at the documentation for __array_finalize__ and __array_wrap__ [1], it would seem the description of the former (this method is called whenever the system internally allocates a new array from obj, where obj is a subclass (subtype) of the ndarray) is far more appropriate than that of the latter (At the end of every ufunc, this method is called on the input...), so I think __array_finalize__ makes most sense from the least-surprise principle.

That said, I can see the case for x.__array_wrap__(obj) being slightly more obvious than obj.__array_finalize__(x), even if it does the right thing in offering obj the opportunity to copy over relevant attributes from x.

Of course, in principle, we could behave more closely similar to np.reshape and use __array_finalize__ for ndarray subclasses and __array_wrap__ for others. But perhaps that should wait for someone actually requesting that behaviour?

[1] http://docs.scipy.org/doc/numpy/reference/arrays.classes.html

@seberg - what shall we do with this one? It would be nice to reach some conclusion, so I can also provide this in astropy/astropy#2327

I still prefer using array wrap a little, but if we view it as an array creation function I am fine with either really. At some point it would be nice to move this into C in any case, but just wishing ;)

@seberg - can we stick to __array_finalize__, by analogy with what np.reshape does? (see comment above). It would be nice to get this in (and not mimick #31)

@seberg @mhvk It would be nice to get this finished off.

Hmmm, array finalize is probably good. But now I am wondering if it isn't cleaner to call x = np.array(x, copy=False, subok=subok) to begin with, and then just view the resulting array result.view(type=class(x))? Have to think about it a little.

@seberg @mhvk Knock, knock ;) This also needs a rebase.

OK, I rebased. @seberg - I'm not quite sure where in as_strided you were thinking of calling x = np.array(x, copy=False, subok=subok).

Sorry for the delays. Just need to figure out the array-like but not subclass thing, but lets finish it up before the week is over :)

@seberg - OK, I made the changes, but in a slightly different way than you suggested, so please have a look.

Thanks for the comment; addressed, and a test added to ensure subclasses without __array_finalize__ work as well.

On DummyArray, now that we are doing the np.array(x), I would think it would indeed be nicer to also use

if shape is None:
    shape = x.shape
if strides is None:
    strides = x.strides
array = np.ndarray(buffer=x, shape=shape, strides=strides, dtype=x.dtype)

It would allow one to remove the check on dtype.kind, but not, as you said, the __array_finalize__.

Do you think I should still do this here, or would this be better for a different PR, since it is getting a bit off-topic? (In princple, I can also back out the np.array(x).)

Actually, have to check what/if this does anynthing to object arrays (or void arrays with object fields).

@seberg - I had almost started adding tests for record types at least, but felt too lazy... But can do so, as it would be good to improve the test coverage.

For object arrays, would a test like the following be OK? Could you give me an example of "void arrays with object fields"?

a = np.array([None, True, False])
as_strided(a, strides=(0,))
# seems to work: array([None, None, None], dtype=object)

Actually, I am surprised it isn't an error :), it certainly is good, just need to see if it changed.

@seberg - I came back to this again, and while a test with an object dtype works, a complex one does not in current master:

import numpy as np
dt = np.dtype([('x', 'i8'), ('y', '?'), ('z', 'O')])
b = np.array([(1, True, None), (2, False, [3, 4, 5])], dtype=dt)
c = np.array([[-1], [-2]])
b_strided, c_strided = np.broadcast_arrays(b, c)
TypeError                                 Traceback (most recent call last)
...
/usr/lib/python3/dist-packages/numpy/lib/stride_tricks.py in as_strided(x, shape, strides)
     31     # Make sure dtype is correct in case of custom dtype
     32     if array.dtype.kind == 'V':
---> 33         array.dtype = x.dtype
     34     return array
     35 

TypeError: Cannot change data-type for object array.

This error disappears if instead of the DummyArray class, I use the

array = np.ndarray(buffer=x, shape=shape, strides=strides, dtype=x.dtype)

assignment suggested above. However, this does not work for non-contiguous data and thus fails one of the tests.

Since the void arrays with object fields fail in current master as well, however, I think this is not really relevant for the current PR, which just aims to get subclasses to work properly. Maybe we can agree that this is OK as is, and raise an issue for the above?

We don't need that kind of error here. Since we don't mess with the dtype and the original array is in charge of doing handling reference counting, it is always safe to do. As long as we have less errors then before things are good.

It fails a test which originally did not exist is what you mean and which never worked? In that case I guess we can call it a minor issue that can be fixed another time.

@seberg - indeed, both with and without this PR as_strided does not work for a void array with object fields, since one cannot directly set the dtype. I raised a new issue for this (#5081), even though this is rather a corner case. Does this leave anything for the present PR?

Yeah, should be good for merging. Won't do it right now, but if I forget about it in the next couple of days, just ping me. Thanks for putting up with all my small worries :).

@seberg - the promised ping to remind you to merge this!

Ack :), lets put it in then. At some point we should maybe just move this whole stuff to C...

Thanks for the patience :)