len(memoryview.shape) might not equal memoryview.ndim

Very interesting! When is that not true?

len(memoryview.shape) might not equal memoryview.ndim

Very interesting! When is that not true?

On my installation :P
I don't know why, but pyobj.shape[i] was [2,0,0,0,0,0,0] in one of the test in tests/test_buffer_region.py

@madsbk is it possible to add something which can explicitly test this ? Or are the tests in buffer_region.py enough ?

len(memoryview.shape) might not equal memoryview.ndim

Very interesting! When is that not true?

On my installation :P
I don't know why, but pyobj.shape[i] was [2,0,0,0,0,0,0] in one of the test in tests/test_buffer_region.py

Ah right! Yeah sometimes C code will allocate a buffer larger than the needed shape. Good catch. Sorry forgot about this.

These tests pass for me -- @Akshay-Venkatesh should we merge ?

Also, can someone ELI5 what was the problem exactly and what this PR is doing?

@Akshay-Venkatesh I can try but I am not as familiar with buffer protocols as @jakirkham / @madsbk @kkraus14 . On my machine (and @madsbk as well) pyobj, in populate_ptr looks like the following:

[2, 0, 0, 0, 0, 0, 0, 0]

@jakirkham suggested that, " C code will allocate a buffer larger than the needed shape" . The only reference I could find about this is in the numpy docs (look at the shape definition). The cython docs recommend simply handling the 1st dimension of shape:

self.shape = pyobj.shape[0]

For reference in a python session we see the following:

In [1]: memoryview(b'hi').shape                                                                                                                                                       
Out[1]: (2,)

In [2]: memoryview(b'hi').ndim                                                                                                                                                        
Out[2]: 1

I confirmed the large buffer size by putting a print statement in the cpdef _populate_ptr definition.

If there are no objects can we/I merge this tomorrow ?

The shape is now a tuple instead of a list.
@quasiben, I think it is ready for merging now.

Apologies for hastily rushing this along. I tested this PR with test_custom_send_recv and got some errors:

>   self.shape[0] = len
E   TypeError: 'tuple' object does not support item assignment

ucp/_libs/ucp_py_buffer_helper.pyx:233: TypeError

In various functions inside the buffer helper, ucx-py is assigning shape and this fails if shape is a tuple.
For example:

@madsbk @Akshay-Venkatesh what are you're thoughts here ? Should this be fixed in a follow on PR or continued to be worked in this PR ?

@quasiben, It should be fixed now; can you confirm?

Thank you @madsbk ! The tests for buffer and send_recv are passing now. I'm going to merge and continue cleaning up tests