Skip to content

ENH: Add support for the abstract scalars to cython code#17150

Merged
seberg merged 1 commit intonumpy:masterfrom
eric-wieser:cython-isinstance
Sep 12, 2020
Merged

ENH: Add support for the abstract scalars to cython code#17150
seberg merged 1 commit intonumpy:masterfrom
eric-wieser:cython-isinstance

Conversation

@eric-wieser
Copy link
Member

@eric-wieser eric-wieser commented Aug 24, 2020

This makes isinstance checks slightly faster..

This aims to replace #16363.

I'm not invested enough to go through the effort of writing the tests it would entail, but a nice follow-up for someone interested would be to add something like:

    # concrete types
    ctypedef class numpy.bool_ [object PyBoolScalarObject]:
        cdef npy_bool obval
    ctypedef class numpy.byte [object PyByteScalarObject]:
        cdef npy_byte obval
    ctypedef class numpy.short [object PyShortScalarObject]:
        cdef npy_short obval
    ctypedef class numpy.intc [object PyIntScalarObject]:
        cdef npy_intc obval
    ctypedef class numpy.int_ [object PyLongScalarObject]:
        cdef npy_int_ obval
    ctypedef class numpy.longlong [object PyLongLongScalarObject]:
        cdef npy_longlong obval
    ctypedef class numpy.ubyte [object PyUByteScalarObject]:
        cdef npy_ubyte obval
    ctypedef class numpy.ushort [object PyUShortScalarObject]:
        cdef npy_ushort obval
    ctypedef class numpy.uintc [object PyUIntScalarObject]:
        cdef npy_uintc obval
    ctypedef class numpy.uint [object PyULongScalarObject]:
        cdef npy_uint obval
    ctypedef class numpy.ulonglong [object PyULongLongScalarObject]:
        cdef npy_ulonglong obval
    ctypedef class numpy.half [object PyHalfScalarObject]:
        pass  # no C type exists
    ctypedef class numpy.single [object PyFloatScalarObject]:
        cdef npy_float obval
    ctypedef class numpy.double [object PyDoubleScalarObject]:
        cdef npy_double obval
    ctypedef class numpy.longdouble [object PyLongDoubleScalarObject]:
        cdef npy_longdouble obval
    ctypedef class numpy.csingle [object PyFloatScalarObject]:
        cdef npy_cfloat obval
    ctypedef class numpy.cdouble [object PyDoubleScalarObject]:
        cdef npy_cdouble obval
    ctypedef class numpy.clongdouble [object PyLongDoubleScalarObject]:
        cdef npy_clongdouble obval
    # TODO: add non-numeric types

This makes `isinstance` checks slightly faster
@eric-wieser

This comment has been minimized.

@eric-wieser
Copy link
Member Author

@scoder, is the only way we can avoid writing everything twice by eventually dropping Cython < 3.0?

@eric-wieser
Copy link
Member Author

eric-wieser commented Aug 25, 2020

I don't seem to have permission to restart the azure run. @mattip, @charris, do you know who does?

@mattip
Copy link
Member

mattip commented Aug 25, 2020

Restarted. You need to be a member of https://dev.azure.com/numpy/numpy. If you have a user name on dev.azure.com I can add you.

@eric-wieser

This comment has been minimized.

@mattip
Copy link
Member

mattip commented Aug 25, 2020

hmm, maybe @charris has to do it. I get a message about that user having an invalid email, but it might be confused since @charris is the adiminstrator not me

@eric-wieser

This comment has been minimized.

@eric-wieser
Copy link
Member Author

eric-wieser commented Aug 25, 2020

@jakirkham, @jbrockmendel, is this all you need?

@charris
Copy link
Member

charris commented Aug 25, 2020

@eric-wieser I get the error "Given email address eric-wieser is invalid". Looks like the full email address you registered with is needed.

@charris
Copy link
Member

charris commented Aug 25, 2020

@eric-wieser The email address you got in the error message worked, you are now added. Not sure what your permissions mean...

@eric-wieser
Copy link
Member Author

Yeah, looks like I have no permisions, but numpy does now appear in my list of projects

@scoder
Copy link
Contributor

scoder commented Aug 25, 2020

is the only way we can avoid writing everything twice by eventually dropping Cython < 3.0?

Take it as a nudge to drop Cython 0.x support rather sooner than later. :)

You could use includes to merge multiple files into one. Question is whether you want a single source for most (but not all) of the content right now (i.e. maintain three distinct files and lose the file history when merging them back together), or whether you accept the redundancy because you prefer cleanly dropping a single file at some point when you remove Cython 0.x support. Personally, I'd go for the latter, but YMMV.

@mattip
Copy link
Member

mattip commented Aug 25, 2020

When Cython3 comes out we can require it and drop the older support pretty quickly. The older one is still shipped with Cython<3, so we shouldn't be breaking anyone's workflow.

@jakirkham
Copy link
Contributor

@jakirkham, is this all you need?

Sorry I've lost the thread. What is this related to?

@eric-wieser
Copy link
Member Author

eric-wieser commented Aug 25, 2020

@jakirkham, @jbrockmendel, in regards to your original motivation for #16363

@jakirkham
Copy link
Contributor

@jakirkham, in regards to your original motivation for #16363

Sorry I'm still lost

@eric-wieser
Copy link
Member Author

eric-wieser commented Aug 25, 2020

Put simply: if this gets merged, can your PR be closed in your opinion?

@jakirkham
Copy link
Contributor

Put simply: if this gets merged, can your PR be closed in your opinion?

Maybe you are meaning to ask @jbrockmendel?

@eric-wieser
Copy link
Member Author

Whoops, I did. Sorry for the confusion.

@jakirkham
Copy link
Contributor

No worries 🙂 Wasn’t sure if there was a relevant NumPy issue I had forgotten 😅

@jbrockmendel
Copy link
Contributor

Put simply: if this gets merged, can your PR be closed in your opinion?

Quite likely. I'd like to check that performance doesn't take a hit as compared with the implementation in my PR.

@jbrockmendel
Copy link
Contributor

Could the same magic be used to make datetime64 and timedelta64 available?

@eric-wieser
Copy link
Member Author

Yes, but let's leave that to a follow up

@seberg
Copy link
Member

seberg commented Sep 12, 2020

Lets put this in, it has been sitting around for way to long and seems completely unproblematic to me. Thanks!

@seberg seberg merged commit 8adeca2 into numpy:master Sep 12, 2020
@charris charris mentioned this pull request Oct 10, 2020
20 tasks
@jbrockmendel
Copy link
Contributor

Could the same magic be used to make datetime64 and timedelta64 available?

Yes, but let's leave that to a follow up

Trying to follow up on this, added

    ctypedef class numpy.datetime64 [object PyObject]:
        pass
    ctypedef class numpy.timedelta64 [object PyObject]:
        pass

along with the declarations in this PR, but at import time I'm getting

RuntimeWarning: numpy.datetime64 size changed, may indicate binary incompatibility. Expected 16 from C header, got 32 from PyObject

which is enough to stop the tests from running. Did you have something else in mind for how the followup would work?

@seberg
Copy link
Member

seberg commented Jan 5, 2021

Is numpy.datetime64 currently exposed as the full struct? You can add an ignore flag like for ndarray I guess, although I am surprised it is necessary, and I am curious whether this is just a matter of recompiling? (assuming this is within NumPy itself, if it is outside you should not have to recompile of course.)

@eric-wieser
Copy link
Member Author

eric-wieser commented Jan 5, 2021

@jbrockmendel, you need to use object PyDatetimeScalarObject as I do in the top PR comment. I suspect you'll need to remove the current PyDatetimeScalarObject cython struct two, to prevent them overlapping.

@jbrockmendel
Copy link
Contributor

When I disable

    ctypedef struct PyDatetimeScalarObject:
        # PyObject_HEAD
        npy_datetime obval
        PyArray_DatetimeMetaData obmeta

    ctypedef struct PyTimedeltaScalarObject:
        # PyObject_HEAD
        npy_timedelta obval
        PyArray_DatetimeMetaData obmeta

and in its place put

    ctypedef class numpy.datetime64 [object PyDatetimeScalarObject]:
        cdef npy_datetime obval
        cdef PyArray_DatetimeMetaData obmeta
    
    ctypedef class numpy.timedelta64 [object PyTimedeltaScalarObject]:
        cdef npy_timedelta obval
        cdef PyArray_DatetimeMetaData obmeta

I get compile-time errors below where PyDatetimeScalarObject and PyTimedeltaScalarObject are referenced. Disabling those functions for now numpy compiles, but when running pytest numpy/core/tests/test_cython.py i get compile-time errors for checks.pyx complaining about just about everything in the cnp namespace.

just passing instead of declaring the attributes on datetime64/timedelta64 gives the same errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants