Skip to content

Use Cython 3.0.x#15402

Merged
saimn merged 7 commits into
astropy:mainfrom
saimn:cython3
Oct 31, 2023
Merged

Use Cython 3.0.x#15402
saimn merged 7 commits into
astropy:mainfrom
saimn:cython3

Conversation

@saimn

@saimn saimn commented Sep 28, 2023

Copy link
Copy Markdown
Contributor

With Cython 3.0 we can now remove all the NPY_NO_DEPRECATED_API warnings, I fixed most of them but a few ones remain (wip).

https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html

Description

Fixes #15315

  • By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

@github-actions

github-actions Bot commented Sep 28, 2023

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • Is a milestone set? Milestone must be set but we cannot check for it on Actions; do not let the green checkmark fool you.
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

@github-actions

Copy link
Copy Markdown
Contributor

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

Comment thread astropy/table/_column_mixins.pyx Outdated
cdef int ndim "nd"
pass
# FIXME: doesn't work with recent Numpy API ?
# cdef int ndim "nd"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wasn't working with the "new" Numpy API. Not sure what it does or why it was needed.
Also maybe it is possible now to import ndarray more directly without resorting to numpy/arrayobject.h.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is old code and I'm not sure of the details, but we probably need to understand it a bit before moving forward with this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed; those changes from #4075 were performance-related, so we should probe a bit beyond passing tests.
However on a simple replication of the indexing timeits in #3930 I find 5-15% runtime increases (~77 ns -> 89 ns for the 1d, 182 ns -> 193 ns for the 2d case), which seems very acceptable.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually since this works now without the ndim method, it would seem the entire custom ndarray is no longer needed, meaning basically 37014a9 could be reverted altogether. Against current main this would be

--- a/astropy/table/_column_mixins.pyx
+++ b/astropy/table/_column_mixins.pyx
@@ -25,6 +25,7 @@ Column is itself an array.
 import sys
 
 import numpy as np
+cimport numpy as cnp
 
 
 cdef tuple INTEGER_TYPES = (int, np.integer)
@@ -45,21 +46,18 @@ cdef extern from "Python.h":
         PyMappingMethods* tp_as_mapping
 
 
-cdef extern from "numpy/arrayobject.h":
-    ctypedef class numpy.ndarray [object PyArrayObject]:
-        pass
-        # FIXME: doesn't work with recent Numpy API ?
-        # cdef int ndim "nd"
+ndarray = np.ndarray
+ctypedef cnp.ndarray ndarray_t
 
 
 ctypedef object (*item_getter)(object, object)
 
 
 cdef inline object base_getitem(object self, object item, item_getter getitem):
-    if (<ndarray>self).ndim > 1 and isinstance(item, INTEGER_TYPES):
+    if (<ndarray_t>self).ndim > 1 and isinstance(item, INTEGER_TYPES):
         return self.data[item]
 
-    dtype_kind = (<ndarray>self).dtype.kind
+    dtype_kind = (<ndarray_t>self).dtype.kind
     if dtype_kind == 'V' and isinstance(item, STRING_TYPES):
         return self.data[item]

This would even restore the current indexing performance in main (tested with numpy 1.26.0, 1.23.5 – 1.22 is breaking pyerfa in my mamba installation).

I suspect there is some more of the boilerplate code @embray mentioned in #4075 that could be removed with Cython 3, but I don't have much deeper insight here...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dhomeier - Yes, I came to the same conclusion while having a deeper look earlier today, see 0e21829.
This was probably needed for older version of Cython (at the time the code was added).

I also tried to remove the includes of PyMappingMethods and PyTypeObject but it seems we still need that to access .tp_as_mapping.mp_subscript.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that __Pyx_PyObject_GetItem would do just that, but I have not found any examples how to use that in Cython (or verify whether that would really give us the base class __getitem__).
In any case as the timings for the 2 and 3d cases indicate, at least in Python 3.11 the overhead from directly calling data[item] is now down to about a factor of 2, so there should not be serious performance repercussions either way.

@saimn saimn mentioned this pull request Sep 28, 2023
@pllim

pllim commented Sep 28, 2023

Copy link
Copy Markdown
Member

Do you also have to pull in #15399 by @nstarman ?

@saimn

saimn commented Sep 28, 2023

Copy link
Copy Markdown
Contributor Author

Given the failures in CI, I guess so.

@nstarman

Copy link
Copy Markdown
Member

Feel free to cherry pick the commit :)

@saimn saimn marked this pull request as ready for review October 28, 2023 20:58
@saimn saimn requested review from a team, dhomeier, larrybradley and taldcroft as code owners October 28, 2023 20:58
@saimn saimn requested review from nstarman and removed request for a team October 28, 2023 20:58
@saimn saimn requested review from astrofrog and removed request for dhomeier, larrybradley, nstarman and taldcroft October 28, 2023 20:58
@saimn

saimn commented Oct 28, 2023

Copy link
Copy Markdown
Contributor Author

PR is ready (and tests were passing before the rebase). I removed all the NPY_1_7_API_VERSION warnings and a few other ones. I think we will need Cython 3 for Numpy 2.0 so we should get this for the first rc.

@taldcroft taldcroft left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For io.ascii, I honestly don't understand the changes but if it is passing tests then that is fine. For table we need to understand that change to the column mixin which provides the fast column item access.

Comment thread astropy/table/_column_mixins.pyx Outdated
cdef int ndim "nd"
pass
# FIXME: doesn't work with recent Numpy API ?
# cdef int ndim "nd"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is old code and I'm not sure of the details, but we probably need to understand it a bit before moving forward with this.

@astrofrog astrofrog left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the changes to timeseries.

@larrybradley larrybradley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stats and convolution look good

@dhomeier dhomeier left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

io.ascii and everything else works, LGTM!

Comment thread astropy/table/_column_mixins.pyx Outdated
cdef int ndim "nd"
pass
# FIXME: doesn't work with recent Numpy API ?
# cdef int ndim "nd"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that __Pyx_PyObject_GetItem would do just that, but I have not found any examples how to use that in Cython (or verify whether that would really give us the base class __getitem__).
In any case as the timings for the 2 and 3d cases indicate, at least in Python 3.11 the overhead from directly calling data[item] is now down to about a factor of 2, so there should not be serious performance repercussions either way.

@saimn

saimn commented Oct 31, 2023

Copy link
Copy Markdown
Contributor Author

Ok I think we are good, thanks for the reviews !

@saimn saimn merged commit 109bf3d into astropy:main Oct 31, 2023
@saimn saimn deleted the cython3 branch October 31, 2023 18:13
meeseeksmachine pushed a commit to meeseeksmachine/astropy that referenced this pull request Oct 31, 2023
pllim added a commit that referenced this pull request Oct 31, 2023
…402-on-v6.0.x

Backport PR #15402 on branch v6.0.x (Use Cython 3.0.x)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cython3 chokes on astropy 5.3.3 (worked in 5.3.2)

7 participants