Skip to content

Update GetItem to support __class_getitem__ for type objects#3518

Merged
scoder merged 2 commits intocython:masterfrom
msg555:master
Apr 20, 2020
Merged

Update GetItem to support __class_getitem__ for type objects#3518
scoder merged 2 commits intocython:masterfrom
msg555:master

Conversation

@msg555
Copy link
Contributor

@msg555 msg555 commented Apr 15, 2020

To address #2753

TODO: Write tests

@msg555 msg555 marked this pull request as draft April 15, 2020 07:50
@da-woods
Copy link
Contributor

Looking at it again, I think what I'd do is merge __Pyx_PyObject_GetIndex into __Pyx_PyObject_GetItem - I don't think it needs to be a separate function. That lets you keep the if (unlikely(!(m && m->sq_item))) { block which I think is necessary for __Pyx_GetItemInt_Fast to be optimized when it's inlined.

Obviously that may be contradicted by others, but this looks to me to be the way to do it with no changes to how well it runs.

@msg555 msg555 force-pushed the master branch 2 times, most recently from c963359 to 5c194be Compare April 15, 2020 10:24
if not self.is_classmethod and self.name in ('__class_getitem__', '__init_subclass__') and env.is_py_class_scope:
from .ExprNodes import NameNode
self.decorators = self.decorators or []
self.decorators.append(DecoratorNode(self.pos, decorator=NameNode(self.pos, name='classmethod')))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self.decorators.append(DecoratorNode(self.pos, decorator=NameNode(self.pos, name='classmethod')))
self.decorators.append(DecoratorNode(self.pos, decorator=NameNode(self.pos, name=EncodedString('classmethod'))))

Apart from that – what is this decorator needed for? Isn't setting self.is_classmethod enough?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I borrowed this logic from another place in the same file where we add a staticmethod decorator. Should that spot likely change as well?

As for why we need the decorator at all - in my testing it seems like setting is_classmethod alone was not enough. I believe the issue is that we need the logic generated from https://github.com/cython/cython/blob/master/Cython/Compiler/Symtab.py#L2006. It probably makes sense for this logic to be generated based off of the is_classmethod setting and for the decorator to be ignored but that refactor is at least currently beyond my familiarity with this codebase.

@msg555 msg555 force-pushed the master branch 3 times, most recently from e675883 to 2db6125 Compare April 16, 2020 03:36
if sys.version_info[0:2] >= (3, 6):
IMPLICIT_CLASSMETHODS.add("__init_subclass__")
if sys.version_info[0:2] >= (3, 7):
IMPLICIT_CLASSMETHODS.add("__class_getitem__")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open to suggestions on a better place for these definitions to exist and/or better way to express this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely the wrong way to check the version of Python - the version used to run Cython isn't necessarily the same as the version of Python the code will run in.

I still don't think this needs version checks - there's no harm in supporting this even on Python 2.7. Nobody should be using these attributes for anything else since I'm fairly sure Python reserves all unused special method names for its own use.

@msg555 msg555 marked this pull request as ready for review April 16, 2020 03:45

#if PY_VERSION_HEX >= 0x03070000
if (likely(PyType_Check(obj))) {
PyObject *meth = __Pyx_PyObject_GetAttrStr(obj, PYIDENT("__class_getitem__"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just noticed that this may raise AttributeError, which the _PyObject_LookupAttr() in the original code does not. There's a __Pyx_PyObject_GetAttrStrNoError() for that purpose. And we should make sure this is covered by tests.

Copy link
Contributor Author

@msg555 msg555 Apr 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, I'll add a test to test behavior when __class_getitem__ does not exist however even with the current implementation it still later raises a TypeError which I guess overwrites the AttributeError. I'll update to use the suggested method, though.

@@ -0,0 +1,135 @@
# mode: run
# tag: pure3.7
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like I'm failing CI from this test on python versions <3.7. Any idea what might be wrong here @da-woods?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Py2, the classes are old-style classes, not types. Let them inherit from (object).

And the _patched test depends on the __init_subclass__ protocol, which is Py3.6+, it seems. You can disable it in Py<3.5 with a @skipIf() decorator.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was actually under the impression that the "tag" comment would instruct the test runner to ignore this test for python < 3.7 which is what I really wanted to do.

Copy link
Contributor

@da-woods da-woods Apr 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"pure3.7" runs the (Cython compiled) tests in all versions of Python, and also runs the tests in uncompiled-Python for Python>=3.7. I think this is correct because it should be possible for Cython to support __class_getitem__ all the time. Essentially it's just providing confirmation that Cython matches Python behaviour (for the versions of Python where this is supported)

The __init_subclass__ is probably something that has to be in the Python interpreter and Cython can't do on its own, so it makes sense to disable conditionally for some versions of Python.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old style classes are actually a bug in current master. With language_level=3, there shouldn't be any old style classes in Py2 any more. I pushed a fix as #3530, let's see if that breaks anything in the test suite.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The __init_subclass__ is probably something that has to be in the Python interpreter and Cython can't do on its own

There's actually #2781 about this. PEP-487 Just needs to get implemented.

@msg555 msg555 force-pushed the master branch 2 times, most recently from e0defa6 to 708fb11 Compare April 17, 2020 23:52
@msg555 msg555 changed the title Default to PyObject_GetItem for unhandle cases Update GetItem to support __class_getitem__ for type objects Apr 17, 2020
@scoder
Copy link
Contributor

scoder commented Apr 19, 2020

@msg555 could you please merge in the latest master branch? That should fix the tests.

@da-woods
Copy link
Contributor

@msg555 could you please merge in the latest master branch? That should fix the tests.

I think the tests are automatically run on the PR merged with the master branch it might not be necessary to merge. I could be wrong but it'll probably be clear when the tests run.

@msg555
Copy link
Contributor Author

msg555 commented Apr 19, 2020

Rebased on master with the suggested changes

@scoder scoder added this to the 3.0 milestone Apr 20, 2020
@scoder scoder added the Python Semantics General Python (3) language semantics label Apr 20, 2020
@scoder scoder merged commit c0ceabe into cython:master Apr 20, 2020
@scoder
Copy link
Contributor

scoder commented Apr 20, 2020

Thanks @msg555 !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Python Semantics General Python (3) language semantics

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants