Add support for C++ scoped enums by shwina · Pull Request #3640 · cython/cython

shwina · 2020-05-26T17:04:45Z

This PR adds support for C++ scoped enumerations, (enum class). Addresses #1603.

Workarounds exist for wrapping external C++ enum classes. These are reported in the issue above. However, they don't solve the following problems:

The names of the enum values leak into the outer scope as described in Support for C++ scoped enum class #1603 (comment).
Converting between Python values and scoped enums requires further awkward workarounds -- see an example of our workaround here.

This PR addresses the above by adding "true" support for scoped enums.

Enum classes are defined using cdef enum class syntax
~~Adds a new CppScopedEnumDefNode node type~~
Adds a new CppScopedEnum PyrexType
Adds a new CppScopedEnumScope for the enum values
~~Adds utilities enum.from_py and enum.to_py for converting between Python int-like objects and enum values~~
Adds support for cpdef enum class, which generates a Python Enum similar to cpdef enum
Docs

Please note: this is my first time contributing to Cython - please forgive coding faux pas on my part :-)

scoder

Looks pretty sane overall. I'll ask @robertwb if he has a comment or two.

scoder · 2020-05-26T18:59:07Z

Cython/Compiler/Parsing.py

            if s.systring != 'enum':
                error(pos, "C struct/union cannot be declared cpdef")
        return p_struct_enum(s, pos, ctx)
+    elif s.sy == 'IDENT' and s.systring == 'enumclass':


enum class seems better here, I think, and should be perfectly backwards compatible.

Also, this should be restricted to C++ mode.

Agreed - enumclass was just easier to add support for quickly, but will change this to enum class.

scoder · 2020-05-26T19:03:26Z

Cython/Compiler/ExprNodes.py

        if entry.is_type and entry.type.is_extension_type:
            self.type_entry = entry
-        if entry.is_type and entry.type.is_enum:
+        if entry.is_type and (entry.type.is_enum or entry.type.is_scoped_enum):


Is a "scoped enum" not an "enum"? I.e. is it better to have separate types and say "or" from time to time, or have scoped enums be a subtype and say "and not" when they should behave differently?

I don't feel strongly here. I can make it such that a scoped enum "is a" enum.

Semantically speaking, I guess the correct thing would be to have is_enum, is_scoped_enum, and is_unscoped_enum, where scoped_enum and unscoped_enum are the two distinct types of enum.

Not advocating for this though, just presenting it as a third option if it's appealing to you :)

scoder · 2020-05-26T19:05:25Z

Cython/Compiler/ModuleNode.py

+        if not enum_values:
+            error(entry.pos, "Empty enum definition not allowed outside a 'cdef extern from' block")


This seems fairly late for such an error. Isn't this something that the parser could detect? Or at least the declaration analysis?

I moved the this check to the parser. Please note however, that for C-style enums, it is still being done during code generation. Happy to change that too if the parser code looks good to you.

scoder · 2020-05-26T19:07:34Z

Cython/Compiler/PyrexTypes.py

        if constructor is not None and best_match([], constructor.all_alternatives()) is None:
            error(pos, "C++ class must have a nullary constructor to be %s" % msg)

+class ScopedEnumType(CType):


Seems worth naming this CppScopedEnumType to make it stick out in the code as something C++ specific.

scoder · 2020-05-26T19:11:01Z

Cython/Compiler/Symtab.py

+        entry = self.lookup_here(name)
+        if not entry:


This suggests an error test (in tests/errors/ with # mode: error) that a redeclaration gets detected and rejected.

Thanks for catching this - I added an error test

tests/run/cpp_scoped_enums.pyx

Cython/Compiler/PyrexTypes.py

…cython into add-scoped-enums-support

…oped-enums-support

da-woods · 2020-06-06T21:05:00Z

docs/src/userguide/wrapping_CPlusPlus.rst

+
+    cdef enum class Cheese:
+        cheddar = 1
+	camembert = 2


Indentation error

da-woods · 2020-06-06T21:09:17Z

Cython/Compiler/Nodes.py

            enum_entry.type.values.append(entry.name)


+class CppScopedEnumDefNode(StatNode):


This (and CppScopedEnumDefItemNode) look relatively similar to their equivalent regular enums. My personal inclination would be to try to do it all in the existing classes, maybe with an optional scope parameter. Or if there's reasons not to do that then use inheritance to capture the shared bits.

There may be good reasons I'm not seeing why this wouldn't work though.

Thanks and 100% agreed about the code repetition here. I'm more inclined to pursue the inheritance route. The two constructs enum and enum class have different enough behaviour that I think trying to implement them both with the same class could become a hindrance in time.

@da-woods I'm having some second thoughts here and would love your input:

CEnumType inherits from CIntLike, and rightly so, as C-style enums are trivially convertible to integers: https://wandbox.org/permlink/b7kkoNz4gI2mjNXx

However, scoped enums are not trivially convertible to integers: https://wandbox.org/permlink/2CB5lmrX04rokcT6. So implicit treatment of a scoped enum as an int at the C++ level would likely break things.

As I mentioned here, I think the right semantics could be captured with something like a BaseEnum class, and subclasses CEnum and CppScopedEnum.

Does that sound reasonable?

Ah apologies, I see that this comment if about the associated Node, not PyrexType.

I think inheritance could still be applied here.

da-woods · 2020-06-06T21:10:37Z

Cython/Compiler/ParseTreeTransforms.py

    def visit_CEnumDefNode(self, node):
        return self.visit_StatNode(node, True)

+    def visit_CppScopedEnumDefNode(self, node):


These are extra changes that could potentially go away if you were able to use the existing CEnumDefNode or inheritance.

da-woods · 2020-06-06T21:15:36Z

Cython/Utility/CppConvert.pyx

    tmp.imag = <double>z.imag()
    return tmp
+
+#################### enum_class.from_py ####################


This stuff looks simple enough that I wonder if it really needs utility code - why not just generate the C++ casts directly?

It's possible that the converted utility code is actually doing more type-checking etc than I'm giving it credit for, of course...

Apologies - what does "generate the C++ casts directly" mean? We basically need to be able to convert Python objects to and from underlying_type (which is only known at runtime). What's the best way to do that?

Looking at it again, it doesn't seem completely unreasonable - it obviously has to generate things like overflow-checks so the code is doing more than it initially looks.

However - here's what I'd try:

The conversions go object -> int_type -> enum and enum -> int_type -> object.
The object <-> int_type is taken from CIntLike so use their implementation. create_from_py_utility_code can become something like:

def create_from_py_utility_code(self, env): if self.from_py_function: return True if CIntLike.create_from_py_utility_code(self, env): self.from_py_function = 'static_cast<%s>(%s)' % ( self.cname, self.from_py_function) return False

i.e. all you've really done is wrapped the existing conversion in a cast.

It's possible I've missed something important here and it won't work, but I think it's worth a go.

Yes! Something like this works, except that for to_py_function, I needed to get clever and use C++ lambdas, to make a callable that correctly does the two chained casts:

def create_to_py_utility_code(self, env): if self.to_py_function is not None: return True if self.underlying_type.create_to_py_utility_code(env): self.to_py_function = '[](const %s& x){return %s((%s)x);}' % ( self.cname, self.underlying_type.to_py_function, self.underlying_type.empty_declaration_code() ) return True

Cython/Compiler/PyrexTypes.py

scoder

I agree with @da-woods that we should use less code on this. Enums and scoped enums do not seem different enough to have separate implementations. The parser can share most of the code, the declaration node can use inheritance, and the types should at least reuse code as much as possible, even if C++ enums are not integers (which, BTW, speaks against setting is_enum for them, because that refers to a C (integer) enum).

scoder · 2020-06-07T07:39:37Z

Cython/Compiler/Parsing.py

+        if s.peek()[1] in ('class', 'struct'):
+            return p_cpp_scoped_enum_definition(s, pos, ctx)
+        else:
+            return p_c_enum_definition(s, pos, ctx)


This would also benefit from a single entry point, instead of peeking ahead. You can see class and struct as modifiers here that change the final appearance of the enum, not the way it's parsed.

As recommended, I've made it such that we have a single parsing function

scoder · 2020-06-07T07:43:12Z

Cython/Utility/CppConvert.pyx

+    cdef enum class enumtype "{{type}}":
+        pass


Why is this needed? Can't we just use {{type}} as cast and return type below?

I struggled a lot with getting that to work. Below is a summary; perhaps I was missing something obvious:

The problem is essentially making {{type}} visible to the utility code. Looking at other examples in the code base, I figured that the outer_module_scope= arg here would take care of that:

env.use_utility_code(CythonUtilityCode.load( "enum_class.to_py", "CppConvert.pyx", context=context, outer_module_scope=env.global_scope(), # need access to types declared in module compiler_directives=env.directives ))

However, it fails when the type hasn't been cimport'd directly, but rather accessed with . notation:

from cheese cimport Cheese cdef Cheese d = int(1) # this worked fine

cimport cheese cdef cheese.Cheese d = int(1) # this didn't work

Then, I tried adding the type to the context so that it would be picked up by context_types:

context.update( { ... self.name: self # add `self` as a context_type } ) env.use_utility_code(CythonUtilityCode.load( "enum_class.to_py", "CppConvert.pyx", context=context, outer_module_scope=env.global_scope(), # need access to types declared in module compiler_directives=env.directives ))

That resulted in breaking the symbol table entry for the enum. This is where I struggled; I eventually traced it to this line, where the entry gets overwritten and loses its enum_values, breaking things downstream.

This is no longer relevant as I removed this utility code :)

…cython into add-scoped-enums-support

Cython/Compiler/Parsing.py

Co-authored-by: scoder <stefan_ml@behnel.de>

Cython/Compiler/Parsing.py

scoder · 2020-06-17T19:04:12Z

Cython/Compiler/Parsing.py

+        if scoped:
+            s.error("Unnamed scoped enum not allowed")
+
+    if s.sy == '(':


Suggested change

if s.sy == '(':

if scoped and s.sy == '(':

This seems worth another error test, BTW.

I just pushed bb544d5 which I think is what you are looking for, but please let me know if I misunderstood :-)

Co-authored-by: scoder <stefan_ml@behnel.de>

scoder · 2020-06-18T12:54:59Z

Thanks!

McSinyx · 2020-06-18T13:48:56Z

Thank you all for the incredible work on implementation and reviewing process! I'm so excited to try this out! 🚀 🥳 🎉

scoder · 2020-07-10T10:10:08Z

This code is generated for the cpdef_scoped_enums_import test. Is it correct?

    namespace Namespace {
        enum class Cheese {
            cheddar = 1,
            camembert = 2
        };
    }
[...]
static enum Namespace::Cheese __pyx_v_30dotted_import_scoped_enum_test_c;
static enum Namespace::Cheese __pyx_v_30dotted_import_scoped_enum_test_d;
[...]
  enum Namespace::Cheese __pyx_t_5;

Note how it says enum instead of enum class in the variable declarations.

I stumbled over this when looking at the appveyor build results because the Py3.4 builds are failing with these errors:

dotted_import_scoped_enum_test.cpp(1308): error C3431: 'Cheese': a scoped enumeration cannot be redeclared as an unscoped enumeration
dotted_import_scoped_enum_test.cpp(1309): error C3431: 'Cheese': a scoped enumeration cannot be redeclared as an unscoped enumeration
dotted_import_scoped_enum_test.cpp(1755): error C3431: 'Cheese': a scoped enumeration cannot be redeclared as an unscoped enumeration

They are using an old MSVC version which doesn't even recognise the -std=c++11 option, so the most likely reason is that it's simply handling this incorrectly, but I thought I'd ask, just to be sure that we can't improve this somehow. Newer MSVC versions seem to accept this code quite happily.

shwina · 2020-07-10T10:41:36Z

Yes, I believe that's correct. See
https://en.cppreference.com/w/cpp/language/elaborated_type_specifier

Specifically:

the enum keyword must be used to refer to an enumeration type (whether scoped or unscoped)

Edit: it also appears that it's an error to use enum class instead of enum when referring to a previously declared scoped enum.

That being said, it's possible we don't have to use the elaborated type specifier. I'll look into it.

shwina · 2020-07-10T16:21:02Z

@scoder it's a question of removing the "enum" from here: https://github.com/cython/cython/blob/master/Cython/Compiler/PyrexTypes.py#L4052

I can certainly put in a PR that will do that, but I guess it's worth bringing up that elaborated type specifiers are useful to disambiguate between types and non-types of the same name. I'm not sure whether this is a concern in Cython-generated code. Happy to do whatever you feel is best here.

shwina · 2020-07-19T11:36:53Z

@scoder Any thoughts here?

da-woods · 2020-07-20T18:01:44Z

My view would be either:

leave it (disable the test for Python 3.4+MSVC)
change enum (for this usage) -> __PYX_ENUM_DECL (or similar) which is defined as empty for Python 3.4+MSVC and enum for everything else.

Given that the elaborated type specifiers have the potential to be somewhat useful then I think it makes sense to keep them where possible.

shwina · 2020-07-20T22:20:14Z

Thanks @da-woods. Happy to look into using a __PYX_ENUM__DECL

@returns

* Fix test in Py2. * Check for exceptions also when @returns() is used, not only for "->" return type annotations. (cythonGH-3664) When you use Python type annotations, it would be weird if you lost Python exception propagation semantics along the way, just by compiling the code. So the default behaviour is "except? -1" here for C integer types. Arguably, this would also be a better default for the decorator case. See cython#3625 (comment) * Add tests that exception propagation works with the "@returns()" decorator. * Update changelog. * Allow selecting specific doctests in test modules with "-k pattern", instead of always running all tests. * Correct the positions reported for errors in f-strings. Closes cython#3674 * Fix f-string error positions in nogil test. * Clarify backwards incompatible special method change in changelog. * Fix many indentation and whitespace issues throughout the code base (cythonGH-3673) … and enforce them with pycodestyle. * Allow selecting specific doctests in test modules with "-k pattern", instead of always running all tests. * Prepare release of 0.29.20. * Update changelog. * Avoid calling PyUnicode_FromUnicode() in Py3. Closes cython#3677 * Update changelog. * Minor code simplification. * Always bind Cython functions * Update changelog. * Avoid an "unused variable" warning for code that gets compiled out in Py3. * Minor code simplification. * Revert "Always bind Cython functions" This reverts commit 6677326. * Update changelog. * Revert "Limited API updates and cleanup for cython#2056. cythonGH-3635)" This reverts commit 02bb311. * Revert "Invoke binop super method via direct slot access." This reverts commit bcb9387. * Revert "Add support for pow operator." This reverts commit d849fb2. * Revert "Python-style binary operation methods." This reverts commit e6a8124. * Re-add "c_api_binop_methods" directive for backwards compatibility after reverting cython#3633 and force it to "False". Closes cython#3688. * Change the default of the "c_api_binop_methods" directive to False. (cythonGH-3644) This is a backwards incompatible change that enables Python semantics for special methods by default, while the directive allows users to opt out of it and go back to the previous C-ish Cython semantics. See cython#2056 * Always bind Cython functions in Py3 (cythonGH-3683) Rebased 1bb26b9 for cython 0.29.x, and made conditional to Python 3. This does not solve the original staticmethod problem in Python 2 but it does resolve classmethod problems in Python 3.9. Therefore, it unbreaks other packages on Python 3.9, improves overall state for Python 3 and does not introduce regressions in Python 2. * Limit the scope of a local variable in a helper function. * Avoid a C compiler warning about a constant condition. * Remove dead code. * Update changelog. * Add support for C++ scoped enums with "enum class" and "enum struct" (cythonGH-3640) Closes cython#1603. * Add missing "PyUnicode_GET_LENGTH" to unicode.pxd (cythonGH-3692) * Add missing "PyUnicode_GET_LENGTH" to unicode.pxd (cythonGH-3692) * Fix usage of deprecated Py_UNICODE API. * Add safety fix to avoid reading a character from the empty string. * Fix prefix of internal function name. * Fix definition of "__Pyx_CyFunction_USED", which should only be #defined and not have a value. * Avoid using "tp_name" when CYTHON_COMPILING_IN_LIMITED_API (cythonGH-3693) * exec() did not allow recent Python syntax features in Py3.8+ due to https://bugs.python.org/issue35975 Closes cython#3695 * Avoid using the "tp_iternext" slot when CYTHON_USE_TYPE_SLOTS is disabled. * Give the "__Pyx_PyObject_GetIterNext" helper macro a more explanatory name. * Avoid unused variable in PyPy etc. * Avoid a call to PyTuple_GET_ITEM() to get the item array pointer if CYTHON_ASSUME_SAFE_MACROS is disabled. See cython#3701 * Enable travis for all branches. * Fix indentation counter for module init function. * Fix FunctionState handling for module cleanup function. * Validate that all temps were correctly released at the end of a function. * Keep reference to module dict around also in limited mode (cythonGH-3707) `PyModule_GetDict` is part of the limited API so we can keep a reference to the module dict around regardless of limited mode being enabled or not. * Validate and fix temp releasing (cythonGH-3708) * Fix a temp leak in the type init code. * Fix temp leaks in fused types initialisation code. * Correctly release the buffer index temps allocated for index calculations. * Make tests fails hard if a temp variable is not released at the end of a generated function. * Fix temp leak in switch statement code. * Make end-to-end tests fail on refnanny output. * Fix result temp leak in PyTypeTestNode. * Fix result temp leak in external type/function import code and enable the refnanny check for them. * Fix temp leak when try-return-finally is used in generators. * Make it explicit when an allocated temp is not meant to be reused. * Fix temp leak when assigning to the real/imag attributes of complex numbers. * Fix temp leak when assigning to a memoryview slice. * Clean up "num_threads" result temp in parallel section, not only in prange loop. * Fix temp leak in Pythran buffer setitem code. * Simplify NumPyMethodCallNode since it does not need the Python function anymore. Previously, it generated code that needlessly looked up the Python function without actually using it. * Fix temp leak when deleting C++ objects. * Add a test that actually reusing temps when deleting C++ objects works correctly. * Test runner: disable keep-alive output in --debug mode to keep a potential pdb console clean. * Fix argument name usage in finally blocks (cythonGH-3713) Fixes cython#3712 (hopefully) by reverting a small part of bbef4d7 * Fix argument name usage in finally blocks (cythonGH-3713) Fixes cython#3712 (hopefully) by reverting a small part of bbef4d7 * Document version-tagged pxd files (cythonGH-3587) * Implement generic optimized loop iterator with indexing and type inference for memoryviews (cythonGH-3617) * Adds bytearray iteration since that was not previously optimised (because it allows changing length during iteration). * Always set `entry.init` for memoryviewslice. * Update changelog. * Make end-to-end tests fail on refnanny output. * Fix FunctionState handling for module cleanup function. * Update change log. * Really only use PyUnicode_FromUnicode() when needed (cythonGH-3697) * Really only use PyUnicode_FromUnicode() for strings that contain lone surrogate, not for normal non-BMP strings and not for surrogate pairs on 16bit Unicode platforms. See cython#3678 * Extend buildenv test to debug a MacOS problem. * Add a test for surrogate pairs in Unicode strings. * Limit PyUnicode_FromUnicode() usage to strings containing lone surrogates. * Accept ambiguity of surrogate pairs in Unicode string literals when generated on 16bit Py2 systems. * Disable testing against NumPy 1.19+ in the 0.29.x branch, which removed C-API features. * Disable testing against NumPy 1.19.0 in the 0.29.x branch, which breaks a C-API call. * Validate and fix temp releasing (cythonGH-3708) (cythonGH-3717) * Validate and fix temp releasing (cythonGH-3708) Backports 92147ba. * Fix a temp leak in the type init code. * Fix temp leaks in fused types initialisation code. * Correctly release the buffer index temps allocated for index calculations. * Make tests fails hard if a temp variable is not released at the end of a generated function. * Fix temp leak in switch statement code. * Make end-to-end tests fail on refnanny output. * Fix result temp leak in PyTypeTestNode. * Fix result temp leak in external type/function import code and enable the refnanny check for them. * Fix temp leak when try-return-finally is used in generators. * Make it explicit when an allocated temp is not meant to be reused. * Fix temp leak when assigning to the real/imag attributes of complex numbers. * Fix temp leak when assigning to a memoryview slice. * Clean up "num_threads" result temp in parallel section, not only in prange loop. * Fix temp leak in Pythran buffer setitem code. * Simplify NumPyMethodCallNode since it does not need the Python function anymore. Previously, it generated code that needlessly looked up the Python function without actually using it. * Fix temp leak when deleting C++ objects. * Add a test that actually reusing temps when deleting C++ objects works correctly. * Improve syntax feature support of Cython CodeWriter (cythonGH-3514) * Disable Py_UNICODE fallback for __Pyx_UnicodeContainsUCS4() in Py3.9 since Py_UNICODE is deprecated and PEP-393 unicode is practically required. * Avoid calling PyUnicode_FromUnicode() in Py3 (cythonGH-3721) See cython#3677 * Add missing name substitution in __Pyx_PyUnicode_Substring() utility code. * Update changelog. * Prepare release of 0.29.21. * Really only use PyUnicode_FromUnicode() when needed (cythonGH-3697) * Really only use PyUnicode_FromUnicode() for strings that contain lone surrogate, not for normal non-BMP strings and not for surrogate pairs on 16bit Unicode platforms. See cython#3678 * Extend buildenv test to debug a MacOS problem. * Add a test for surrogate pairs in Unicode strings. * Limit PyUnicode_FromUnicode() usage to strings containing lone surrogates. * Accept ambiguity of surrogate pairs in Unicode string literals when generated on 16bit Py2 systems. * Disable Py_UNICODE fallback for __Pyx_UnicodeContainsUCS4() in Py3.9 since Py_UNICODE is deprecated and PEP-393 unicode is practically required. * Fix test in 16-bit Unicode deployments. * Update changelog. * Always consider 0-sized arrays as C- and F-contiguous (cythonGH-3728) Fixes cython#2093 * Always consider 0-sized arrays as C- and F-contiguous (cythonGH-3728) Fixes cython#2093 * Update changelog. * Add missing unordered_map template defaults (cythonGH-3686) * Add missing unordered_map template defaults (cythonGH-3686) * Update changelog. * Improve test output in case of failures. * Using Py_UNICODE to store lone surrogates makes Py3 join surrogate pairs on 16-bit Unicode platforms (Windows) when reading them back in, although we correctly processed them before. Instead, we now use the "unicode_escape" codec to store byte strings, because it can return surrogate characters (which the other codecs cannot). * Disable test in Py2.6. * Update changelog. * Use the more appropriate CYTHON_USE_TYPE_SLOTS guard for accessing the binop number slot. * Fix typo in error message. * Update the documentation on the arithmetic special methods and issue a "backwards compatibility" warning when the reversed method is not implemented. See cython#2056 * Add the new "c_api_binop_methods" directive to the documentation. * PyPy does not support PyType_GetSlot(). Use type slots instead. * Reformat doc paragraph. * Document C inline properties. * Beautify example output. * Use inline properties on the "PyComplex" builtin type declared in "cpython.complex" to provide C level access to the "real" and "imag" attributes (which Cython provides anyway for the 'undeclared' builtin type). * Prevent compile error when the result of repr() is assigned to a "unicode" LHS with language_level=3. Closes cython#3736 * Handle `for x in cpp_function_call()` (cythonGH-3667) Fixes cython#3663 This ensures that rvalues here are saved as temps, while keeping the existing behaviour for `for x in deref(vec)`, where the pointer for vec is copied, meaning it doesn't crash if vec is reassigned. The bit of this change liable to have the biggest effect is that I've changed the result type of dereference(x) and x[0] (where x is a c++ type) to a reference rather than value type. I think this is OK because it matches what C++ does. If that isn't a sensible change then I can probably inspect the loop sequence more closely to try to detect this. * Clarify changelog entry on ways to deal with the incompatible binop method change. * Remove Google Analytics script from documentation to avoid tracking our users. * Add donation banner to documentation. * Create bug template * Update issue templates * Delete unused custom ticket template * Add handshake emoji to donations banner to make it more visible. * Don't create CReference in C (only c++) (cythonGH-3746) * Fixed reference types being passed to getitemint (cythonGH-3755) * Reorder test module to restore the "invalid - valid" order. * Update changelog. * Prepare release of 3.0a6. * Add test comments on how "memslice.pyx" and "memoryview.pyx" relate. * Print test dependency versions to help with test failure debugging. * Add CI builds for different CPU architectures on travis. * Update issue templates * Add type "Py_hash_t" in pure Python mode. * Update changelog. * Clarify changelog entry. * Remove dead code and dead comments from "numpy/__init__.pxd". * Change "Py_intptr_t" declaration in numpy.pxd to what CPython uses as fallback. "int" seems overly lazy if it tends to be larger on many systems. * Readability improvements in "numpy.pxd". * Add a comment that numpy.pxd is maintained by the NumPy project. * Try to fix NumPy test failures by not setting the "NPY_NO_DEPRECATED_API" #define for NumPy 1.19.[01]. * Use NumPy 1.18.x for testing on travis as long as 1.19.[01+] ships a numpy.pxd that is incompatible with Cython 3.0. * Remove useless "extern" modifiers from cdef classes declared in "extern" sections. * Do not depend on the default type inference in "cpython/array.pxd". * Avoid unused C variable warning by moving declaration and usage under the same condition. Closes cython#3763 * Use a generic shape accessor for pythranized array (cythonGH-3762) This is a follow up to cython#3307 * Silence a C compiler warning about a constant condition. * Remove an unused function that had been added for the now-deleted "__getbuffer__" implementation. * Support PEP-560 ("__class_getitem__") for extension classes (cythonGH-3765) * Add "make checks" target to run code checks. * In bug template, ask for Python version in addition to Cython version * Support simple, non-strided views of "cython.array". Closes cython#3775 * Remove unused cimports. * Update changelog. * Set PYTHONHOME in embedding test to fix compilation issues in Py3.8/macOS. * Fix unrelated test after changing MemoryView.pyx. * Revert "Set PYTHONHOME in embedding test to fix compilation issues in Py3.8/macOS." This reverts commit a333d6a. The change did not resolve the test issue in travis. * Fix Python 3.4 + MSVC issue with elaborated type specifier for enum (cythonGH-3782) * Add more cimport_from_pyx tests (cythonGH-3786) There's now a fairly wide range of valid syntax for declaring things in pyx files and it should all be supported when cimporting from them. * Split known types into separate lines to let them have their own VCS history. * Add a note on PayPal fees for small payments. * Avoid merged-in code picking up directives from main module (cythonGH-3785) Fixes cython#1071 * Fix cygdb (cythonGH-3542) * Cython debugger documentation: Added link to an installation script. * Got a new libpython.py from the cpython source distribution. * Default language level in tests is now 3 instead of 2 * Migrated codefile from python 2 to python 3. * Added testcase for the cy list command in cygdb. * Temporarily removing test case that freezes gdb. * Fixed a bug that broke several Cygdb tests. The cython_debug/cython_debug_info_* files map the names of the C-functions generated by the Cython compiler to the names of the functions in the *.pyx source. If the function was defined using "def" (and not "cpdef" or "cdef") in the *.pyx source file, the C-function named in cython_debug/cython_debug_info_* used to be __pyx_pw_*, which is the name of the wrapper function and now it is __pyx_f_*, which is the name of the actual function. This makes some Cygdb tests pass that did not pass before. * Better error messages: If a cygdb command raises, a traceback will be printed. * Fixed a bug in cygdb. The following now works: 1. Start cygdb 2. Type "cy exec" and hit enter 3. Type some other lines 4. Type "end" and hit enter. -> These "other lines" will get executed * Fixed a bug in cygdb: cy list now works outside of functions. * Added print_hr_allmarkers function for easier debugging. * Fixed a bug that broke cygdb: cy break did not work if you put the breakpoint outside of a function if there was e.g. the following somewhere in your *.pyx file: cdef class SomeClass(): pass * Added a Cygdb test for printing global variables. * Fixing cygdb: Replaced cy print with a simple, working solution. * If an exception in Cygdb occurs, a stacktrace will be printed. * Fixed a bug that broke cy break -p * Bugfix: The compiler now writes out correctly which cython linenumber and path corresponds to which c linenumber. * Set language_level=2 in runtests.py * Update changelog. * Add a "gdb" test tag that depends on being able to run gdb. * Update CPython "test_fstring" copy to Py3.9. * Improved documentation for annotation typing (cythonGH-3799) Mainly by moving it to a separate section to make it easier to find, however also added a small amount of extra information about some of the obvious limitations. * Fix `return None` in CodeWriter. (cythonGH-3795) * Rename "GCC_DIAGNOSTIC" macro to make it Cython specific and make it available to other utility code sections. * Add a test for unpacking large PyLong values. * Silence gcc diagnostics whenever -1 is cast to something user provided. (cythonGH-3803) Fixes cythonGH-2749. * Define extern `PyBUF_MAX_NDIM` (cythonGH-3811) Ensure that Cython exposes `PyBUF_MAX_NDIM` from Python as part of `cpython.buffer` to allow access to developers. * Change test to use only integer calculations to prevent platform specific rounding issues, while keeping a reasonable distribution of integers across the PyLong digit ranges. * Created a glossary and added one new entry (cythonGH-3810) * Call destructors for structs in C++ (cythonGH-3228) In C++ mode, structs can contain C++ classes. Therefore structs should have their destructors called to ensure any class contained is also destructed. Also, a bit more thorough about ensuring constructor is generated where necessary. Closes cythonGH-3226. * Rename "GCC_DIAGNOSTIC" macro to make it Cython specific and make it available to other utility code sections. * Silence gcc diagnostics whenever -1 is cast to something user provided. (cythonGH-3803) Fixes cythonGH-2749. * Allow cast to ctuple (cythonGH-3812) * Make asyncio.iscoroutinefunction() recognise Cython compiled coroutines. (cythonGH-3427) Python's asyncio.coroutines uses an object to tag objects as coroutine functions. We now read this object and use it to tag Cython compiled coroutines as well. It also includes tests to make sure `asyncio.iscoroutinefunction()` works as expected. This doesn't fix `inspect.iscouroutinefunction()` (which uses a flag that can trigger undesirable behaviour for cython functions). Closes cython#2273 * Do not cover lines that were excluded in the coveragerc config file (cythonGH-3682) Closes cython#3680. * Add doc support for cpdef enum (cythonGH-3813) * Update changelog. * Add "check_size ignore" to all NumPy.pxd class declarations to silence the useless size warnings about them. See numpy/numpy#432 (comment) Also remove the useless "extern" modifiers from cdef classes declared in "extern" sections. * Update changelog. * Update changelog. * Support utility code in headers (cythonGH-3779) Specifically this allows public packed structs but may also allow other public declarations that require small amounts of utility code. * Prevent overflowing hash values for "Machines.Node" due to arbitrarily large unsigned pointer values. Closes cython#3840 * Extract "error_type" handling from the type creation functions. * Remember in the AnnotationsWriter when a serialised expression contained unknown/unsupported nodes. * Do not include "u" string prefix in annotations since tools that process them probably expect Py3 string semantics anyway. * Keep AnnotationNode instead of the bare annotation expression in "entry.annotation" to get access to the string representation. * Fix test after removing the 'u' prefix from unicode string annotation values. * Add support for PEP 526 `__annotations__` in class body. (cythonGH-3829) Closes cython#2552 * Some more glossary entries (cythonGH-3836) * Rename test file to avoid ambiguity with the C "const" modifier. * Join '*' and '**' parsing in declarators to avoid differences for 'const' parsing etc. * Set language level in Cython's speed-up .pxd files since it no longer depends on the .py files that they correspond to. * Declare "scanner.sy" as "unicode" string to optimise its usage. * Use more recent C-API functions on tear-down of the embedding code. * Improve some wordings in README.rst (cythonGH-3852) * Restore Py2 compatibility in test. * Replace useless comment. * Fixed assorted crashes with fused types (cythonGH-3853) Show a more detailed error for unspecializable fused return types. * Add a more common and more versatile example to the Verbatim C-Code documentation. * Fix RST typo. * Make C code C89 again. * Add support for forwarding references (cythonGH-3821) See, for example, https://isocpp.org/blog/2012/11/universal-references-in-c11-scott-meyers Closes cython#3814 * Avoid C compiler warnings about unused variables in test. * Avoid C compiler warning about unused variable in test. * Test Py3.7 and Py3.8 in C and C++ mode under appveyor. Run both in the same job since the machines are quite fast with parallel processes, but setting one up is slow. * Fix test compile failure in MSVC. * Allow creation of wrappers for cdef functions with memoryviews (cythonGH-3856) Fixes cython#3843 * Clean up and test type identifier escaping. - hash() hashing lead to unpredictable random prefixes for long names across multiple runs - use a single regex run instead of repeated calls to replace() * Move a memoryview test out of the "run" test directory since memoryview tests tend to be slow and have their own test directory. * Fix test compile failure in MSVC. * Clarify the section on exception return values. * Be a bit more paranoid about macro usage in the refnanny code. * Looks like the usual macro on Windows is "_WIN32" and not "WIN32". Let's support both, to be on the safe side. * Try to fix platform specific test once more. "synchapi.h" is not supposed to be included directly, and "windows.h" can break ... other stuff. Let's see what we can do. * Fix ReST typo. * docs: fix typos, minor clarification. * Minor docs clarification on error return values. (cythonGH-3859) * Replace deprecated Element.getiterator() with Element.iter(). (cythonGH-3864) `xml.etree.ElementTree.Element.getiterator()` was deprecated in Python 2.7 & 3.2 and removed in the freshly released Python 3.9. * Prefer Element.iter() over the deprecated Element.getiterator(). (cythonGH-3865) `xml.etree.ElementTree.Element.getiterator()` was deprecated in Python 2.7 & 3.2 and removed in Python 3.9. * Split a combined assert in two to avoid an unused C temp variable when assertions are disabled. (cythonGH-3870) * Split a combined assert in two to avoid an unused C temp variable when assertions are disabled. (cythonGH-3870) * Disable a test that fails in Py2 in 0.29.x since 'str' is 'unicode' in cython3, also in Python 2. * Rename test file to avoid ambiguity with the C "const" modifier. * Use more recent C-API functions on tear-down of the embedding code. * Make type identifier escaping deterministic: hash() hashing lead to unpredictable random prefixes for long names across multiple Python runs. * Clarify the section on exception return values. * Fix ReST typo. * docs: fix typos, minor clarification. * Minor docs clarification on error return values. (cythonGH-3859) * Add Python 3.9 to appveyor build. * Resolve merge conflict. * Removed `same_as` methods from Plex.Actions.Action (cythonGH-3847) It seems to be unused and it looked like the existing implementation was faulty. * Fix test after changing the hash method used for escaping long type descriptions. * Add warning for common user mistake and fix crash relating to annotated attributes. (cythonGH-3850) Closes cython#3830 * Improve error reporting when users mistakenly write "&&" or "||" instead of Python's "and" and "or" operators. (cythonGH-3858) * Disable the embedding test on MacOS-X to get the builds green again. Too many PRs depend on the travis tests to give a green light. * Simplify the output stream capturing for the C compiler runs by using a temp file instead of threads. * Help reporters see the first paragraph of the bug report template. * Detect _MSC_VER for __PYX_STD_MOVE_IF_SUPPORTED to support MSVC support even when "__cplusplus" is not set appropriately (cythonGH-3792) * Fix memoryview casts involving fused types (cythonGH-3882) I think this approach is more satisfactory than the old way it used to "work", where "fused_to_specific" was permanently added to the module scope containing the fused type (in this case the Cython scope), was applied in "Scope.lookup_type", but continued to have an effect on the scope forever. Closes cython#3881 * Add O_DIRECT to posix/fcntl.pxd (cythonGH-3894) Closes cython#3242 Co-authored-by: Stefan Behnel <stefan_ml@behnel.de> Co-authored-by: Jeroen Demeyer <jeroen.k.demeyer@gmail.com> Co-authored-by: Robert Bradshaw <robertwb@gmail.com> Co-authored-by: Michał Górny <mgorny@gentoo.org> Co-authored-by: Ashwin Srinath <3190405+shwina@users.noreply.github.com> Co-authored-by: Thomas A Caswell <tcaswell@gmail.com> Co-authored-by: Matthias Braun <matze@braunis.de> Co-authored-by: da-woods <dw-git@d-woods.co.uk> Co-authored-by: Tao He <sighingnow@gmail.com> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Clemens <chofreither@gmail.com> Co-authored-by: will <timwilloney@gmail.com> Co-authored-by: serge-sans-paille <serge.guelton@telecom-bretagne.eu> Co-authored-by: Yuan <pyslvs@gmail.com> Co-authored-by: Volker-Weissmann <39418860+Volker-Weissmann@users.noreply.github.com> Co-authored-by: Tao He <linzhu.ht@alibaba-inc.com> Co-authored-by: cf-natali <cf.natali@gmail.com> Co-authored-by: jakirkham <jakirkham@gmail.com> Co-authored-by: Tasha "Ren" Chin <tasha.s.chin@gmail.com> Co-authored-by: Pedro Marques da Luz <pmdaluz@gmail.com> Co-authored-by: Mathias Laurin <Mathias.Laurin+github.com@gmail.com> Co-authored-by: matham <moiein2000@gmail.com> Co-authored-by: Sean <stewart.d.sean@gmail.com> Co-authored-by: Rajat Dash <49777622+DARKPROGRAMMER16@users.noreply.github.com> Co-authored-by: ptype <ptype@users.noreply.github.com> Co-authored-by: Nick Pope <nick.pope@flightdataservices.com> Co-authored-by: Jeppe Dakin <jeppe_dakin@hotmail.com> Co-authored-by: Zackery Spytz <zspytz@gmail.com>

Add initial support for C++ scoped enums

4ad92d4

shwina marked this pull request as ready for review May 26, 2020 17:26

scoder reviewed May 26, 2020

View reviewed changes

shwina commented May 26, 2020

View reviewed changes

tests/run/cpp_scoped_enums.pyx Outdated Show resolved Hide resolved

shwina commented May 27, 2020

View reviewed changes

Cython/Compiler/PyrexTypes.py Outdated Show resolved Hide resolved

shwina added 16 commits May 28, 2020 15:43

Use enum class instead of enumclass to define scoped ennums

849ea46

Add c++11 tag/compile options to enum class tests

bdb51da

Don't pass context types to enum class utilities

2c5383e

Add support for specifying underlying_type of an enum class

4182f27

Remove stale code

e9c6335

Fix scoped enum generation to include underlying type

bd77e63

Move check for empty scoped enum to parsing stage

9bbf964

Fix cname handling for scoped enum items

9051c7c

Merge branch 'add-scoped-enums-support' of https://github.com/shwina/…

d0c6df2

…cython into add-scoped-enums-support

Disallow unnamed scoped enums

72e57d5

remove duplicate file

38a323d

Fix bug which skipped last enum entry in scoped enum

ea65898

Added more tests for scoped enums

14525a5

Rename to CppScopedEnumType

53575c7

Check first if existing entry is a scoped_enum

79fde22

Document scoped enumerations

c8a96af

shwina changed the title ~~[WIP] Add support for C++ scoped enums~~ Add support for C++ scoped enums Jun 6, 2020

shwina added 5 commits June 6, 2020 15:52

Merge branch 'master' of https://github.com/cython/cython into add-sc…

9d4feda

…oped-enums-support

Replace assertions with return values in tests

ec891f3

Rename ScopedEnum -> CppScopedEnum

67d9e40

Better names for scoped enum utilities

b6a85ba

Try a fix for Py2.7 Cpdef enums

546befc

da-woods reviewed Jun 6, 2020

View reviewed changes

Fix Py2.7 scoped enum utility

2f823ea

scoder reviewed Jun 7, 2020

View reviewed changes

Apply suggestions from code review

e8a99c2

scoder added this to the 3.0 milestone Jun 17, 2020

scoder added C++ Cython Language Feature feature labels Jun 17, 2020

shwina added 3 commits June 17, 2020 13:19

Fix parsing for enum struct and add a test for it

13fc586

Rename base_code -> type_name for clarity

debf520

Merge branch 'add-scoped-enums-support' of https://github.com/shwina/…

871483c

…cython into add-scoped-enums-support

scoder reviewed Jun 17, 2020

View reviewed changes

Cython/Compiler/Parsing.py Outdated Show resolved Hide resolved

Update Cython/Compiler/Parsing.py

765b8b7

Co-authored-by: scoder <stefan_ml@behnel.de>

scoder reviewed Jun 17, 2020

View reviewed changes

shwina and others added 5 commits June 17, 2020 16:10

Update Cython/Compiler/Parsing.py

1ac8a2a

Co-authored-by: scoder <stefan_ml@behnel.de>

Update Cython/Compiler/Parsing.py

a1c4148

Co-authored-by: scoder <stefan_ml@behnel.de>

Test that we reject specifying the underlying type of a C-style enum

bb544d5

Restrict C++ enum error test to C++ mode.

cc1ec50

Fix expected line numbers in error test

57875f3

scoder merged commit 2c7c22f into cython:master Jun 18, 2020

shwina mentioned this pull request Aug 13, 2020

Fix Python 3.4 + MSVC issue with elaborated type specifier for enum #3782

Merged

da-woods mentioned this pull request Sep 1, 2020

Cython 0.29 broke support for C++ enum classes #2749

Closed

bdice mentioned this pull request Oct 27, 2020

Enable scikit-build on Windows (without static members) glotzerlab/freud#676

Merged

11 tasks

jpfeuffer mentioned this pull request Apr 14, 2021

Silence pyopenms redeclaration warnings? OpenMS/OpenMS#4505

Closed

		if not enum_values:
		error(entry.pos, "Empty enum definition not allowed outside a 'cdef extern from' block")

		enum_entry.type.values.append(entry.name)


		class CppScopedEnumDefNode(StatNode):

Uh oh!

Conversation

shwina commented May 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scoder left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shwina Jun 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shwina Jun 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

scoder left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shwina Jun 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shwina commented May 26, 2020 •

edited

Loading

shwina Jun 6, 2020 •

edited

Loading

shwina Jun 9, 2020 •

edited

Loading

shwina Jun 7, 2020 •

edited

Loading

shwina Jun 17, 2020 •

edited

Loading

shwina commented Jul 10, 2020 •

edited

Loading

da-woods commented Jul 20, 2020 •

edited

Loading