[RFC] Add support for forwarding references by shwina · Pull Request #3821 · cython/cython

shwina · 2020-09-09T14:06:18Z

Adds support for C++11 forwarding references. Please see my comment here for more details about the scope of this PR.

Would greatly appreciate input on the high level approach here -- I hope the changes aren't too intrusive.

Closes #3814.

shwina · 2020-09-09T14:07:33Z

Cython/Compiler/Lexicon.py

    diphthong = Str("==", "<>", "!=", "<=", ">=", "<<", ">>", "**", "//",
                    "+=", "-=", "*=", "/=", "%=", "|=", "^=", "&=",
-                    "<<=", ">>=", "**=", "//=", "->", "@=")
+                    "<<=", ">>=", "**=", "//=", "->", "@=", "&&")


Is this OK? I was afraid that this would be too intrusive, but it might become necessary if Cython wants to support rvalue references more broadly going forward

Probably ok, although it might have an impact on the error reporting if users accidentally write a && b instead of a and b in a C-ish context (certainly happened to me more than once). Might be worth special-casing that in the parser to really give a good error message. The same applies to || then. Doesn't have to be part of this PR, though.
Note that this wouldn't fit into the error test that you wrote. Should be more something like errors/c_mistakes.pyx or so.

I created a PR for the error reporting in #3858.

shwina · 2020-09-09T14:08:41Z

Cython/Compiler/Nodes.py

        return self.base.analyse(ref_type, env, nonempty=nonempty, visibility=visibility, in_pxd=in_pxd)


+class CppRvalueReferenceDeclaratorNode(CReferenceDeclaratorNode):


I'm using inheritance here purely for code reuse. An rvalue reference does not have an "is-a" relationship with lvalue reference. Open to other suggestions here, and ditto with CppRvalueReferenceType/CReferenceType.

The main issue would be if there was a visit_CReferenceDeclaratorNode in a transform somewhere and your new node accidentally got caught by it. The other way would be to move the implementation to a common base class that's shared between CppRvalueReferenceType and CReferenceType.

A common base class sounds good. I'll go with that

shwina · 2020-09-13T13:30:40Z

cc @da-woods if you could look at this when you have a chance :)

da-woods

Right now it looks fairly simple to me and I can't see any huge problems. There's definitely some stuff that isn't tested though.

The main thing that occurs to me is what happens when you return a forward reference (like std::move)? Suppose you have two functions:

int&& move_but_sometimes_raise_a_cpp_exception(int&&) except +
int&& move_but_sometimes_raise_a_python_exception(int&&) except *

In both cases we'd currently take a temporary while handling the exception. For regular references this would have the type CFakeReferenceType<int> (CFakeReferenceType actually holds a pointer which is assumed to remain valid for its lifetime...). For rvalue references I think the temporary should be of type int (i.e. it's initialized using operator=(int&&) if possible, but doesn't rely on a pointer to a temporary remaining valid).

The other thing that occurs to me is "what should happen on an older C++ compiler?" Right now we're fairly careful about generating c++98 code by default. Are there any cases where this could generate c++11-only code from something that works fine now. (I can't think of any right now, but we should be careful)

da-woods · 2020-09-13T13:52:19Z

Cython/Compiler/Nodes.py

        return self.base.analyse(ref_type, env, nonempty=nonempty, visibility=visibility, in_pxd=in_pxd)


+class CppRvalueReferenceDeclaratorNode(CReferenceDeclaratorNode):


The main issue would be if there was a visit_CReferenceDeclaratorNode in a transform somewhere and your new node accidentally got caught by it. The other way would be to move the implementation to a common base class that's shared between CppRvalueReferenceType and CReferenceType.

da-woods · 2020-09-13T13:57:35Z

Cython/Compiler/ExprNodes.py

+        # function call argument is an lvalue. See:
+        # https://en.cppreference.com/w/cpp/language/template_argument_deduction#Deduction_from_a_function_call
+        arg_types = [arg.type for arg in args]
+        if func_type.is_cfunction:


This section looks to happen on both template and non-template functions? That makes me wonder if it should be in the code that deals specifically with template functions? (I could be wrong about this though).

I moved it down to what I think is the part that deals only with templates. Does that look right?

tests/run/cpp_forwarding_ref.pyx

da-woods · 2020-09-13T14:39:13Z

One further variation on the above:

cdef extern from "whatever.hpp":
    cdef cppclass C:
        pass

   C get_C() except +  # can throw exception
   void accept_C(C&&)

accept_C(get_C())

This'll get translated into something like:

try {
   temp = get_C();
} except (...) {
   // handle exception
}
accept_C(temp);

which I don't think will work. We'll therefore need to move the temp. Hopefully this'll get caught by the __PYX_MOVE_IF_SUPPORTED mechanism, but it'll need checking (probably in https://github.com/cython/cython/blob/master/tests/compile/cpp_temp_assignment.pyx)

shwina · 2020-09-16T13:36:07Z

which I don't think will work. We'll therefore need to move the temp. Hopefully this'll get caught by the __PYX_MOVE_IF_SUPPORTED mechanism, but it'll need checking (probably in https://github.com/cython/cython/blob/master/tests/compile/cpp_temp_assignment.pyx)

@da-woods Indeed I think it does get caught by __PYX_MOVE_IF_SUPPORTED. However, it's not just C++ classes that need to be handled this way, but also fundamental types (right?). For example, this won't compile currently:

cdef extern from "whatever.hpp":
   int get_int() except +  # can throw exception
   void accept[T](T&&)

accept(get_int())

because it gets translated to:

try {
   temp = get_int();
} except (...) {
   // handle exception
}
accept<int>(temp);  // needs to be accept<int>(__PYX_MOVE_IF_SUPPORTED(temp));

for which, we'll need to remove the restriction is_cpp_class in https://github.com/cython/cython/blob/master/Cython/Compiler/ExprNodes.py#L505. Would that be too unreasonable?

FWIW, this problem isn't unique to forwarding references and will show up when adding support for rvalue references as well.

da-woods · 2020-09-16T16:18:01Z

we'll need to remove the restriction is_cpp_class in https://github.com/cython/cython/blob/master/Cython/Compiler/ExprNodes.py#L505. Would that be too unreasonable?

You may be over-estimating my importance in the decision-making process. However:

I think it's a reasonable change - makes the generated code slightly less readable, but it doesn't really cost anything to try to move a simple type. The only thing is that you may need to add a check to see if we're in c++ mode. I may have assumed that this was true if is_cpp_class is there.

The other option would be to do something like if type.is_cpp_class or target_type.is_rvalue_reference:. But it's probably just easier to try to move everything in C++ mode.

shwina · 2020-09-16T19:06:10Z

The only thing is that you may need to add a check to see if we're in c++ mode.

I'm struggling to find an easy way to do this -- any pointers?

da-woods · 2020-09-16T19:11:24Z

I'm struggling to find an easy way to do this -- any pointers?

It's something like env.context.cpp I think.

scoder · 2020-09-30T15:00:31Z

Ah - if you go up as far as generate_result_code or similar then you'll be able to get it from the code argument.

Not necessarily. There are also usages in calculate_result_code(), which doesn't know the env … nor the code object.

code.globalstate.module_node.scope.context.cpp looks like it should work. I'm sure there is probably a better way though.

I've been wondering for a while if the CCodeWriter shouldn't simply know whether it's generating C or C++ code. I wouldn't mind telling it (or allowing it to find out indirectly, as with your expression above). But that also won't help here since calculate_result_code() still wouldn't know it.

The Way That Always Works is to use yet another C macro, based on __cplusplus this time. Because – why not.

shwina · 2020-09-30T15:23:22Z

The Way That Always Works is to use yet another C macro, based on __cplusplus this time. Because – why not.

Isn't this essentially what __PYX_STD_MOVE_IF_SUPPORTED is?

scoder · 2020-09-30T15:37:47Z

If you do remove it then check that we don't generate __PYX_MOVE_IF_SUPPORTED in C mode. It'd probably be harmless (since it wouldn't be supported) but I don't think it should be there.

I think it's ok to have it. We have lots of C macros that do something useful in some cases and nothing at all in others. That's one of their main use cases. Let's just use it here.

…into add-forwarding-references

shwina · 2020-09-30T15:50:53Z

OK, so I made it such that we use __PYX_STD_MOVE_IF_SUPPORTED whether in C or C++ mode, with a note that it would be nice to restrict to C++ mode.

shwina · 2020-09-30T16:20:29Z

Please feel free to let me know if the following should be discussed in a separate GitHub issue:

Tangentially, the changes in this PR allow the Cython definition of std::move in libcpp.utility to be greatly simplified to just this:

cdef extern from "<utility>" namespace "std":
    cdef T&& move[T](T&&)

However, this will require the following change to the test cpp_move.pyx:

cdef extern from *:
    """
    #include <string>
    template<typename T> const char* move_helper(T&) { return "lvalue-ref"; }
    template<typename T> const char* move_helper(T&&) { return "rvalue-ref"; }
    """
    const char* move_helper[T](T&&)

shwina · 2020-09-30T20:31:38Z

Hmm, always using __PYX_STD_MOVE_IF_AVAILABLE doesn't work either. In a macro, like this for example:

#define _get_addr(_x, _idx) (&_x+_idx)

The following fails in C++ mode:

_get_addr(__PYX_STD_MOVE...(x), __PYX_STD_MOVE..(y));

..since you can't take the address of an rvalue. std::move(x) effectively makes x an rvalue.

da-woods · 2020-09-30T21:12:48Z

My impression was that the logic in 51c4208 was right? (I'd probably have a force rather than optional parameter, but that's just what I find slightly clearer).

Essentially there's two cases:

When we have a function that accepts an r-value reference then we should always move a temp, whatever type it is. If we don't do that then the code definitely won't compile.
For any other function then moving a temp is usually an optimization so should be done carefully (i.e. with the check for cpp_class and the macro to make sure that it's a known good compiler that compiler supports it).

Unless I'm misunderstanding something then I think it's better to go back to something very close to that commit.

da-woods · 2020-09-30T21:17:17Z

Please feel free to let me know if the following should be discussed in a separate GitHub issue:

Tangentially, the changes in this PR allow the Cython definition of std::move in libcpp.utility to be greatly simplified to just this:

The cleanup sounds like a good idea but it'd probably be best to do it separately afterwards I think.

shwina · 2020-09-30T21:49:09Z

I've done that, and changed the kwarg to force_move. Apologies for the back and forth on this.

scoder · 2020-10-01T06:32:30Z

Cython/Compiler/PyrexTypes.py

+class CReferenceType(CReferenceBaseType):
+
+    is_reference = 1
+    is_fake_reference = 0


Doesn't look like it was moved.

Cython/Compiler/Nodes.py

scoder · 2020-10-01T06:41:40Z

Cython/Compiler/ExprNodes.py

+    def _make_move_result_rhs(self, result, allow_move=True, force_move=False):
+        if force_move or (


Sorry to add to the back-and-forth on this, but I think optional=False is a better choice here. Using force_move=False means that I have to doubly opt-in to get what I call. That seems wrong. When I say move_result_rhs(), it should generate a move() for the argument, unless that's really not possible (i.e. in non-C++ mode). I would even prefer an explicit (C?-)compile time error if I request a move() but it cannot be done. If I want it to be best-effort, then that should be visible in the place where it's used. I should not have to jump through two levels of method calls to understand that the code that I'm looking at only sometimes does what it says.

No worries. That makes sense -- I changed it to optional=False. What do you think about removing the allow_move kwarg here? I think the combinationn of allow_move and optional make the function hard to reason about. I propose instead of:

condition = .... return _make_move_result_rhs(result, allow_move=condition, optional=optional)

We have:

condition = ... return _make_move_result_rhs(result, optional=optional) if condition else result

removing the allow_move kwarg

Good idea. Why call the method when the caller already knows that it's not needed?

I cleaned up the methods, let's hope I didn't mess more things up than I improved.

Cython/Compiler/PyrexTypes.py

scoder · 2020-10-03T09:13:51Z

Thanks!

@returns

* Fix test in Py2. * Check for exceptions also when @returns() is used, not only for "->" return type annotations. (cythonGH-3664) When you use Python type annotations, it would be weird if you lost Python exception propagation semantics along the way, just by compiling the code. So the default behaviour is "except? -1" here for C integer types. Arguably, this would also be a better default for the decorator case. See cython#3625 (comment) * Add tests that exception propagation works with the "@returns()" decorator. * Update changelog. * Allow selecting specific doctests in test modules with "-k pattern", instead of always running all tests. * Correct the positions reported for errors in f-strings. Closes cython#3674 * Fix f-string error positions in nogil test. * Clarify backwards incompatible special method change in changelog. * Fix many indentation and whitespace issues throughout the code base (cythonGH-3673) … and enforce them with pycodestyle. * Allow selecting specific doctests in test modules with "-k pattern", instead of always running all tests. * Prepare release of 0.29.20. * Update changelog. * Avoid calling PyUnicode_FromUnicode() in Py3. Closes cython#3677 * Update changelog. * Minor code simplification. * Always bind Cython functions * Update changelog. * Avoid an "unused variable" warning for code that gets compiled out in Py3. * Minor code simplification. * Revert "Always bind Cython functions" This reverts commit 6677326. * Update changelog. * Revert "Limited API updates and cleanup for cython#2056. cythonGH-3635)" This reverts commit 02bb311. * Revert "Invoke binop super method via direct slot access." This reverts commit bcb9387. * Revert "Add support for pow operator." This reverts commit d849fb2. * Revert "Python-style binary operation methods." This reverts commit e6a8124. * Re-add "c_api_binop_methods" directive for backwards compatibility after reverting cython#3633 and force it to "False". Closes cython#3688. * Change the default of the "c_api_binop_methods" directive to False. (cythonGH-3644) This is a backwards incompatible change that enables Python semantics for special methods by default, while the directive allows users to opt out of it and go back to the previous C-ish Cython semantics. See cython#2056 * Always bind Cython functions in Py3 (cythonGH-3683) Rebased 1bb26b9 for cython 0.29.x, and made conditional to Python 3. This does not solve the original staticmethod problem in Python 2 but it does resolve classmethod problems in Python 3.9. Therefore, it unbreaks other packages on Python 3.9, improves overall state for Python 3 and does not introduce regressions in Python 2. * Limit the scope of a local variable in a helper function. * Avoid a C compiler warning about a constant condition. * Remove dead code. * Update changelog. * Add support for C++ scoped enums with "enum class" and "enum struct" (cythonGH-3640) Closes cython#1603. * Add missing "PyUnicode_GET_LENGTH" to unicode.pxd (cythonGH-3692) * Add missing "PyUnicode_GET_LENGTH" to unicode.pxd (cythonGH-3692) * Fix usage of deprecated Py_UNICODE API. * Add safety fix to avoid reading a character from the empty string. * Fix prefix of internal function name. * Fix definition of "__Pyx_CyFunction_USED", which should only be #defined and not have a value. * Avoid using "tp_name" when CYTHON_COMPILING_IN_LIMITED_API (cythonGH-3693) * exec() did not allow recent Python syntax features in Py3.8+ due to https://bugs.python.org/issue35975 Closes cython#3695 * Avoid using the "tp_iternext" slot when CYTHON_USE_TYPE_SLOTS is disabled. * Give the "__Pyx_PyObject_GetIterNext" helper macro a more explanatory name. * Avoid unused variable in PyPy etc. * Avoid a call to PyTuple_GET_ITEM() to get the item array pointer if CYTHON_ASSUME_SAFE_MACROS is disabled. See cython#3701 * Enable travis for all branches. * Fix indentation counter for module init function. * Fix FunctionState handling for module cleanup function. * Validate that all temps were correctly released at the end of a function. * Keep reference to module dict around also in limited mode (cythonGH-3707) `PyModule_GetDict` is part of the limited API so we can keep a reference to the module dict around regardless of limited mode being enabled or not. * Validate and fix temp releasing (cythonGH-3708) * Fix a temp leak in the type init code. * Fix temp leaks in fused types initialisation code. * Correctly release the buffer index temps allocated for index calculations. * Make tests fails hard if a temp variable is not released at the end of a generated function. * Fix temp leak in switch statement code. * Make end-to-end tests fail on refnanny output. * Fix result temp leak in PyTypeTestNode. * Fix result temp leak in external type/function import code and enable the refnanny check for them. * Fix temp leak when try-return-finally is used in generators. * Make it explicit when an allocated temp is not meant to be reused. * Fix temp leak when assigning to the real/imag attributes of complex numbers. * Fix temp leak when assigning to a memoryview slice. * Clean up "num_threads" result temp in parallel section, not only in prange loop. * Fix temp leak in Pythran buffer setitem code. * Simplify NumPyMethodCallNode since it does not need the Python function anymore. Previously, it generated code that needlessly looked up the Python function without actually using it. * Fix temp leak when deleting C++ objects. * Add a test that actually reusing temps when deleting C++ objects works correctly. * Test runner: disable keep-alive output in --debug mode to keep a potential pdb console clean. * Fix argument name usage in finally blocks (cythonGH-3713) Fixes cython#3712 (hopefully) by reverting a small part of bbef4d7 * Fix argument name usage in finally blocks (cythonGH-3713) Fixes cython#3712 (hopefully) by reverting a small part of bbef4d7 * Document version-tagged pxd files (cythonGH-3587) * Implement generic optimized loop iterator with indexing and type inference for memoryviews (cythonGH-3617) * Adds bytearray iteration since that was not previously optimised (because it allows changing length during iteration). * Always set `entry.init` for memoryviewslice. * Update changelog. * Make end-to-end tests fail on refnanny output. * Fix FunctionState handling for module cleanup function. * Update change log. * Really only use PyUnicode_FromUnicode() when needed (cythonGH-3697) * Really only use PyUnicode_FromUnicode() for strings that contain lone surrogate, not for normal non-BMP strings and not for surrogate pairs on 16bit Unicode platforms. See cython#3678 * Extend buildenv test to debug a MacOS problem. * Add a test for surrogate pairs in Unicode strings. * Limit PyUnicode_FromUnicode() usage to strings containing lone surrogates. * Accept ambiguity of surrogate pairs in Unicode string literals when generated on 16bit Py2 systems. * Disable testing against NumPy 1.19+ in the 0.29.x branch, which removed C-API features. * Disable testing against NumPy 1.19.0 in the 0.29.x branch, which breaks a C-API call. * Validate and fix temp releasing (cythonGH-3708) (cythonGH-3717) * Validate and fix temp releasing (cythonGH-3708) Backports 92147ba. * Fix a temp leak in the type init code. * Fix temp leaks in fused types initialisation code. * Correctly release the buffer index temps allocated for index calculations. * Make tests fails hard if a temp variable is not released at the end of a generated function. * Fix temp leak in switch statement code. * Make end-to-end tests fail on refnanny output. * Fix result temp leak in PyTypeTestNode. * Fix result temp leak in external type/function import code and enable the refnanny check for them. * Fix temp leak when try-return-finally is used in generators. * Make it explicit when an allocated temp is not meant to be reused. * Fix temp leak when assigning to the real/imag attributes of complex numbers. * Fix temp leak when assigning to a memoryview slice. * Clean up "num_threads" result temp in parallel section, not only in prange loop. * Fix temp leak in Pythran buffer setitem code. * Simplify NumPyMethodCallNode since it does not need the Python function anymore. Previously, it generated code that needlessly looked up the Python function without actually using it. * Fix temp leak when deleting C++ objects. * Add a test that actually reusing temps when deleting C++ objects works correctly. * Improve syntax feature support of Cython CodeWriter (cythonGH-3514) * Disable Py_UNICODE fallback for __Pyx_UnicodeContainsUCS4() in Py3.9 since Py_UNICODE is deprecated and PEP-393 unicode is practically required. * Avoid calling PyUnicode_FromUnicode() in Py3 (cythonGH-3721) See cython#3677 * Add missing name substitution in __Pyx_PyUnicode_Substring() utility code. * Update changelog. * Prepare release of 0.29.21. * Really only use PyUnicode_FromUnicode() when needed (cythonGH-3697) * Really only use PyUnicode_FromUnicode() for strings that contain lone surrogate, not for normal non-BMP strings and not for surrogate pairs on 16bit Unicode platforms. See cython#3678 * Extend buildenv test to debug a MacOS problem. * Add a test for surrogate pairs in Unicode strings. * Limit PyUnicode_FromUnicode() usage to strings containing lone surrogates. * Accept ambiguity of surrogate pairs in Unicode string literals when generated on 16bit Py2 systems. * Disable Py_UNICODE fallback for __Pyx_UnicodeContainsUCS4() in Py3.9 since Py_UNICODE is deprecated and PEP-393 unicode is practically required. * Fix test in 16-bit Unicode deployments. * Update changelog. * Always consider 0-sized arrays as C- and F-contiguous (cythonGH-3728) Fixes cython#2093 * Always consider 0-sized arrays as C- and F-contiguous (cythonGH-3728) Fixes cython#2093 * Update changelog. * Add missing unordered_map template defaults (cythonGH-3686) * Add missing unordered_map template defaults (cythonGH-3686) * Update changelog. * Improve test output in case of failures. * Using Py_UNICODE to store lone surrogates makes Py3 join surrogate pairs on 16-bit Unicode platforms (Windows) when reading them back in, although we correctly processed them before. Instead, we now use the "unicode_escape" codec to store byte strings, because it can return surrogate characters (which the other codecs cannot). * Disable test in Py2.6. * Update changelog. * Use the more appropriate CYTHON_USE_TYPE_SLOTS guard for accessing the binop number slot. * Fix typo in error message. * Update the documentation on the arithmetic special methods and issue a "backwards compatibility" warning when the reversed method is not implemented. See cython#2056 * Add the new "c_api_binop_methods" directive to the documentation. * PyPy does not support PyType_GetSlot(). Use type slots instead. * Reformat doc paragraph. * Document C inline properties. * Beautify example output. * Use inline properties on the "PyComplex" builtin type declared in "cpython.complex" to provide C level access to the "real" and "imag" attributes (which Cython provides anyway for the 'undeclared' builtin type). * Prevent compile error when the result of repr() is assigned to a "unicode" LHS with language_level=3. Closes cython#3736 * Handle `for x in cpp_function_call()` (cythonGH-3667) Fixes cython#3663 This ensures that rvalues here are saved as temps, while keeping the existing behaviour for `for x in deref(vec)`, where the pointer for vec is copied, meaning it doesn't crash if vec is reassigned. The bit of this change liable to have the biggest effect is that I've changed the result type of dereference(x) and x[0] (where x is a c++ type) to a reference rather than value type. I think this is OK because it matches what C++ does. If that isn't a sensible change then I can probably inspect the loop sequence more closely to try to detect this. * Clarify changelog entry on ways to deal with the incompatible binop method change. * Remove Google Analytics script from documentation to avoid tracking our users. * Add donation banner to documentation. * Create bug template * Update issue templates * Delete unused custom ticket template * Add handshake emoji to donations banner to make it more visible. * Don't create CReference in C (only c++) (cythonGH-3746) * Fixed reference types being passed to getitemint (cythonGH-3755) * Reorder test module to restore the "invalid - valid" order. * Update changelog. * Prepare release of 3.0a6. * Add test comments on how "memslice.pyx" and "memoryview.pyx" relate. * Print test dependency versions to help with test failure debugging. * Add CI builds for different CPU architectures on travis. * Update issue templates * Add type "Py_hash_t" in pure Python mode. * Update changelog. * Clarify changelog entry. * Remove dead code and dead comments from "numpy/__init__.pxd". * Change "Py_intptr_t" declaration in numpy.pxd to what CPython uses as fallback. "int" seems overly lazy if it tends to be larger on many systems. * Readability improvements in "numpy.pxd". * Add a comment that numpy.pxd is maintained by the NumPy project. * Try to fix NumPy test failures by not setting the "NPY_NO_DEPRECATED_API" #define for NumPy 1.19.[01]. * Use NumPy 1.18.x for testing on travis as long as 1.19.[01+] ships a numpy.pxd that is incompatible with Cython 3.0. * Remove useless "extern" modifiers from cdef classes declared in "extern" sections. * Do not depend on the default type inference in "cpython/array.pxd". * Avoid unused C variable warning by moving declaration and usage under the same condition. Closes cython#3763 * Use a generic shape accessor for pythranized array (cythonGH-3762) This is a follow up to cython#3307 * Silence a C compiler warning about a constant condition. * Remove an unused function that had been added for the now-deleted "__getbuffer__" implementation. * Support PEP-560 ("__class_getitem__") for extension classes (cythonGH-3765) * Add "make checks" target to run code checks. * In bug template, ask for Python version in addition to Cython version * Support simple, non-strided views of "cython.array". Closes cython#3775 * Remove unused cimports. * Update changelog. * Set PYTHONHOME in embedding test to fix compilation issues in Py3.8/macOS. * Fix unrelated test after changing MemoryView.pyx. * Revert "Set PYTHONHOME in embedding test to fix compilation issues in Py3.8/macOS." This reverts commit a333d6a. The change did not resolve the test issue in travis. * Fix Python 3.4 + MSVC issue with elaborated type specifier for enum (cythonGH-3782) * Add more cimport_from_pyx tests (cythonGH-3786) There's now a fairly wide range of valid syntax for declaring things in pyx files and it should all be supported when cimporting from them. * Split known types into separate lines to let them have their own VCS history. * Add a note on PayPal fees for small payments. * Avoid merged-in code picking up directives from main module (cythonGH-3785) Fixes cython#1071 * Fix cygdb (cythonGH-3542) * Cython debugger documentation: Added link to an installation script. * Got a new libpython.py from the cpython source distribution. * Default language level in tests is now 3 instead of 2 * Migrated codefile from python 2 to python 3. * Added testcase for the cy list command in cygdb. * Temporarily removing test case that freezes gdb. * Fixed a bug that broke several Cygdb tests. The cython_debug/cython_debug_info_* files map the names of the C-functions generated by the Cython compiler to the names of the functions in the *.pyx source. If the function was defined using "def" (and not "cpdef" or "cdef") in the *.pyx source file, the C-function named in cython_debug/cython_debug_info_* used to be __pyx_pw_*, which is the name of the wrapper function and now it is __pyx_f_*, which is the name of the actual function. This makes some Cygdb tests pass that did not pass before. * Better error messages: If a cygdb command raises, a traceback will be printed. * Fixed a bug in cygdb. The following now works: 1. Start cygdb 2. Type "cy exec" and hit enter 3. Type some other lines 4. Type "end" and hit enter. -> These "other lines" will get executed * Fixed a bug in cygdb: cy list now works outside of functions. * Added print_hr_allmarkers function for easier debugging. * Fixed a bug that broke cygdb: cy break did not work if you put the breakpoint outside of a function if there was e.g. the following somewhere in your *.pyx file: cdef class SomeClass(): pass * Added a Cygdb test for printing global variables. * Fixing cygdb: Replaced cy print with a simple, working solution. * If an exception in Cygdb occurs, a stacktrace will be printed. * Fixed a bug that broke cy break -p * Bugfix: The compiler now writes out correctly which cython linenumber and path corresponds to which c linenumber. * Set language_level=2 in runtests.py * Update changelog. * Add a "gdb" test tag that depends on being able to run gdb. * Update CPython "test_fstring" copy to Py3.9. * Improved documentation for annotation typing (cythonGH-3799) Mainly by moving it to a separate section to make it easier to find, however also added a small amount of extra information about some of the obvious limitations. * Fix `return None` in CodeWriter. (cythonGH-3795) * Rename "GCC_DIAGNOSTIC" macro to make it Cython specific and make it available to other utility code sections. * Add a test for unpacking large PyLong values. * Silence gcc diagnostics whenever -1 is cast to something user provided. (cythonGH-3803) Fixes cythonGH-2749. * Define extern `PyBUF_MAX_NDIM` (cythonGH-3811) Ensure that Cython exposes `PyBUF_MAX_NDIM` from Python as part of `cpython.buffer` to allow access to developers. * Change test to use only integer calculations to prevent platform specific rounding issues, while keeping a reasonable distribution of integers across the PyLong digit ranges. * Created a glossary and added one new entry (cythonGH-3810) * Call destructors for structs in C++ (cythonGH-3228) In C++ mode, structs can contain C++ classes. Therefore structs should have their destructors called to ensure any class contained is also destructed. Also, a bit more thorough about ensuring constructor is generated where necessary. Closes cythonGH-3226. * Rename "GCC_DIAGNOSTIC" macro to make it Cython specific and make it available to other utility code sections. * Silence gcc diagnostics whenever -1 is cast to something user provided. (cythonGH-3803) Fixes cythonGH-2749. * Allow cast to ctuple (cythonGH-3812) * Make asyncio.iscoroutinefunction() recognise Cython compiled coroutines. (cythonGH-3427) Python's asyncio.coroutines uses an object to tag objects as coroutine functions. We now read this object and use it to tag Cython compiled coroutines as well. It also includes tests to make sure `asyncio.iscoroutinefunction()` works as expected. This doesn't fix `inspect.iscouroutinefunction()` (which uses a flag that can trigger undesirable behaviour for cython functions). Closes cython#2273 * Do not cover lines that were excluded in the coveragerc config file (cythonGH-3682) Closes cython#3680. * Add doc support for cpdef enum (cythonGH-3813) * Update changelog. * Add "check_size ignore" to all NumPy.pxd class declarations to silence the useless size warnings about them. See numpy/numpy#432 (comment) Also remove the useless "extern" modifiers from cdef classes declared in "extern" sections. * Update changelog. * Update changelog. * Support utility code in headers (cythonGH-3779) Specifically this allows public packed structs but may also allow other public declarations that require small amounts of utility code. * Prevent overflowing hash values for "Machines.Node" due to arbitrarily large unsigned pointer values. Closes cython#3840 * Extract "error_type" handling from the type creation functions. * Remember in the AnnotationsWriter when a serialised expression contained unknown/unsupported nodes. * Do not include "u" string prefix in annotations since tools that process them probably expect Py3 string semantics anyway. * Keep AnnotationNode instead of the bare annotation expression in "entry.annotation" to get access to the string representation. * Fix test after removing the 'u' prefix from unicode string annotation values. * Add support for PEP 526 `__annotations__` in class body. (cythonGH-3829) Closes cython#2552 * Some more glossary entries (cythonGH-3836) * Rename test file to avoid ambiguity with the C "const" modifier. * Join '*' and '**' parsing in declarators to avoid differences for 'const' parsing etc. * Set language level in Cython's speed-up .pxd files since it no longer depends on the .py files that they correspond to. * Declare "scanner.sy" as "unicode" string to optimise its usage. * Use more recent C-API functions on tear-down of the embedding code. * Improve some wordings in README.rst (cythonGH-3852) * Restore Py2 compatibility in test. * Replace useless comment. * Fixed assorted crashes with fused types (cythonGH-3853) Show a more detailed error for unspecializable fused return types. * Add a more common and more versatile example to the Verbatim C-Code documentation. * Fix RST typo. * Make C code C89 again. * Add support for forwarding references (cythonGH-3821) See, for example, https://isocpp.org/blog/2012/11/universal-references-in-c11-scott-meyers Closes cython#3814 * Avoid C compiler warnings about unused variables in test. * Avoid C compiler warning about unused variable in test. * Test Py3.7 and Py3.8 in C and C++ mode under appveyor. Run both in the same job since the machines are quite fast with parallel processes, but setting one up is slow. * Fix test compile failure in MSVC. * Allow creation of wrappers for cdef functions with memoryviews (cythonGH-3856) Fixes cython#3843 * Clean up and test type identifier escaping. - hash() hashing lead to unpredictable random prefixes for long names across multiple runs - use a single regex run instead of repeated calls to replace() * Move a memoryview test out of the "run" test directory since memoryview tests tend to be slow and have their own test directory. * Fix test compile failure in MSVC. * Clarify the section on exception return values. * Be a bit more paranoid about macro usage in the refnanny code. * Looks like the usual macro on Windows is "_WIN32" and not "WIN32". Let's support both, to be on the safe side. * Try to fix platform specific test once more. "synchapi.h" is not supposed to be included directly, and "windows.h" can break ... other stuff. Let's see what we can do. * Fix ReST typo. * docs: fix typos, minor clarification. * Minor docs clarification on error return values. (cythonGH-3859) * Replace deprecated Element.getiterator() with Element.iter(). (cythonGH-3864) `xml.etree.ElementTree.Element.getiterator()` was deprecated in Python 2.7 & 3.2 and removed in the freshly released Python 3.9. * Prefer Element.iter() over the deprecated Element.getiterator(). (cythonGH-3865) `xml.etree.ElementTree.Element.getiterator()` was deprecated in Python 2.7 & 3.2 and removed in Python 3.9. * Split a combined assert in two to avoid an unused C temp variable when assertions are disabled. (cythonGH-3870) * Split a combined assert in two to avoid an unused C temp variable when assertions are disabled. (cythonGH-3870) * Disable a test that fails in Py2 in 0.29.x since 'str' is 'unicode' in cython3, also in Python 2. * Rename test file to avoid ambiguity with the C "const" modifier. * Use more recent C-API functions on tear-down of the embedding code. * Make type identifier escaping deterministic: hash() hashing lead to unpredictable random prefixes for long names across multiple Python runs. * Clarify the section on exception return values. * Fix ReST typo. * docs: fix typos, minor clarification. * Minor docs clarification on error return values. (cythonGH-3859) * Add Python 3.9 to appveyor build. * Resolve merge conflict. * Removed `same_as` methods from Plex.Actions.Action (cythonGH-3847) It seems to be unused and it looked like the existing implementation was faulty. * Fix test after changing the hash method used for escaping long type descriptions. * Add warning for common user mistake and fix crash relating to annotated attributes. (cythonGH-3850) Closes cython#3830 * Improve error reporting when users mistakenly write "&&" or "||" instead of Python's "and" and "or" operators. (cythonGH-3858) * Disable the embedding test on MacOS-X to get the builds green again. Too many PRs depend on the travis tests to give a green light. * Simplify the output stream capturing for the C compiler runs by using a temp file instead of threads. * Help reporters see the first paragraph of the bug report template. * Detect _MSC_VER for __PYX_STD_MOVE_IF_SUPPORTED to support MSVC support even when "__cplusplus" is not set appropriately (cythonGH-3792) * Fix memoryview casts involving fused types (cythonGH-3882) I think this approach is more satisfactory than the old way it used to "work", where "fused_to_specific" was permanently added to the module scope containing the fused type (in this case the Cython scope), was applied in "Scope.lookup_type", but continued to have an effect on the scope forever. Closes cython#3881 * Add O_DIRECT to posix/fcntl.pxd (cythonGH-3894) Closes cython#3242 Co-authored-by: Stefan Behnel <stefan_ml@behnel.de> Co-authored-by: Jeroen Demeyer <jeroen.k.demeyer@gmail.com> Co-authored-by: Robert Bradshaw <robertwb@gmail.com> Co-authored-by: Michał Górny <mgorny@gentoo.org> Co-authored-by: Ashwin Srinath <3190405+shwina@users.noreply.github.com> Co-authored-by: Thomas A Caswell <tcaswell@gmail.com> Co-authored-by: Matthias Braun <matze@braunis.de> Co-authored-by: da-woods <dw-git@d-woods.co.uk> Co-authored-by: Tao He <sighingnow@gmail.com> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Clemens <chofreither@gmail.com> Co-authored-by: will <timwilloney@gmail.com> Co-authored-by: serge-sans-paille <serge.guelton@telecom-bretagne.eu> Co-authored-by: Yuan <pyslvs@gmail.com> Co-authored-by: Volker-Weissmann <39418860+Volker-Weissmann@users.noreply.github.com> Co-authored-by: Tao He <linzhu.ht@alibaba-inc.com> Co-authored-by: cf-natali <cf.natali@gmail.com> Co-authored-by: jakirkham <jakirkham@gmail.com> Co-authored-by: Tasha "Ren" Chin <tasha.s.chin@gmail.com> Co-authored-by: Pedro Marques da Luz <pmdaluz@gmail.com> Co-authored-by: Mathias Laurin <Mathias.Laurin+github.com@gmail.com> Co-authored-by: matham <moiein2000@gmail.com> Co-authored-by: Sean <stewart.d.sean@gmail.com> Co-authored-by: Rajat Dash <49777622+DARKPROGRAMMER16@users.noreply.github.com> Co-authored-by: ptype <ptype@users.noreply.github.com> Co-authored-by: Nick Pope <nick.pope@flightdataservices.com> Co-authored-by: Jeppe Dakin <jeppe_dakin@hotmail.com> Co-authored-by: Zackery Spytz <zspytz@gmail.com>

shwina added 11 commits September 3, 2020 15:55

Add an rvalue reference type

1595131

Parse into CppRvalueReferenceDeclaratorNode

2078384

Hack to handle forwarding referenes

b8c91d5

Localize the forwarding reference handling a bit more

d903253

Define and use is_forwarding_reference()

512b125

Better code reuse

c950497

Fix name

3ba1a86

Add simple forwarding test

e184733

Whitespace

6586aef

Merge branch 'master' into add-forwarding-references

eb1f744

Undo changes to move/utility

938842b

shwina commented Sep 9, 2020

View reviewed changes

shwina added 2 commits September 10, 2020 07:47

Rename test

ca3e12e

Style

4fa7d7f

shwina changed the title ~~[WIP] Add support for forwarding references~~ [RFC] Add support for forwarding references Sep 10, 2020

da-woods reviewed Sep 13, 2020

View reviewed changes

shwina added 3 commits September 14, 2020 09:31

Add CBaseReferenceDeclaratorNode for code reuse

c37d19b

Move forwarding ref logic

52e0ce9

Add a common CReferenceBaseType class

c83ec36

shwina changed the title ~~[RFC] Add support for forwarding references~~ [WIP] Add support for forwarding references Sep 14, 2020

Raise on unsupported use of &&

63bcf8b

shwina changed the title ~~[WIP] Add support for forwarding references~~ [RFC] Add support for forwarding references Sep 15, 2020

Fix line numbers in error tests

c2cb580

Merge branch 'master' into add-forwarding-references

44176bb

shwina added 2 commits September 30, 2020 11:49

Use PYX_STD_MOVE_IF_SUPPORTED and remove is_cpp_class restriction

41addc5

Merge branch 'add-forwarding-references' of github.com:shwina/cython …

ed246fa

…into add-forwarding-references

Formatting

725075f

Introduce force_move kwarg to _make_move_result_rhs

915e67a

scoder reviewed Oct 1, 2020

View reviewed changes

shwina and others added 4 commits October 1, 2020 07:39

Move is_fake_reference up to CReferenceBaseType

837c0e3

Test for reference base type being a Python object

0e5b3e8

Change force_move=False to optional=False

94c8e21

Clean up and simplify the move_result_rhs() methods.

8108139

scoder reviewed Oct 3, 2020

View reviewed changes

Cython/Compiler/PyrexTypes.py Outdated Show resolved Hide resolved

Simplify formatting.

b942e84

scoder merged commit 90684ac into cython:master Oct 3, 2020

scoder added this to the 3.0 milestone Oct 3, 2020

scoder added C++ Cython Language Feature feature labels Oct 3, 2020

shwina mentioned this pull request Oct 8, 2020

[RFC] Support for C++17 execution policy parameter #3790

Merged

bdice mentioned this pull request Mar 14, 2023

[Python] Building with Cython 3 shows warnings apache/arrow#34564

Closed

		return self.base.analyse(ref_type, env, nonempty=nonempty, visibility=visibility, in_pxd=in_pxd)


		class CppRvalueReferenceDeclaratorNode(CReferenceDeclaratorNode):

		def _make_move_result_rhs(self, result, allow_move=True, force_move=False):
		if force_move or (

Uh oh!

Conversation

shwina commented Sep 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shwina commented Sep 13, 2020

Uh oh!

da-woods left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

da-woods commented Sep 13, 2020

Uh oh!

shwina commented Sep 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

da-woods commented Sep 16, 2020

Uh oh!

shwina commented Sep 16, 2020

Uh oh!

da-woods commented Sep 16, 2020

Uh oh!

scoder commented Sep 30, 2020

Uh oh!

shwina commented Sep 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scoder commented Sep 30, 2020

Uh oh!

shwina commented Sep 30, 2020

Uh oh!

shwina commented Sep 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shwina commented Sep 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

da-woods commented Sep 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

da-woods commented Sep 30, 2020

Uh oh!

shwina commented Sep 30, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

scoder commented Oct 3, 2020

Uh oh!

Reviewers

shwina commented Sep 9, 2020 •

edited

Loading

da-woods left a comment •

edited

Loading

shwina commented Sep 16, 2020 •

edited

Loading

shwina commented Sep 30, 2020 •

edited

Loading

shwina commented Sep 30, 2020 •

edited

Loading

shwina commented Sep 30, 2020 •

edited

Loading

da-woods commented Sep 30, 2020 •

edited

Loading