Wow, cool! Thank you for working on this!

I'm having a quick look over the PR now and so far, it looks lovely.

Regarding 2), if the only reason for using a header file is output ordering, then you can rely on the StringIOTree to do that for you. See the code_layout in Code.py, which structures the generated C file. You can add new sections if necessary and write code into any of the sections at "any time" (code.globalstate.parts["…"]), well, it shouldn't get too messy, but you probably get the idea. When you just need to "go back a little", there's also code.insertion_point(), which gives you a new code writer to insert code at this point later, usually within the same code generation function.

Regarding testing, I would add a new test configuration in travis for this (or a couple, say for Py3.5/8/9), and maybe also add a new option --limited-api to runtests.py to enable CYTHON_LIMITED_API. We might also want to have a tag like nolimitedapi to mark tests that simply don't work together with the Limited API.

Awesome thanks for all the feedback! I'll do the following changes:

1) Make all current tests pass
2) Generated all limited api code in the same *.c file
3) Address all the inline comments
4) Integrate the limited api into the CI

I'll be pushing all of the changes here to confirm with the CI that everything passes. I'll let you know once it's ready for a second round of review

Thanks for all the comments! Something came up this week and I haven't been able to focus on the generator but I should get back to it starting Monday, expect more updates soon :)

@scoder I just realized, after attempting to integrate with the CI, getting all the unittests up and running with the Limited API would still require much more work.

What about setting an initial milestone of running something simpler/smaller. From there we can start adding more support on separate PRs. Otherwise, this PR will just keep getting bigger and bigger. So, is there a way to select a subset of the unittests to run with the Limited API?

Thoughts?

Possibly the thing to do is to mark the limited API tests as "allow_failures", like PyPy currently is

(You shouldn't take this as any more than a suggestion though - I'm definitely not one of the people who decides on anything!)

Since this is essentially a new and independent compilation mode for users, I think it's ok if it doesn't start off with all features available (although a loud compilation #error would be better than silently missing some C code in that case). The problem with CI here is that once we allow it to fail, it becomes difficult to notice regressions. That's a big problem for the PyPy support already.

Suggestion: once the remaining C compilation errors are fixed, we can see if it's doable to blacklist the "not for now" tests in a separate bugs file, like we do for some tests that are known to fail in MS-Windows or PyPy. That way, we could keep this code in the required set of passing configurations.

Whew, finally managed to get it under a testing hook. I had to blacklist a couple of tests using --no-file, --backend=c, and limited_api_bugs.txt. This should make it easier to address all the pending comments and start working towards supporting all the blacklisted tests (file, cpp, and limited_api_bugs). Once I address the in-line comments I'll remove the RFC title!

Friendly ping @scoder , let me know if you have any questions 🙂

When trying to convert a project with -DCYTHON_LIMITED_API=1 using this pull request I run into a lot of errors of the form:
XXX___pyx_scope_struct_XXX_genexpr was not declared in this scope.
Seems to occur when there are nested if/elif statements.

@idenc This is not aiming to be 100% support for the Limited API. This is the first step to get the core infra in which covers a wide range of cases. Intentionally avoiding to add more stuff in this PR as it's already big enough. From here, we'll start tackling all of the cases, like the one that you just uncovered.

TL;DR stay put, this is still work in progress! 🙂

@scoder I have more changes lined up to expand/have better LIMITED_API support. Ideally I'd like to merge this PR before adding any more changes. Let me know what you think!

@scoder Thanks for the review, I just went through all of the comments. Let me know what you think!

@scoder Ready for another round of reviews!

@scoder ready again! 🙂

Nice! Let's get this feature in!

Thanks! Also, feel free to ping me if any issues pop up in the issue tracker related to the LIMTED_API!

@idenc if all goes well, more Limited API support (including removing the pyx_scope errors) will be coming soon.

@tekknolagi @eduardo-elizondo please base any future PRs on #3611 where I refactored the whole design into more suitable feature sets. There are still a couple of issues left in that branch that need fixing, but not many. Help on investigating those issues would also be appreciated.

I also created ticket #3689 for further cleanups of the current implementation.

@scoder thanks, this is pretty great! Yeah I think @tekknolagi (or one of our peers) will be leading the upstreaming effort

Yes sorry for not contacting and pushing our changes to open source earlier; our team has been working on a number of cython/limited API improvements as well. I will try to at least get some branch up on github today so you can see what to expect. We'll take a look at rebasing on #3611

I just exported our work here: https://github.com/MatzeB/cython/commits/limited will rebase and make that into pull requests in the coming days.