Pgen is literally the oldest piece of technology in the CPython repo -- it was the first thing I wrote for Python over 29 years ago. It's not aged well, and building it requires various #if[n]def PGEN hacks in other parts of the code; it also depends more and more on CPython internals. There already is a replacement written in pure Python (Lib/lib2to3/pgen/), it just needs some glue to actually generate the graminit.[ch] files. Note that several other essential generation steps (everything listed for regen-all except regen-importlib and clinic) already depend on having a working Python interpreter around, so let's not worry about the bootstrapping process.

Hello!

I can help if you consider it appropriate :-)

@eamanu, feel free to submit a PR. I won't be available to guide you through the steps; there are other forums e.g. Zulip.

I didn't look at the PR yet, but this is a wonderful idea. I planned to do this just after rewriting generators of token-related data and code in Python (issue30455), but Pablo has overfoot me. Great!

New changeset 1f24a719e7be5e49b876a5dc7daf21d01ee69faa by Pablo Galindo in branch 'master':
bpo-35808: Retire pgen and use pgen2 to generate the parser (GH-11814)
https://github.com/python/cpython/commit/1f24a719e7be5e49b876a5dc7daf21d01ee69faa