As stated in PEP 617, the old parser will be removed in Python 3.10

New changeset 9727694f08cad4b019d2939224e3416312b1c0e1 by Lysandros Nikolaou in branch 'master':
bpo-40939: Generate keyword.py using the new parser (GH-20800)
https://github.com/python/cpython/commit/9727694f08cad4b019d2939224e3416312b1c0e1

>>> __new_parser__
  File "<stdin>", line 1
    __new_parser__
    ^
SyntaxError: You found it!

"new", "ex" or "ng" are not really future proof names. Can we rename the keyword to "__peg_parser__"?

> Can we rename the keyword to "__peg_parser__"?

I guess we could just remove this, as soon as the old parser is out. We were only using this to differentiate between the two parsers, when we were testing enabling/disabling the old one. I could get a PR ready to be merged after GH-20768 is there.

> I guess we could just remove this, as soon as the old parser is out. We were only using this to differentiate between the two parsers, when we were testing enabling/disabling the old one. I could get a PR ready to be merged after GH-20768 is there.

I would personally would like to keep the easter egg, but I assume is better to rename it to "__peg_parser__".

New changeset 961edf7979ca34d6fe104a1cce005aa8cac35821 by Miss Islington (bot) in branch '3.9':
bpo-40939: Generate keyword.py using the new parser (GH-20800)
https://github.com/python/cpython/commit/961edf7979ca34d6fe104a1cce005aa8cac35821

> I would personally would like to keep the easter egg, but I assume is better to rename it to "__peg_parser__".

Ok then! On it.

I did'nt ask to remove the easter egg. I'm just asking to avoid the "new" name. In my experience, each time that a "new" thing happens, later we have to use "new extended", "new_v2" or worse name :-)

Oh, if the name changes, please change it in 3.9 as well.


Look at this amazing names of the 5 flavors of functions parsing a string:

PyParser_ParseString()
PyParser_ParseStringFlags()
PyParser_ParseStringFlagsFilename()
PyParser_ParseStringFlagsFilenameEx() <= public!
PyParser_ParseStringObject()

Same for parsing a file:

PyParser_ParseFile()
PyParser_ParseFileFlags()
PyParser_ParseFileFlagsEx() <= public!
PyParser_ParseFileObject()

Or PyRun functions:

PyRun_String()
PyRun_AnyFile()
PyRun_AnyFileEx() <= public!
PyRun_AnyFileFlags()
PyRun_SimpleString()
PyRun_SimpleFile()
PyRun_SimpleFileEx() <= public!
PyRun_InteractiveOne()
PyRun_InteractiveLoop()
PyRun_File()
PyRun_FileEx() <= public!
PyRun_FileFlags()

ceval.c:

PyEval_EvalCode()
PyEval_EvalCodeEx() <= public!
_PyEval_EvalCodeWithName()
_PyEval_EvalCode()

I cannot count the number of "Ex" functions that we have :-)

Py_Finalize() -> :Py_FinalizeEx() <= public!
PyErr_Print() -> PyErr_PrintEx() <= public!
PySys_SetArgv() -> PySys_SetArgvEx() <= public!
PyErr_WarnEx() <= public!

_PyBytes_FormatEx()
_PyDict_MergeEx()
_Py_DecodeLocaleEx(), _Py_EncodeLocaleEx()
struct PyMemAllocatorEx
_Py_DecodeUTF8Ex(), _Py_EncodeUTF8Ex()
etc.

Honestly I see no reason to keep that easter egg. Can we remove it please?

> New changeset 961edf7979ca34d6fe104a1cce005aa8cac35821 by Miss Islington (bot) in branch '3.9':
> bpo-40939: Generate keyword.py using the new parser (GH-20800)

This change broke this buildbot:

AMD64 Arch Linux VintageParser 3.9:
https://buildbot.python.org/all/#/builders/765/builds/67
1 test failed: test_keyword

Ok, let's remove it. Lysandros can you modify PR 20802 to remove it?

> Lysandros can you modify PR 20802 to remove it?

Yup!

> This change broke this buildbot:

GH-20802 fixes this breakage.

New changeset bcd7deed9118e365c1225de2a2e1a81bf988c6ab by Lysandros Nikolaou in branch 'master':
bpo-40939: Remove PEG parser easter egg (__new_parser__) (#20802)
https://github.com/python/cpython/commit/bcd7deed9118e365c1225de2a2e1a81bf988c6ab

New changeset 1ed83adb0e95305af858bd41af531e487f54fee7 by Pablo Galindo in branch 'master':
bpo-40939: Remove the old parser (GH-20768)
https://github.com/python/cpython/commit/1ed83adb0e95305af858bd41af531e487f54fee7

A few remaining references to the good old times of the old parser:

Programs/_testembed.c:488:    putenv("PYTHONOLDPARSER=1");
Programs/_testembed.c:676:    putenv("PYTHONOLDPARSER=1");
Tools/scripts/run_tests.py:29:    if 'PYTHONOLDPARSER' not in os.environ:

> A few remaining references to the good old times of the old parser:

Thanks Victor, opened https://github.com/python/cpython/pull/20815 to address those.

New changeset 436b648910c27baf8164a6d46d746d36d8a93478 by Pablo Galindo in branch 'master':
bpo-40939: Remove some extra references to PYTHONOLDPARSER (GH-20815)
https://github.com/python/cpython/commit/436b648910c27baf8164a6d46d746d36d8a93478

New changeset 3782497cc22e70b41e32ac09cb06d3948074d8a7 by Pablo Galindo in branch '3.9':
[3.9] bpo-40939: Fix test_keyword for the old parser (GH-20814)
https://github.com/python/cpython/commit/3782497cc22e70b41e32ac09cb06d3948074d8a7

New changeset 756180b4bfa09bb77394a2b3754d331181d4f28c by Pablo Galindo in branch 'master':
bpo-40939: Clean and adapt the peg_generator directory after deleting the old parser (GH-20822)
https://github.com/python/cpython/commit/756180b4bfa09bb77394a2b3754d331181d4f28c

Shouldn't the following files be deleted too?

Include/bitset.h
Include/grammar.h
Include/graminit.h
Include/parsetok.h
Include/node.h
Python/graminit.c
Parser/node.c

Also declarations:
PyNode_Compile in Include/compile.h
PyParser_SimpleParse* in Include/pythonrun.h

And PyParser_ASTFrom* API need new implementations.

I'm currently testing a commit that removes all these files on my fork, before I push it upstream. A question that I'm not 100% sure about is if we can already remove the symbol module. I guess it's okay since it got deprecated in 3.9 (bpo-40759) and the old parser is also out, but just to make sure.

You can delete symbol.py -- it has no use now that the old parser is gone.

We should probably also update the regeneration targets in the Makefile. (At least review them.)

New changeset 314858e2763e76e77029ea0b691d749c32939087 by Lysandros Nikolaou in branch 'master':
bpo-40939: Remove the old parser (Part 2) (GH-21005)
https://github.com/python/cpython/commit/314858e2763e76e77029ea0b691d749c32939087

New changeset d301d9473e9a9b78d6e6678e9fe5ef66d46084e1 by Lysandros Nikolaou in branch '3.9':
[3.9] bpo-40939: Deprecate the PyParser_SimpleParse* functions (GH-21012)
https://github.com/python/cpython/commit/d301d9473e9a9b78d6e6678e9fe5ef66d46084e1

> bpo-40939: Remove the old parser (Part 2) (GH-21005)

This change removes PyNode_Compile() which is part of the public C API. Would you mind to also deprecate it, as you did for PyParser_xxx() functions?

New changeset 564cd187677ae8d1488c4d8ae649aea34ebbde07 by Lysandros Nikolaou in branch 'master':
bpo-40939: Rename PyPegen* functions to PyParser* (GH-21016)
https://github.com/python/cpython/commit/564cd187677ae8d1488c4d8ae649aea34ebbde07

Thanks Victor for the catch! I've opened a PR that deprecates PyNode_Compile in 3.9.

New changeset 8ae5e8ec8147e6434454e66953c25848b848711a by Lysandros Nikolaou in branch '3.9':
[3.9] bpo-40939: Deprecate PyNode_Compile (GH-21036)
https://github.com/python/cpython/commit/8ae5e8ec8147e6434454e66953c25848b848711a

New changeset 8d02f91dc6139a13b6efa9bd5a5b4bdd7ddcc29d by Ned Deily in branch 'master':
bpo-40939: run autoreconf to fix configure{,.ac} disparity (GH-21152)
https://github.com/python/cpython/commit/8d02f91dc6139a13b6efa9bd5a5b4bdd7ddcc29d

There are some difficulties with removing Grammar/Grammar, since it is used to generate the full grammar in the reference docs (Doc/reference/grammar.rst). Producing a similar grammar from the PEG grammar is currently painful because our PEG grammar contains a number of "invalid_*" rules that just exist to be able to produce more useful error messages.

See https://github.com/we-like-parsers/cpython/issues/135.

The continued existence of Grammar/Grammar has confused at least one person, see issue41362.

Grepping for Grammar/Grammar in the docs, there are a few other occurrences:

library/token.rst:14:of the parse tree (terminal tokens).  Refer to the file :file:`Grammar/Grammar`
library/parser.rst:43::file:`Grammar/Grammar` in the standard Python distribution.  The parse trees
library/symbol.rst:15:names.  Refer to the file :file:`Grammar/Grammar` in the Python distribution for
reference/grammar.rst:7:.. literalinclude:: ../../Grammar/Grammar

See also https://github.com/python/cpython/pull/19969 (Pablo's attempt at replacing the grammar in the reference docs with something derived from Grammar/python.gram).

New changeset 72cabb2aa636272e608285f5a6ba83b62be9be4e by Pablo Galindo in branch 'master':
bpo-40939: Use the new grammar for the grammar specification documentation (GH-19969)
https://github.com/python/cpython/commit/72cabb2aa636272e608285f5a6ba83b62be9be4e

The old parser is completely gone from the 3.10 branch. Closing.

New changeset e6b2d93f0c3891827f609ecac1ced21e1626ed0a by Guido van Rossum in branch '3.9':
[3.9] bpo-40939: Use the new grammar for the grammar specification documentation (GH-19969) (#21641)
https://github.com/python/cpython/commit/e6b2d93f0c3891827f609ecac1ced21e1626ed0a

New changeset b3fbff7289176ba1a322e6899c3d4a04880ed5a7 by Lysandros Nikolaou in branch 'master':
bpo-40939: Remove even more references to the old parser (GH-21642)
https://github.com/python/cpython/commit/b3fbff7289176ba1a322e6899c3d4a04880ed5a7

New changeset 5a8364780b7e881385f6fabcf072d599e80f51b8 by Terry Jan Reedy in branch 'master':
bpo-41808: Add What's New 3.9 entry missing from master (#22294)
https://github.com/python/cpython/commit/5a8364780b7e881385f6fabcf072d599e80f51b8