bpo-36541: Add lib2to3 grammar PEP-570 pos-only arg parsing #23759

gpshead · 2020-12-14T08:20:44Z

lib2to3 is effectively EOL, but it was never taught to parse positional only args ala PEP-570 that shipped in 3.8. This fixes that oversight.

Even though destined to wind up on PyPI instead of in the stdlib, this should help give it a proper sendoff and unblock things still using it under 3.8.x and 3.9.x from falling over on modern syntax.

Notably the blib2to3 within Black already backported these https://github.com/python/cpython/blob/3.8/Grammar/Grammar updates.

https://bugs.python.org/issue36541

This adds 3.8's PEP-570 support to lib2to3's pgen2. lib2to3, while being deprecated is still used by things to parse all versions of Python code today. We need it to support parsing modern 3.8 and 3.9 constructs.

Lib/lib2to3/tests/test_parser.py

pablogsal

LGTM

I left a minor comment

miss-islington · 2020-12-14T17:10:14Z

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8, 3.9.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

…-23759) Add positional only args support to lib2to3 pgen2. This adds 3.8's PEP-570 support to lib2to3's pgen2. lib2to3, while being deprecated is still used by things to parse all versions of Python code today. We need it to support parsing modern 3.8 and 3.9 constructs. Also add tests for complex *expr and **expr's. (cherry picked from commit 42c9f0f) Co-authored-by: Gregory P. Smith <greg@krypto.org>

bedevere-bot · 2020-12-14T17:10:28Z

GH-23768 is a backport of this pull request to the 3.9 branch.

bedevere-bot · 2020-12-14T17:10:39Z

GH-23769 is a backport of this pull request to the 3.8 branch.

…-23759) Add positional only args support to lib2to3 pgen2. This adds 3.8's PEP-570 support to lib2to3's pgen2. lib2to3, while being deprecated is still used by things to parse all versions of Python code today. We need it to support parsing modern 3.8 and 3.9 constructs. Also add tests for complex *expr and **expr's. (cherry picked from commit 42c9f0f) Co-authored-by: Gregory P. Smith <greg@krypto.org>

Add positional only args support to lib2to3 pgen2. This adds 3.8's PEP-570 support to lib2to3's pgen2. lib2to3, while being deprecated is still used by things to parse all versions of Python code today. We need it to support parsing modern 3.8 and 3.9 constructs. Also add tests for complex *expr and **expr's. (cherry picked from commit 42c9f0f) Co-authored-by: Gregory P. Smith <greg@krypto.org>

bedevere-bot · 2020-12-14T20:18:17Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora LTO + PGO 3.8 has failed when building commit 06bfd03.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/277/builds/209) and take a look at the build logs.
Check if the failure is related to this commit (06bfd03) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/277/builds/209

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 1        
remote: Enumerating objects: 11, done.        
remote: Counting objects:   9% (1/11)        
remote: Counting objects:  18% (2/11)        
remote: Counting objects:  27% (3/11)        
remote: Counting objects:  36% (4/11)        
remote: Counting objects:  45% (5/11)        
remote: Counting objects:  54% (6/11)        
remote: Counting objects:  63% (7/11)        
remote: Counting objects:  72% (8/11)        
remote: Counting objects:  81% (9/11)        
remote: Counting objects:  90% (10/11)        
remote: Counting objects: 100% (11/11)        
remote: Counting objects: 100% (11/11), done.        
remote: Total 12 (delta 10), reused 10 (delta 10), pack-reused 1        
From https://github.com/python/cpython
 * branch                  3.8        -> FETCH_HEAD
Reset branch '3.8'

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make[2]: [Makefile:1778: clean] Error 1 (ignored)
Objects/longobject.c: In function ‘_PyLong_Frexp’:
Objects/longobject.c:2989:33: warning: ‘x_digits[0]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 2989 |                     x_digits[0] |= 1;
      |                                 ^~
Objects/longobject.c: In function ‘PyLong_AsDouble’:
Objects/longobject.c:2989:33: warning: ‘x_digits[0]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 2989 |                     x_digits[0] |= 1;
      |                                 ^~
In function ‘assemble_lnotab’,
    inlined from ‘assemble_emit’ at Python/compile.c:5692:25,
    inlined from ‘assemble’ at Python/compile.c:6031:18:
Python/compile.c:5646:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5646 |         *lnotab++ = k;
      |         ~~~~~~~~~~^~~
In function ‘assemble_lnotab’,
    inlined from ‘assemble_emit’ at Python/compile.c:5692:25,
    inlined from ‘assemble’ at Python/compile.c:6031:18:
Python/compile.c:5646:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5646 |         *lnotab++ = k;
      |                   ^
In function ‘assemble_lnotab’,
    inlined from ‘assemble_emit’ at Python/compile.c:5692:25,
    inlined from ‘assemble’ at Python/compile.c:6031:18:
Python/compile.c:5646:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5646 |         *lnotab++ = k;
      |                   ^
In function ‘assemble_lnotab’,
    inlined from ‘assemble_emit’ at Python/compile.c:5692:25,
    inlined from ‘assemble’ at Python/compile.c:6031:18:
Python/compile.c:5646:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5646 |         *lnotab++ = k;
      |                   ^
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c: In function ‘load_float’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c:5395:9: warning: ‘s’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 5395 |     d = PyOS_string_to_double(s, &endptr, PyExc_OverflowError);
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c: In function ‘load_string’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c:5444:30: warning: ‘s’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 5444 |     if (len >= 2 && s[0] == s[len - 1] && (s[0] == '\'' || s[0] == '"')) {
      |                              ^
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c: In function ‘load_int’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c:5181:17: warning: ‘s’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 5181 |         value = PyLong_FromString(s, NULL, 0);
      |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c: In function ‘load_get’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c:6227:11: warning: ‘s’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 6227 |     key = PyLong_FromString(s, NULL, 10);
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c: In function ‘load_persid’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c:6107:15: warning: ‘s’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 6107 |         pid = PyUnicode_DecodeASCII(s, len - 1, "strict");
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_testcapimodule.c: In function ‘PyInit__testcapi’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_testcapimodule.c:6328:5: warning: ‘tp_print’ is deprecated [-Wdeprecated-declarations]
 6328 |     MyList_Type.tp_print = 0;
      |     ^~~~~~~~~~~
In file included from ./Include/object.h:746,
                 from ./Include/pytime.h:6,
                 from ./Include/Python.h:85,
                 from /home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_testcapimodule.c:15:
./Include/cpython/object.h:260:30: note: declared here
  260 |     Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
      |                              ^~~~~~~~
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_testcapimodule.c: In function ‘test_buildvalue_issue38913’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_testcapimodule.c:6463:15: warning: variable ‘res’ set but not used [-Wunused-but-set-variable]
 6463 |     PyObject *res;
      |               ^~~
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/socketmodule.c: In function ‘getsockaddrarg’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/socketmodule.c:2316:9: warning: ‘strncpy’ specified bound 64 equals destination size [-Wstringop-truncation]
 2316 |         strncpy((char *)sa->salg_name, name, sizeof(sa->salg_name));
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Parser/listnode.c: In function ‘list1node’:
Parser/listnode.c:66:1: warning: ‘/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Parser/listnode.gcda’ profile count data file not found [-Wmissing-profile]
   66 | }
      | ^
Python/frozenmain.c: In function ‘Py_FrozenMain’:
Python/frozenmain.c:129:1: warning: ‘/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/frozenmain.gcda’ profile count data file not found [-Wmissing-profile]
  129 | }
      | ^
Python/pyfpe.c: In function ‘PyFPE_dummy’:
Python/pyfpe.c:15:1: warning: ‘/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/pyfpe.gcda’ profile count data file not found [-Wmissing-profile]
   15 | }
      | ^
/usr/bin/ld: python.lto.o: in function `run_mod':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/pythonrun.c:1178: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: python.lto.o: in function `symtable_add_def_helper':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: undefined reference to `_Py_Mangle'
/usr/bin/ld: python.lto.o: in function `builtin_compile':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/bltinmodule.c:808: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: python.lto.o: in function `Py_CompileStringObject':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/pythonrun.c:1254: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: python.lto.o: in function `symtable_lookup':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1009: undefined reference to `_Py_Mangle'
/usr/bin/ld: python.lto.o: in function `symtable_record_directive':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1163: undefined reference to `_Py_Mangle'
/usr/bin/ld: python.lto.o: in function `type_new':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Objects/typeobject.c:2504: undefined reference to `_Py_Mangle'
/usr/bin/ld: python.lto.o: in function `symtable_visit_expr':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: undefined reference to `_Py_Mangle'
/usr/bin/ld: python.lto.o:/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: undefined reference to `_Py_Mangle'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:578: python] Error 1
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: _testembed.lto.o: in function `run_mod':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/pythonrun.c:1178: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: _testembed.lto.o: in function `symtable_add_def_helper':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: undefined reference to `_Py_Mangle'
/usr/bin/ld: _testembed.lto.o: in function `builtin_compile':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/bltinmodule.c:808: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: _testembed.lto.o: in function `Py_CompileStringObject':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/pythonrun.c:1254: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: _testembed.lto.o: in function `symtable_lookup':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1009: undefined reference to `_Py_Mangle'
/usr/bin/ld: _testembed.lto.o: in function `symtable_record_directive':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1163: undefined reference to `_Py_Mangle'
/usr/bin/ld: _testembed.lto.o: in function `type_new':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Objects/typeobject.c:2504: undefined reference to `_Py_Mangle'
/usr/bin/ld: _testembed.lto.o: in function `symtable_visit_expr':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: undefined reference to `_Py_Mangle'
/usr/bin/ld: _testembed.lto.o:/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: undefined reference to `_Py_Mangle'
/usr/bin/ld: _testembed.lto.o:/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: more undefined references to `_Py_Mangle' follow
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:709: Programs/_testembed] Error 1
make: *** [Makefile:521: profile-opt] Error 2

…-23759) Add positional only args support to lib2to3 pgen2. This adds 3.8's PEP-570 support to lib2to3's pgen2. lib2to3, while being deprecated is still used by things to parse all versions of Python code today. We need it to support parsing modern 3.8 and 3.9 constructs. Also add tests for complex *expr and **expr's.

gpshead added 2 commits December 14, 2020 07:58

Add positional only args support to lib2to3 pgen2.

16989da

This adds 3.8's PEP-570 support to lib2to3's pgen2. lib2to3, while being deprecated is still used by things to parse all versions of Python code today. We need it to support parsing modern 3.8 and 3.9 constructs.

Add tests for complex *expr and **expr's.

093b468

gpshead added type-bug An unexpected behavior, bug, or error tests Tests in the Lib/test dir needs backport to 3.8 labels Dec 14, 2020

the-knights-who-say-ni added the CLA signed label Dec 14, 2020

bedevere-bot added the awaiting core review label Dec 14, 2020

NEWS via blurb

6e728b0

gpshead self-assigned this Dec 14, 2020

gpshead mentioned this pull request Dec 14, 2020

Python 3.8 support google/yapf#772

Closed

pablogsal reviewed Dec 14, 2020

View reviewed changes

Lib/lib2to3/tests/test_parser.py Show resolved Hide resolved

pablogsal approved these changes Dec 14, 2020

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Dec 14, 2020

Add another test case.

5d7321c

gpshead merged commit 42c9f0f into python:master Dec 14, 2020

bedevere-bot removed the awaiting merge label Dec 14, 2020

gpshead deleted the lib2to3-grammar-positional-only branch December 14, 2020 17:10

bedevere-bot removed needs backport to 3.9 labels Dec 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

bpo-36541: Add lib2to3 grammar PEP-570 pos-only arg parsing #23759

bpo-36541: Add lib2to3 grammar PEP-570 pos-only arg parsing #23759

Uh oh!

gpshead commented Dec 14, 2020 •

edited by bedevere-bot

Loading

Uh oh!

Uh oh!

pablogsal left a comment

Uh oh!

miss-islington commented Dec 14, 2020

Uh oh!

bedevere-bot commented Dec 14, 2020

Uh oh!

bedevere-bot commented Dec 14, 2020

Uh oh!

bedevere-bot commented Dec 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

bpo-36541: Add lib2to3 grammar PEP-570 pos-only arg parsing #23759

bpo-36541: Add lib2to3 grammar PEP-570 pos-only arg parsing #23759

Uh oh!

Conversation

gpshead commented Dec 14, 2020 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

pablogsal left a comment

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Dec 14, 2020

Uh oh!

bedevere-bot commented Dec 14, 2020

Uh oh!

bedevere-bot commented Dec 14, 2020

Uh oh!

bedevere-bot commented Dec 14, 2020

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gpshead commented Dec 14, 2020 •

edited by bedevere-bot

Loading