In Fedora, when building Python 3.11.0a2 with Python 3.11.0a2 for regen, we found that regen-frozen fails.

It can be reproduced either on the v3.11.0a2 tag or on the main branch.

On tag v3.11.0a2 with 3.11.0a2 installed in $PATH:

# in root of cpython local clone:
$ git clean -fdx
$ mkdir -p build/optimized
$ cd build/optimized
$ ../../configure
$ make regen-all PYTHON_FOR_REGEN=python3.11
...
ERROR: missing _freeze_module


On the main branch:

# in root of cpython local clone:
$ git clean -fdx
$ mkdir -p build/optimized
$ cd build/optimized
$ ../../configure
$ make
$ make regen-all PYTHON_FOR_REGEN=./python
(success)

However, the working tree is dirty:

$ git diff
diff --git a/Parser/parser.c b/Parser/parser.c
index b3aa35989ed..c26cd6eeb05 100644
--- a/Parser/parser.c
+++ b/Parser/parser.c
@@ -1,4 +1,4 @@
-// @generated by pegen from ./Grammar/python.gram
+// @generated by pegen from ../../Grammar/python.gram
 #include "pegen.h"
 
 #if defined(Py_DEBUG) && defined(Py_BUILD_CORE)
diff --git a/Tools/peg_generator/pegen/grammar_parser.py b/Tools/peg_generator/pegen/grammar_parser.py
index 6e9f7d3d11d..fbbbfad76b0 100644
--- a/Tools/peg_generator/pegen/grammar_parser.py
+++ b/Tools/peg_generator/pegen/grammar_parser.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python3.8
-# @generated by pegen from ./Tools/peg_generator/pegen/metagrammar.gram
+# @generated by pegen from ../../Tools/peg_generator/pegen/metagrammar.gram
 
 import ast
 import sys


And if we install Python somewhere:

$ make
$ make install DESTDIR=/tmp/python

And use that one again for regen:

$ cd ../..
$ git clean -fdx
$ mkdir -p build/optimized
$ cd build/optimized
$ ../../configure
$ make regen-all PYTHON_FOR_REGEN=/tmp/python/usr/local/bin/python3.11
...
# Regenerate Lib/keyword.py from Grammar/python.gram and Grammar/Tokens
# using Tools/peg_generator/pegen
PYTHONPATH=../../Tools/peg_generator /tmp/python/usr/local/bin/python3.11 -m pegen.keywordgen \
	../../Grammar/python.gram \
	../../Grammar/Tokens \
	../../Lib/keyword.py.new
/tmp/python/usr/local/bin/python3.11 ../../Tools/scripts/update_file.py ../../Lib/keyword.py ../../Lib/keyword.py.new
/tmp/python/usr/local/bin/python3.11 ../../Tools/scripts/freeze_modules.py
ERROR: missing _freeze_module
make: *** [Makefile:1259: regen-frozen] Error 1



It fails. I've isolated the failure to:

$ make regen-frozen PYTHON_FOR_REGEN=/tmp/python/usr/local/bin/python3.11
/tmp/python/usr/local/bin/python3.11 ../../Tools/scripts/freeze_modules.py
ERROR: missing _freeze_module
make: *** [Makefile:1259: regen-frozen] Error 1

git bisect says:

ff8859d965815e8b5af346bd90299cfa5568c855 is the first new commit
commit ff8859d965815e8b5af346bd90299cfa5568c855
Author: Victor Stinner <vstinner@python.org>
Date:   Thu Oct 7 21:19:13 2021 +0200

    bpo-45402: Fix test_tools.test_sundry() (GH-28786)
    
    Fix test_tools.test_sundry() when Python is built out of tree: fix
    how the freeze_modules.py tool locates the _freeze_module program.

 bpo-45402.jlQvep.rst">Misc/NEWS.d/next/Tests/2021-10-07-13-43-01.bpo-45402.jlQvep.rst | 3 +++
 Tools/scripts/freeze_modules.py                                 | 7 ++++---
 2 files changed, 7 insertions(+), 3 deletions(-)
 create mode 100644 bpo-45402.jlQvep.rst">Misc/NEWS.d/next/Tests/2021-10-07-13-43-01.bpo-45402.jlQvep.rst

It seems the fix for an out-of-tree build bug introduced another one.

It also seems that it is *not* necessary to use Python 3.11 for regen-frozen to get the failure:

$ make regen-frozen PYTHON_FOR_REGEN=python3.10
python3.10 ../../Tools/scripts/freeze_modules.py
ERROR: missing _freeze_module
make: *** [Makefile:1259: regen-frozen] Error 1

$ make regen-frozen PYTHON_FOR_REGEN=python3.9
python3.9 ../../Tools/scripts/freeze_modules.py
ERROR: missing _freeze_module
make: *** [Makefile:1259: regen-frozen] Error 1


My guess is that the fix for test_tools.test_sundry() actually broke the real usage. It has:

        # When building out of the source tree, get the tool from directory
        # of the Python executable
        TOOL = os.path.dirname(sys.executable)
        TOOL = os.path.join(TOOL, 'Programs', '_freeze_module')

But sys.executable is *not* the Python we have built, but the Python we use for regen.
E.g. when Python for regen is /usr/bin/python3.X, it tries to use /usr/bin/Programs/_freeze_module which obviously is not there.

The assumption that sys.executable is the freshly built Python is only correct in the tests, but not in reality.

New changeset 253b7a0a9fef1d72a4cb87b837885576e68e917c by Victor Stinner in branch 'main':
bpo-45866: pegen strips directory of "generated from" header (GH-29777)
https://github.com/python/cpython/commit/253b7a0a9fef1d72a4cb87b837885576e68e917c

New changeset b6defde2afe656db830d6fedf74ca5f6225f5928 by Victor Stinner in branch '3.10':
bpo-45866: pegen strips directory of "generated from" header (GH-29777) (GH-29792)
https://github.com/python/cpython/commit/b6defde2afe656db830d6fedf74ca5f6225f5928

New changeset 93a540d74c654819ad04d72fcdcf827d0e259999 by Victor Stinner in branch '3.9':
bpo-45866: pegen strips directory of "generated from" header (GH-29777) (GH-29792) (GH-29797)
https://github.com/python/cpython/commit/93a540d74c654819ad04d72fcdcf827d0e259999

It seems like all bugs reported in this issue has been fixed, so I close the issue. Thanks for the bug report ;-)

> -# @generated by pegen from ./Tools/peg_generator/pegen/metagrammar.gram
> +# @generated by pegen from ../../Tools/peg_generator/pegen/metagrammar.gram

This issue is now fixed in 3.9, 3.10 and main branches.


> ERROR: missing _freeze_module

I'm no longer able to reproduce this error on an up-to-date main branch. The issue has been fixed by this change:

commit 5be98e57b3c3b36d1a1176b49c73b8822c6380e7
Author: Guido van Rossum <guido@python.org>
Date:   Tue Nov 23 08:56:06 2021 -0800

    bpo-45873: Get rid of bootstrap_python (#29717)
    
    Instead we use $(PYTHON_FOR_REGEN) .../deepfreeze.py with the
    frozen .h file as input, as we did for Windows in bpo-45850.
    
    We also get rid of the code that generates the .h files
    when make regen-frozen is run (i.e., .../make_frozen.py),
    and the MANIFEST file.
    
    Restore Python 3.8 and 3.9 as Windows host Python again
    
    Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>

New changeset 9a7611a7c4da6406383f8ffcea272ded6f890f1e by Victor Stinner in branch 'main':
bpo-45866: Fix typo in the NEWS entry (GH-29798)
https://github.com/python/cpython/commit/9a7611a7c4da6406383f8ffcea272ded6f890f1e

New changeset fc16ea9c8b8769af8a4c0c16fed7eba2e8bf4019 by Miss Islington (bot) in branch '3.10':
bpo-45866: Fix typo in the NEWS entry (GH-29798)
https://github.com/python/cpython/commit/fc16ea9c8b8769af8a4c0c16fed7eba2e8bf4019

New changeset b3f14dacfea54332e2ddde792142d818b3554dbc by Miss Islington (bot) in branch '3.9':
bpo-45866: Fix typo in the NEWS entry (GH-29798)
https://github.com/python/cpython/commit/b3f14dacfea54332e2ddde792142d818b3554dbc