Add support for Python 3.12 by danyeaw · Pull Request #7670 · pyinstaller/pyinstaller

danyeaw · 2023-05-31T00:53:15Z

Python 3.12 is now in beta and this PR adds support.

bwoodsend · 2023-05-31T20:23:46Z

I've got a half baked branch for the imp->importlib transition.

danyeaw · 2023-06-01T01:42:52Z

@bwoodsend Oh nice! Could we get those changes merged in first, or could I cherry-pick them in to this PR?

bwoodsend · 2023-06-02T17:53:37Z

Working on it...

There's also this lovely surprise from the changelog:

bpo-44113: Deprecate the following functions to configure the Python initialization:

PySys_AddWarnOptionUnicode()

PySys_AddWarnOption()

PySys_AddXOption()

PySys_HasWarnOptions()

Py_SetPath()

Py_SetProgramName()

Py_SetPythonHome()

Py_SetStandardStreamEncoding()

_Py_SetProgramFullPath()

Use the new PyConfig API of the Python Initialization Configuration instead (PEP 587).

although it looks like we're safe for another year.

rokm · 2023-06-03T10:21:46Z

Working on it...

There's also this lovely surprise from the changelog:

bpo-44113: Deprecate the following functions to configure the Python initialization:

PySys_AddWarnOptionUnicode()

PySys_AddWarnOption()

PySys_AddXOption()

PySys_HasWarnOptions()

Py_SetPath()

Py_SetProgramName()

Py_SetPythonHome()

Py_SetStandardStreamEncoding()

_Py_SetProgramFullPath()

Use the new PyConfig API of the Python Initialization Configuration instead (PEP 587).

although it looks like we're safe for another year.

That doesn't look too bad. Last time I looked at the interpreter initialization/configuration, I was (mistakenly?) under impression we would have to deal with config struct directly, which would be a pain to do in version agnostic way. But this looks like an opaque structure with an API that abstracts the actual struct layout away, so it's quite nice for our use case.

It's available from python 3.8 on, so we can start making the switch once we drop support for 3.7.

bwoodsend · 2023-06-04T10:41:28Z

#7676 is merged, let's see where a rebase takes us...

bwoodsend · 2023-06-04T10:45:02Z

... but not using a merge commit!!!

rokm · 2023-06-04T11:01:35Z

It will eventually take us here: https://github.com/rokm/pyinstaller/actions/runs/5166038863

Overall, it's not that terrible...

FAILED tests/unit/test_bytecode.py::test_nested_codes - AssertionError: assert [('bar', []),...('fizz', [3])] == [('bar', []), ('range', [10])]
  Left contains one more item: ('fizz', [3])
  Full diff:
  - [('bar', []), ('range', [10])]
  + [('bar', []), ('range', [10]), ('fizz', [3])]
  ?

I'm not sure what this one is about yet...

FAILED tests/functional/test_basic.py::test_distutils_submod[onedir] - Failed: Running exe /Users/runner/work/_temp/pytest-of-runner/pytest-0/popen-gw0/test_distutils_submod_onedir_0/dist/test_source/test_source failed with return-code 1.
FAILED tests/functional/test_basic.py::test_distutils_submod[onefile] - Failed: Running exe /Users/runner/work/_temp/pytest-of-runner/pytest-0/popen-gw0/test_distutils_submod_onefile_0/dist/test_source failed with return-code 1.
FAILED tests/functional/test_basic.py::test_python_makefile[onedir-from distutils ] - Failed: Running exe /Users/runner/work/_temp/pytest-of-runner/pytest-0/popen-gw0/test_python_makefile_onedir_fr0/dist/test_source/test_source failed with return-code 1.
FAILED tests/functional/test_basic.py::test_python_makefile[onefile-from distutils ] - Failed: Running exe /Users/runner/work/_temp/pytest-of-runner/pytest-0/popen-gw0/test_python_makefile_onefile_f0/dist/test_source failed with return-code 1.

These tests using distutils should not be failing as distutils from setuptools should be available.

My current hypothesis is that the current modulegraph's approach of looping over search locations, taking importer for it, and trying to look up the module for it was working for <= 3.11, because stdlib distutils was discoverable by built-in FileFinder (for example, in /usr/lib64/python3.11/distutils/init.py`, so the stdlib module was found, and then the hook took care of collecting setuptools' variant.

With 3.12, we don't have stdlib distutils anymore, so location-found finder's fail to find the module, and the hook is not ran at all.

I'll see if we can completely replace this block:

pyinstaller/PyInstaller/lib/modulegraph/modulegraph.py

Lines 2952 to 3026 in fd0a25a

    
           try: 
        
               for search_dir in search_dirs: 
        
                   # PEP 302-compliant importer making loaders for this directory. 
        
                   importer = pkgutil.get_importer(search_dir) 
        
                   # If this directory is not importable, continue. 
        
                   if importer is None: 
        
                       # self.msg(4, "_find_module_path importer not found", search_dir) 
        
                       continue 
        
                   # Get the PEP 302-compliant loader object loading this module. 
        
                   # 
        
                   # If this importer defines the PEP 451-compliant find_spec() 
        
                   # method, use that, and obtain loader from spec. This should 
        
                   # be available on python >= 3.4. 
        
                   if hasattr(importer, 'find_spec'): 
        
                       loader = None 
        
                       spec = importer.find_spec(module_name) 
        
                       if spec is not None: 
        
                           loader = spec.loader 
        
                           namespace_dirs.extend(spec.submodule_search_locations or []) 
        
                   # Else if this importer defines the PEP 302-compliant find_loader() 
        
                   # method, use that. 
        
                   elif hasattr(importer, 'find_loader'): 
        
                       loader, loader_namespace_dirs = importer.find_loader( 
        
                           module_name) 
        
                       namespace_dirs.extend(loader_namespace_dirs) 
        
                   # Else if this importer defines the Python 2-specific 
        
                   # find_module() method, fall back to that. Despite the method 
        
                   # name, this method returns a loader rather than a module. 
        
                   elif hasattr(importer, 'find_module'): 
        
                       loader = importer.find_module(module_name) 
        
                   # Else, raise an exception. 
        
                   else: 
        
                       raise ImportError( 
        
                           "Module %r importer %r loader unobtainable" % (module_name, importer)) 
        
                   # If this module is not loadable from this directory, continue. 
        
                   if loader is None: 
        
                       # self.msg(4, "_find_module_path loader not found", search_dir) 
        
                       continue 
        
                   # Absolute path of this module. If this module resides in a 
        
                   # compressed archive, this is the absolute path of this module 
        
                   # after extracting this module from that archive and hence 
        
                   # should not exist; else, this path should typically exist. 
        
                   pathname = None 
        
                   # If this loader defines the PEP 302-compliant get_filename() 
        
                   # method, preferably call that method first. Most if not all 
        
                   # loaders (including zipimporter objects) define this method. 
        
                   if hasattr(loader, 'get_filename'): 
        
                       pathname = loader.get_filename(module_name) 
        
                   # Else if this loader provides a "path" attribute, defer to that. 
        
                   elif hasattr(loader, 'path'): 
        
                       pathname = loader.path 
        
                   # Else, raise an exception. 
        
                   else: 
        
                       raise ImportError( 
        
                           "Module %r loader %r path unobtainable" % (module_name, loader)) 
        
                   # If no path was found, this is probably a namespace package. In 
        
                   # such case, continue collecting namespace directories. 
        
                   if pathname is None: 
        
                       self.msg(4, "_find_module_path path not found", pathname) 
        
                       continue 
        
                   # Return such metadata. 
        
                   path_data = (pathname, loader) 
        
                   break 
        
               # Else if this is a namespace package, return such metadata. 
        
               else: 
        
                   if namespace_dirs: 
        
                       path_data = (namespace_dirs[0], 
        
                                    NAMESPACE_PACKAGE(namespace_dirs))

with importlib.util.find_spec, which should work for all meta-finders and ignore the search paths...

FAILED tests/functional/test_import.py::test_import_metapath1[onedir] - Failed: Running exe /tmp/pytest-of-runner/pytest-0/popen-gw1/test_import_metapath1_onedir_0/dist/test_source/test_source failed with return-code 1.
FAILED tests/functional/test_import.py::test_import_metapath1[onefile] - Failed: Running exe /tmp/pytest-of-runner/pytest-0/popen-gw1/test_import_metapath1_onefile_0/dist/test_source failed with return-code 1.

I think these are caused by lack of find_spec method on the custom VendoredImporter that the test use. The custom importer seems to be taken from setuptools, so we can try syncing with its latest version.

.github/workflows/ci.yml

bwoodsend · 2023-06-04T19:15:02Z

I'll deal with the extraneous fizz.

rokm · 2023-06-04T19:26:58Z

Looks like it's also time to implement proper dependency scanning for Tcl/Tk libraries in splash, because the tcl86t.dll shipped with Windows build of 3.12b1 depends on zlib1.dll and we fail to pick that up...

bwoodsend · 2023-06-04T21:39:42Z

This fizz thing looks harmless.

The code it's scanning for calls to functions is:

def foo():
    bar()
    return [fizz(3) for i in range(10)]

Until now, the body of a comprehension loop (i.e. the fizz(3) bit) was a separate code object nested inside the first but now it has been inlined so that this test fails but as soon as you switch over to recursive scanning (which PyInstaller uses exclusively) you get the same results.

Before:

(Pdb++) foo_code.co_names
('bar', 'range')  # <-- no fizz()
(Pdb++) foo_code.co_consts[1]
<code object <listcomp> at 0x7fd4487b1630, file "<no file>", line 4>  # <-- nested code object
(Pdb++) foo_code.co_consts[1].co_names
('fizz',)  # <-- there's fizz!

Verses after:

(Pdb) foo_code.co_names
('bar', 'range', 'fizz')
(Pdb) foo_code.co_consts
(None, 10, 3)

bwoodsend · 2023-06-05T14:04:49Z

Does it still make sense to be dragging distutils support to 3.12 given that the package only sort of exists now?

rokm · 2023-06-05T14:11:14Z

Does it still make sense to be dragging distutils support to 3.12 given that the package only sort of exists now?

I think it does - a simple import distutils program should trigger collection of setuptools' version under 3.12, if available.

But we have whole summer to figure out the best way to handle this (i.e., the one with minimal changes compared to how things work right now and what our current hooks are made to handle)... despite that "high-priority" label

bwoodsend · 2023-06-14T21:07:00Z

Let's merge this now rather than wait for the official 3.12 release? There should be very little difference between Python's beta releases and the real ones.

rokm · 2023-06-14T21:47:04Z

I suppose we could, if we're fine with depending on development version of pywin32-ctypes for python 3.12.

Whatever issues may come up between now and October, we would have to deal with either way, and earlier exposure means faster feedback.

Python 3.12 removed LOAD_METHOD opcode; LOAD_ATTR now behaves like LOAD_METHOD if the low bit of its oparg is set. This means that LOAD_ATTR pushes namei>>1 on stack. (This change is similar to what python 3.11 did with LOAD_GLOBAL).

The latest release of pywin32-ctypes (0.2.0) is incompatible with python 3.12, so we need to install the version from git main branch. This version updated the custom module path finder to be compliant with PEP 451.

Update the `VendorImporter` used by `test_import_metapath1` to the version available in the latest release of `setuptools` (67.8.0). This version is compatible with PEP 451, and hence python 3.12.

In Python <= 3.11, the compiled bytecode for [foo() for i in range(10)] is two code objects - the code object for the `foo()` bit being embedded in the code object for everything else. In Python >= 3.12, it's one flat code object. This makes it unsuitable as a test for scanning nested code objects since it's only the correct situation on older Pythons. Swap out the function call in the body of a comprehension loop with calling a function in a lambda which still does the right thing with any Python version.

Starting with python 3.12, `distutils` is not part of stdlib anymore, and it is provided solely by `setuptools`. Due to limitations of the modulegraph, the latter cannot find and properly analyze the `setuptools`-provided `distutils` (and its submodules) under mapped names, so we need the pre-safe-import-module hook to mark the `distutils` as a run-time package. Marking the package as a run-time one means that we also need to mark its submodules and subpackages as run-time ones.

bwoodsend

Well I approve of the bits you two wrote.

rokm · 2023-06-17T14:29:23Z

We still need to update the README and add a news fragment, then it should be ready for merge.

danyeaw · 2023-06-17T23:17:27Z

Thanks @bwoodsend and @rokm for the teamwork on this one!

danyeaw force-pushed the python-3.12 branch from ec80cc7 to 3bcd71c Compare May 31, 2023 02:34

Legorooj added feature Feature request @high labels Jun 1, 2023

rokm mentioned this pull request Jun 3, 2023

Replace deprecated imp with importlib #7676

Merged

bwoodsend force-pushed the python-3.12 branch 2 times, most recently from 3bcd71c to 58f9dbb Compare June 4, 2023 10:45

rokm reviewed Jun 4, 2023

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

rokm force-pushed the python-3.12 branch from 9360021 to d427196 Compare June 4, 2023 11:16

bwoodsend force-pushed the python-3.12 branch from cf2b140 to 34b5b31 Compare June 4, 2023 23:17

rokm force-pushed the python-3.12 branch from ebde471 to d6da320 Compare June 5, 2023 16:06

rokm force-pushed the python-3.12 branch from a143a4a to 088ffac Compare June 13, 2023 20:40

bwoodsend mentioned this pull request Jun 14, 2023

Let there be symbolic links #7619

Merged

danyeaw and others added 4 commits June 14, 2023 23:26

setup: Unblock installation on Python 3.12.

3af96ec

ci: Enable testing with Python 3.12-dev.

855da04

Update opcodes for Python 3.12

9a56ff0

Get version with importlib metadata

c59fb76

bwoodsend force-pushed the python-3.12 branch from 088ffac to d7fada4 Compare June 14, 2023 22:29

rokm and others added 5 commits June 15, 2023 00:39

setup: python 3.12 requires pywin32-ctypes from git main

8d178a4

The latest release of pywin32-ctypes (0.2.0) is incompatible with python 3.12, so we need to install the version from git main branch. This version updated the custom module path finder to be compliant with PEP 451.

tests: fix test_import_metapath1 for python 3.12

bf6fbad

Update the `VendorImporter` used by `test_import_metapath1` to the version available in the latest release of `setuptools` (67.8.0). This version is compatible with PEP 451, and hence python 3.12.

rokm force-pushed the python-3.12 branch from d7fada4 to 97ed81c Compare June 14, 2023 22:40

bwoodsend approved these changes Jun 17, 2023

View reviewed changes

Add Python 3.12 to README, add news fragment

3b7d9d7

danyeaw marked this pull request as ready for review June 17, 2023 18:01

rokm approved these changes Jun 17, 2023

View reviewed changes

bwoodsend enabled auto-merge (rebase) June 17, 2023 20:22

bwoodsend merged commit 5b85fcc into pyinstaller:develop Jun 17, 2023

danyeaw deleted the python-3.12 branch June 17, 2023 23:17

github-actions bot locked as resolved and limited conversation to collaborators Aug 17, 2023

Uh oh!

Conversation

danyeaw commented May 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bwoodsend commented May 31, 2023

Uh oh!

danyeaw commented Jun 1, 2023

Uh oh!

bwoodsend commented Jun 2, 2023

Uh oh!

rokm commented Jun 3, 2023

Uh oh!

bwoodsend commented Jun 4, 2023

Uh oh!

bwoodsend commented Jun 4, 2023

Uh oh!

rokm commented Jun 4, 2023

Uh oh!

Uh oh!

bwoodsend commented Jun 4, 2023

Uh oh!

rokm commented Jun 4, 2023

Uh oh!

bwoodsend commented Jun 4, 2023

Uh oh!

bwoodsend commented Jun 5, 2023

Uh oh!

rokm commented Jun 5, 2023

Uh oh!

bwoodsend commented Jun 14, 2023

Uh oh!

rokm commented Jun 14, 2023

Uh oh!

bwoodsend left a comment

Choose a reason for hiding this comment

Uh oh!

rokm commented Jun 17, 2023

Uh oh!

danyeaw commented Jun 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

danyeaw commented May 31, 2023 •

edited

Loading