Skip to content

pkgutil.iter_modules is broken on MacOS #7884

@gentlegiantJGC

Description

@gentlegiantJGC

Description of the issue

I am trying to build an application on MacOS that uses pkguilt.iter_modules to dynamically find and load modules however it is not finding any modules.
The modules are there and can be imported with importlib.import_module.
Note that I am using PyInstaller with FBS but I don't believe the latter changes anything that would effect this.

The following is an example project structure

my_package
    my_module_1
    my_module_2
    my_module_3

Here is the code that does not work

import importlib
import pkgutil
m = importlib.import_module("my_package")
print("Found nested modules", list(pkgutil.iter_modules(m.__path__, m.__name__ + ".")))

The last line prints an empty list when built with PyInstaller.

I have done a bit of a dive into the code and I believe this is a rather complex mixup with paths and symbolic links.
https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py

It looks like #6539 had a go at supporting symbolic links but it only resolves symbolic links up to the _MEIPASS directory.

On MacOS _MEIPASS is equal to "/path/to/my_package.app/Contents/MacOS/"

my_package.__path__ is equal to ["/path/to/my_package.app/Contents/MacOS/my_package"]

/path/to/my_package.app/Contents/MacOS/my_package is a symbolic link to /path/to/my_package.app/Contents/Resources/my_package when there are data files present in that package.

The iter_modules hook expands symbolic links in the path variable which converts "/path/to/my_package.app/Contents/MacOS/my_package" to "/path/to/my_package.app/Contents/Resources/my_package"

The following code then skips over the directory because it does not start with SYS_PREFIX

                if not pkg_path.startswith(SYS_PREFIX):
                    # If the path does not start with sys._MEIPASS, it cannot be a bundled package.
                    continue

I don't know how this should be implemented but we need to resolve the expected path relative to the _MEIPASS directory and then expand symbolic links and only then skip if the package path does not start with the directory.

Context information (for bug reports)

  • Output of pyinstaller --version: 5.13.0
  • Version of Python: Python 3.11.4 (main, Aug 22 2023, 11:47:24) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
  • Platform: OS X
  • How you installed Python: pyenv
  • Did you also try this on another platform? Does it work there? This only effects MacOS as far as I am aware

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions