Windows: attempt to preserve DLL parent directory structure#7028
Windows: attempt to preserve DLL parent directory structure#7028rokm merged 3 commits intopyinstaller:developfrom
Conversation
For DLLs collected from site-packages directories during the binary dependency analysis, attempt to preserve the parent directory structure instead of collecting them to the top-level directory. Ultimately, this behavior should be fixed on all OSes, but for macOS and linux, we need to first implement support for symbolic links.
Due to preservation of the directory layout, the pywintypes DLLs (pywintypes3X.dll and pythoncom3X.dll) are not collected into top-level application directory anymore, but rather into the pywin32_system32 sub-directory. Add this sub-directory to sys.path to keep the pywintypes' loader code happy...
5a2fd4a to
0e74a90
Compare
|
I don't quite follow why we need symlinks on the unix platforms. Is it because we're currently finding these dependent |
It is really necessary only for macOS due to path rewriting. We might get rid of that altogether at a later stage, but during transition period, the idea is to make symlinks back to top-level directory to satisfy the assumptions of the path rewriting (similarly to how On linux, this should not be necessary (as existing relative rpaths should already do the trick, as long as relative path relations between involved libraries are preserved), but it is still nice to have as a fallback in case it is needed for some corner case. (And yes, the overreaching problem here is that on POSIX systems, we cannot manipulate linker search paths dynamically, as we can on Windows by modifying
Yeah, but that approach is something I am not particularly looking forward to, because it is essentially the same thing as our macOS path rewriting. Plus, it will introduce a new dependency ( |
…DLLs In helpers that explicitly collect DLLs from Qt binary path (`get_qt_binaries`, `get_qt_network_ssl_binaries`), preserve the directory structure if the DLLs are collected from the python package directory (e.g., PyPI wheels). The change applies only to Windows, as that is the only platform where we explicitly collect (some) DLLs from the Qt binary path.
0e74a90 to
81bee58
Compare
| dll_file_paths = glob.glob(dll_path) | ||
| for dll_file_path in dll_file_paths: | ||
| to_include.append((dll_file_path, dst_dll_path)) | ||
| dll_file_path = pathlib.Path(dll_file_path).resolve() |
There was a problem hiding this comment.
Based on earlier discussion about Path.resolve(), I've now added the call here as well. I don't think it should make much difference, but it makes the changes in the PR consistent (i.e., all added pathlib.Path() instances are explicitly resolved now).
| dll_names = ('libeay32.dll', 'ssleay32.dll', 'libssl-1_1-x64.dll', 'libcrypto-1_1-x64.dll') | ||
| binaries = [] | ||
| for location in locations: | ||
| location = pathlib.Path(location).resolve() |
There was a problem hiding this comment.
Added a Path.resolve() call here as well, as per above comment.
|
Can not import PyAV after upgrade PyInstaller to this commit. logs: |
What version of python and what version of PyAV? And how did you install them; is this python.org python and pip, anaconda, etc.? FWIW, running test_av (which is the same as your sample code) across all supported python versions and OSes, succeeds with PyAV 9.1, 9.1.1, and 9.2 (these are PyPI wheels). |
|
@rokm Environment is:
code: import av
print('hello')
print(av.__version__)pack: logs:
|
|
Hmmm, I can indeed reproduce this if I mix But I don't think we'll be doing anything about it, because this is purely anaconda-induced problem: It looks like by default (at least on that python version), anaconda python blocks Therefore, """"""# start delvewheel patch
def _delvewheel_init_patch_0_0_21():
import os
import sys
libs_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir, 'av.libs'))
if sys.version_info[:2] >= (3, 8):
conda_workaround = sys.version_info[:3] < (3, 9, 9) and os.path.exists(os.path.join(sys.base_prefix, 'conda-meta'))
if conda_workaround:
# backup the state of the environment variable CONDA_DLL_SEARCH_MODIFICATION_ENABLE
conda_dll_search_modification_enable = os.environ.get('CONDA_DLL_SEARCH_MODIFICATION_ENABLE')
os.environ['CONDA_DLL_SEARCH_MODIFICATION_ENABLE'] = '1'
os.add_dll_directory(libs_dir)
if conda_workaround:
# restore the state of the environment variable CONDA_DLL_SEARCH_MODIFICATION_ENABLE
if conda_dll_search_modification_enable is None:
os.environ.pop('CONDA_DLL_SEARCH_MODIFICATION_ENABLE', None)
else:
os.environ['CONDA_DLL_SEARCH_MODIFICATION_ENABLE'] = conda_dll_search_modification_enable
else:
from ctypes import WinDLL
with open(os.path.join(libs_dir, '.load-order-av-9.2.0')) as file:
load_order = file.read().split()
for lib in load_order:
WinDLL(os.path.join(libs_dir, lib))
_delvewheel_init_patch_0_0_21()
del _delvewheel_init_patch_0_0_21
# end delvewheel patchUnfortunately, the frozen application does not contain 'conda-meta' directory, and therefore that work-around is not enabled when So if you want to mix conda python and PyPI Or switch to anaconda python >= 3.9.9, which (based on the |
|
@rokm But the built exe prints nothing after adding the environment variable |
|
You are right. What about: import sys
if getattr(sys, "frozen", False):
os.environ["PATH"] = os.path.join(sys._MEIPASS, "av.libs") + os.pathsep + os.environ["PATH"]
import av
print('hello')
print(av.__version__)Based on another work-around found in # Some Python versions distributed by Conda have a buggy `os.add_dll_directory`
# which prevents binary wheels from finding the FFmpeg DLLs in the `av.libs`
# directory. We work around this by adding `av.libs` to the PATH.
if (
os.name == "nt"
and sys.version_info[:2] in ((3, 8), (3, 9))
and os.path.exists(os.path.join(sys.base_prefix, "conda-meta"))
):
os.environ["PATH"] = (
os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir, "av.libs"))
+ os.pathsep
+ os.environ["PATH"]
) |
|
@rokm Great! |
When collecting a DLL that was discovered via link-time dependency analysis of a collected binary/extension, attempt to preserve
its parent directory structure instead of collecting it into application's top-level directory.
This aims to preserve the parent directory structure of DLLs bundled with python packages in PyPI wheels, while the DLLs
collected from system directories (as well as from
Library\bindirectory of the Anaconda's environment) are still collected intotop-level application directory (because there is no directory structure to preserve there).
Ultimately, this behavior should be fixed on all OSes, but for macOS and linux, we need to first implement support for symbolic
links. On Windows, we won't be using symbolic links anyway, so we can make the change in advance and see (and fix) the fallout this causes.
Besides, our hand is kind of forced by #6924 and the fix for it, #6925. There, we improve the tracking of DLL search paths for binary dependency analysis in order to be able to find more DLLs. Previously, similar problems were handled by hooks that collected those "unreachable" DLLs, and the hooks typically preserved the DLLs parent directory structure. On the other hand, prior to this PR, binary dependency analysis always collected the discovered DLLs into top-level directory. So a DLL discovered and collected via both mechanisms ended up duplicated; and because more DLLs can be discovered by binary dependency analysis due to #6925, this leads to duplication. But, after changes in this PR, both mechanisms should collect into the original sub-directory, and the duplication will be handled on the TOC level.
This does introduce another type of DLL duplication, if multiple packages bundle a copy of the same DLL (e.g.,
vcruntime140.dll). I suppose if this becomes a problem, we could refine the mechanism to collect well-known-and-common DLLs into top-level directory again. But on the other hand, such duplication might already happen when DLL collection is done by a hook, and in some cases, the package expects to find that particular DLL copy in its library (sub)directory anyway (e.g., packages usingdelvewheelon python 3.7 due to loading via load order file).