{Packaging} Bump PyWin32 to 305 by jiasli · Pull Request #24982 · Azure/azure-cli

jiasli · 2022-12-26T08:08:01Z

Change

Bump PyWin32 (to >= 303) so that Azure CLI no longer loads pywintypes310.dll or pythoncom310.dll from the current working directory (CWD), but always from C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\pywin32_system32\.

Context

`LoadLibrary` and `LoadLibraryEx` Win32 API

According to https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order#standard-search-order-for-desktop-applications, even if SafeDllSearchMode is enabled and it places the user's current directory later in the search order, the current directory is still searched by LoadLibrary and LoadLibraryEx. LoadLibraryEx supports additional flags such as LOAD_LIBRARY_SEARCH_DEFAULT_DIRS and LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR. When these flags are specified, directories in the standard search path are not searched, so the current directory is no longer searched.

Python 3.8's DLL loading mechanism change

https://docs.python.org/3/whatsnew/3.8.html#changes-in-the-python-api

DLL dependencies for extension modules and DLLs loaded with ctypes on Windows are now resolved more securely. Only the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies. Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library. Note that Windows 7 users will need to ensure that Windows Update KB2533623 has been installed (this is also verified by the installer). (Contributed by Steve Dower in bpo-36085.)

bpo-36085 is migrated to python/cpython#80266.

In summary, when loading DLLs:

in Python 3.7, LoadLibraryExW is called with LOAD_WITH_ALTERED_SEARCH_PATH.
in Python 3.8, LoadLibraryExW is called with LOAD_LIBRARY_SEARCH_DEFAULT_DIRS and LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR, more securely (bpo-36085: Enable better DLL resolution on Windows python/cpython#12302)

PyWin32's DLL loading mechanism change

Since version 303, PyWin32's _win32sysloader switched from calling LoadLibrary to LoadLibraryEx with the same LoadLibraryEx flags as Python itself mhammond/pywin32#1794.

Therefore, PyWin32's change causes the current directory no longer to be searched, but its change log doesn't mention this:

https://github.com/mhammond/pywin32/blob/d73b0200eba81fa551b90d15cf5da097f4197f8f/CHANGES.txt#L92-L93

Tweaks to how DLLs are loaded and our installation found, which should improve virtualenv support and version mismatch issues (#1787, #1794)

Testing guide

The test can be done with system Python, without Azure CLI's embedded Python.

With PyWin32 302, DLL can be loaded from the current directory:

> cd D:\temp
> pip install pywin32==302
> mv $env:LOCALAPPDATA\Programs\Python\Python310\Lib\site-packages\pywin32_system32\pythoncom310.dll D:\temp
> python -c "import win32com"

With PyWin32 305, DLL cannot be loaded from the current directory and results in an ImportError:

> cd D:\temp
> pip install pywin32==305
> mv $env:LOCALAPPDATA\Programs\Python\Python310\Lib\site-packages\pywin32_system32\pythoncom310.dll D:\temp
> python -c "import win32com"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\jiasli\AppData\Local\Programs\Python\Python310\lib\site-packages\win32com\__init__.py", line 6, in <module>
    import pythoncom
  File "C:\Users\jiasli\AppData\Local\Programs\Python\Python310\lib\site-packages\pythoncom.py", line 4, in <module>
    pywintypes.__import_pywin32_system_module__("pythoncom", globals())
  File "C:\Users\jiasli\AppData\Local\Programs\Python\Python310\lib\site-packages\win32\lib\pywintypes.py", line 105, in __import_pywin32_system_module__
    raise ImportError("No system module '%s' (%s)" % (modname, filename))
ImportError: No system module 'pythoncom' (pythoncom310.dll)

yonzhan · 2022-12-26T09:27:21Z

Bump PyWin32 to fix vulnerability issue

jiasli · 2023-01-03T05:20:38Z

Why/How `pythoncom310.dll` is loaded

I put these lines at site-packages\win32\lib\pywintypes.py:L30:

print(filename)
import traceback
traceback.print_stack()

Result:

> az --version
pywintypes310.dll
  File "runpy.py", line 196, in _run_module_as_main
  File "runpy.py", line 86, in _run_code
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/__main__.py", line 14, in <module>
  File "<frozen importlib._bootstrap>", line 1078, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/telemetry.py", line 19, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/telemetry/__init__.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\portalocker/__init__.py", line 4, in <module>
  File "<frozen importlib._bootstrap>", line 1078, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\portalocker/portalocker.py", line 12, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\win32\lib\pywintypes.py", line 120, in <module>
    __import_pywin32_system_module__("pywintypes", globals())
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\win32\lib\pywintypes.py", line 32, in __import_pywin32_system_module__
    traceback.print_stack()
pythoncom310.dll
  File "runpy.py", line 196, in _run_module_as_main
  File "runpy.py", line 86, in _run_code
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/__main__.py", line 39, in <module>
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\azure\cli\core\__init__.py", line 895, in get_default_cli
    from azure.cli.core.azlogging import AzCliLogging
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/azlogging.py", line 30, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 25, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/extension/__init__.py", line 11, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 945, in _find_spec
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\_distutils_hack/__init__.py", line 97, in find_spec
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\_distutils_hack/__init__.py", line 108, in spec_for_distutils
  File "importlib\__init__.py", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\setuptools/__init__.py", line 16, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\setuptools/version.py", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\pkg_resources/__init__.py", line 80, in <module>
  File "<frozen importlib._bootstrap>", line 1078, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\pkg_resources/extern/__init__.py", line 52, in create_module
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\pkg_resources/extern/__init__.py", line 37, in load_module
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\Python\Lib\site-packages\pkg_resources/_vendor/appdirs.py", line 560, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\win32com/__init__.py", line 6, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\pythoncom.py", line 4, in <module>
  File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\win32\lib\pywintypes.py", line 32, in __import_pywin32_system_module__
    traceback.print_stack()

The call stack indicates pythoncom310.dll was loaded by site-packages\pkg_resources/_vendor/appdirs.py:L560:

if system == "win32":
    try:
        import win32com.shell
        _get_win_folder = _get_win_folder_with_pywin32
    except ImportError:
        try:
            from ctypes import windll
            _get_win_folder = _get_win_folder_with_ctypes
        except ImportError:
            try:
                import com.sun.jna
                _get_win_folder = _get_win_folder_with_jna
            except ImportError:
                _get_win_folder = _get_win_folder_from_registry

That's explains why win32com is imported.

The dev environment doesn't have this issue because the dev environment pins setuptools to 52.0.0:

azure-cli/requirements.txt

Line 2 in 99c5fb4

setuptools==52.0.0

which doesn't injects _distutils_hack. But the Windows MSI bundles latest setuptools 65.6.3 and results in the following call chain:

setuptools injects _distutils_hack
_distutils_hack calls setuptools
setuptools calls pkg_resources ([BUG] Importing setuptools is slow and takes nearly half a second pypa/setuptools#3441)
pkg_resources imports win32com

pywin32

75c8ca1

ghost requested review from kairu-ms, wangzelin007 and yonzhan December 26, 2022 08:08

ghost added the Auto-Assign Auto assign by bot label Dec 26, 2022

ghost assigned jiasli Dec 26, 2022

ghost added the Packaging label Dec 26, 2022

jiasli requested a review from bebound December 26, 2022 08:23

yonzhan added this to the Dec 2022 (2023-01-10) milestone Dec 26, 2022

yonzhan requested a review from jsntcy December 26, 2022 09:28

bebound approved these changes Dec 27, 2022

View reviewed changes

jsntcy approved these changes Jan 3, 2023

View reviewed changes

yonzhan approved these changes Jan 3, 2023

View reviewed changes

jiasli merged commit 8e7b30b into Azure:dev Jan 3, 2023

jiasli deleted the pywin32 branch January 3, 2023 09:34

jiasli mentioned this pull request Jan 4, 2023

{Packaging} Bump setuptools to 52.0.0 #16655

Merged

bebound mentioned this pull request Jan 4, 2023

Support Python 3.11 #24494

Closed

4 tasks

jiasli mentioned this pull request Jan 4, 2023

{CI} Bump setuptools from 52.0.0 to 65.5.1 #24992

Merged

avgale pushed a commit to avgale/azure-cli that referenced this pull request Aug 24, 2023

{Packaging} Bump PyWin32 to 305 (Azure#24982)

186c104

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{Packaging} Bump PyWin32 to 305#24982

{Packaging} Bump PyWin32 to 305#24982
jiasli merged 1 commit intoAzure:devfrom
jiasli:pywin32

jiasli commented Dec 26, 2022 •

edited

Loading

Uh oh!

yonzhan commented Dec 26, 2022 •

edited

Loading

Uh oh!

jiasli commented Jan 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

jiasli commented Dec 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change

Context

LoadLibrary and LoadLibraryEx Win32 API

Python 3.8's DLL loading mechanism change

PyWin32's DLL loading mechanism change

Testing guide

Uh oh!

yonzhan commented Dec 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jiasli commented Jan 3, 2023

Why/How pythoncom310.dll is loaded

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jiasli commented Dec 26, 2022 •

edited

Loading

`LoadLibrary` and `LoadLibraryEx` Win32 API

yonzhan commented Dec 26, 2022 •

edited

Loading

Why/How `pythoncom310.dll` is loaded