Skip to content

zipapp virtual environment creation fails if zipapp path is symlinked #1949

New issue
New issue
Closed
Closed
zipapp virtual environment creation fails if zipapp path is symlinked#1949
Assignees
petamas
Labels
bughelp-wanted
@andyneff

Description

@andyneff
andyneff
opened on Sep 25, 2020

Issue

On a fresh new system, trying to use the zipapp to setup a virtualenv when the pyz files is referenced by a symlink, it fails saying KeyError: "There is no item named 'nv/create/via_global_ref/_virtualenv.py' in the archive"

But it works on my machine?

Yes, if you ever used the virtualenv (zipapp?) successfully, it creates files in ~/Library/Application Support/virtualenv (on macOS) or ~/.local/share/virtualenv/ (on Linux) and the existence of these files (unzip and wheel) makes the zipapp work. By deleting that directory, you should be able to reproduce the problem. This was first discovered on travis-ci, where the machine is new every time.

Environment

  • OS: macOS 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64.
  • Python 3.7.7. (from brew), also tried 3.7.9

Also reproduced on

  • OS: fedora:31
  • Python 3.7.9

Output of the virtual environment creation

bash-3.2$ (cd /var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo; python3 /var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz -vvv --with-traceback /var/folders/g0/944401t11gz1jl7vxt
nzqs_m0001yj/T/foo/pipenv-2020.6.2)
206 setup logging to NOTSET [DEBUG report:43]
219 find interpreter for spec PythonSpec(path=/usr/local/opt/python/bin/python3.7) [INFO builtin:44]
219 proposed PythonInfo(spec=CPython3.7.7.final.0-64, exe=/usr/local/opt/python/bin/python3.7, platform=darwin, version='3.7.7 (default, Mar 10 2020, 15:43:33) \n[Clang 11.0.0 (clang-1100.0.33.17)]', encoding_fs_io=utf-8-utf-8) [INFO builtin:50]
219 accepted PythonInfo(spec=CPython3.7.7.final.0-64, exe=/usr/local/opt/python/bin/python3.7, platform=darwin, version='3.7.7 (default, Mar 10 2020, 15:43:33) \n[Clang 11.0.0 (clang-1100.0.33.17)]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:52]
224 filesystem is not case-sensitive [DEBUG info:28]
263 create virtual environment via CPython3Posix(dest=/private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2, clear=False, global=False) [INFO session:52]
263 write /private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2/pyvenv.cfg [DEBUG pyenv_cfg:34]
263     home = /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
263     implementation = CPython [DEBUG pyenv_cfg:38]
263     version_info = 3.7.7.final.0 [DEBUG pyenv_cfg:38]
263     virtualenv = 20.0.31 [DEBUG pyenv_cfg:38]
263     include-system-site-packages = false [DEBUG pyenv_cfg:38]
264     base-prefix = /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
264     base-exec-prefix = /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
264     base-executable = /usr/local/opt/python/bin/python3.7 [DEBUG pyenv_cfg:38]
264 remove file /private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2/bin/python [DEBUG _sync:38]
264 symlink /usr/local/opt/python/bin/python3.7 to /private/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/foo/pipenv-2020.6.2/bin/python [DEBUG _sync:44]
265 Attempting to acquire lock 4396469136 on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [DEBUG filelock:270]
265 Lock 4396469136 acquired on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [INFO filelock:274]
265 extract /var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/_virtualenv.py to /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py [DEBUG zipapp:19]
267 Attempting to release lock 4396469136 on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [DEBUG filelock:315]
267 Lock 4396469136 released on /Users/a-andy/Library/Application Support/virtualenv/unzip/20.0.31/_virtualenv.py.lock [INFO filelock:318]
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/__main__.py", line 168, in <module>
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/__main__.py", line 164, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/__main__.py", line 16, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/run/__init__.py", line 28, in cli_run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/run/session.py", line 46, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/run/session.py", line 53, in _create
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/creator.py", line 157, in run
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/via_global_self_do.py", line 86, in create
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 88, in create
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 91, in install_patch
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/builtin/cpython/cpython3.py", line 27, in env_patch_text
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/create/via_global_ref/api.py", line 102, in env_patch_text
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/app_data/base.py", line 54, in ensure_extracted
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/app_data/via_disk_folder.py", line 81, in extract
  File "/var/folders/g0/944401t11gz1jl7vxtnzqs_m0001yj/T/tmp.7cHdIsKw/virtualenv.pyz/virtualenv/util/zipapp.py", line 22, in extract

  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/zipfile.py", line 1431, in getinfo
    'There is no item named %r in the archive' % name)
KeyError: "There is no item named 'nv/create/via_global_ref/_virtualenv.py' in the archive"

Note: Notice /virtuale is missing from the KeyError, the exact length /private which is the difference between the TMPDIR and the real path of the TMPDIR on macOS

On Fedora, the last line becomes:

KeyError: "There is no item named '/virtualenv/create/via_global_ref/_virtualenv.py' in the archive"

Steps to reproduce

  1. docker run -it --rm fedora:31
  2. mkdir -p /tmp/foo/bar
  3. ln -s /tmp/foo/bar /tmp/foo/bar2
  4. curl -L -o /tmp/foo/bar/virtualenv.pyz https://bootstrap.pypa.io/virtualenv/3.7/virtualenv.pyz
  5. python3 /tmp/foo/bar2/virtualenv.pyz /tmp/foo/blah

Here are some one-liners to reproduce the error

docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" /tmp/blah'

docker run -it --rm fedora:32 bash -c 'REAL=/tmp/foo/car; SYM=/tmp/foo/bbb2; mkdir -p "${REAL}"; ln -s "${REAL}" "${SYM}"; curl -L -o "${REAL}/virtualenv.pyz" https://bootstrap.pypa.io/virtualenv/3.8/virtualenv.pyz; python3 "${SYM}/virtualenv.pyz" /tmp/blah'

Not all values of REAL/SYM fail. It appears if the length of the symlink path and realpath differ, it fails. If they are the exact same length, no error occurs

Fail

REAL SYM
/tmp/foo/bar /tmp/bar/bar2
/tmp/foo/bar /tmp/foo/bc
/tmp/foo/car /tmp/foo/bb
/tmp/bb /tmp/aaa

Does not fail

REAL SYM
/tmp/foo /tmp/bar
/tmp/foo/bar /tmp/foo/abc
/tmp/foo/car /tmp/foo/bbb

Metadata

Metadata

Assignees

Labels

bughelp-wanted

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions