Skip to content

Installation issue: python+ctypes #29478

Closed
#29480
Closed
Installation issue: python+ctypes#29478
#29480
Labels
build-error
@sethrj

Description

@sethrj
sethrj
opened on Mar 12, 2022

Steps to reproduce the issue

$ spack spec -I python
Input spec
--------------------------------
 -   python

Concretized
--------------------------------
[+]  python@3.9.10%gcc@8.5.0+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,4c24573,f2fd060 arch=linux-centos7-x86_64
[+]      ^bzip2@1.0.8%gcc@8.5.0~debug~pic+shared arch=linux-centos7-x86_64
[+]          ^diffutils@3.8%gcc@8.5.0 arch=linux-centos7-x86_64
[+]              ^libiconv@1.16%gcc@8.5.0 libs=shared,static arch=linux-centos7-x86_64
[+]      ^expat@2.4.6%gcc@8.5.0+libbsd arch=linux-centos7-x86_64
[+]          ^libbsd@0.11.5%gcc@8.5.0 arch=linux-centos7-x86_64
[+]              ^libmd@1.0.4%gcc@8.5.0 arch=linux-centos7-x86_64
[+]      ^gdbm@1.19%gcc@8.5.0 arch=linux-centos7-x86_64
[+]          ^readline@8.1%gcc@8.5.0 arch=linux-centos7-x86_64
[+]              ^ncurses@6.2%gcc@8.5.0~symlinks+termlib abi=none arch=linux-centos7-x86_64
[+]                  ^pkgconf@1.8.0%gcc@8.5.0 arch=linux-centos7-x86_64
[+]      ^gettext@0.21%gcc@8.5.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-centos7-x86_64
[+]          ^libxml2@2.9.12%gcc@8.5.0~python arch=linux-centos7-x86_64
[+]              ^xz@5.2.5%gcc@8.5.0~pic libs=shared,static arch=linux-centos7-x86_64
[+]              ^zlib@1.2.11%gcc@8.5.0+optimize+pic+shared arch=linux-centos7-x86_64
[+]          ^tar@1.34%gcc@8.5.0 arch=linux-centos7-x86_64
[+]      ^libffi@3.4.2%gcc@8.5.0 arch=linux-centos7-x86_64
[+]      ^openssl@1.1.1m%gcc@8.5.0~docs certs=system arch=linux-centos7-x86_64
[+]          ^perl@5.34.0%gcc@8.5.0+cpanm+shared+threads arch=linux-centos7-x86_64
[+]              ^berkeley-db@18.1.40%gcc@8.5.0+cxx~docs+stl patches=b231fcc arch=linux-centos7-x86_64
[+]      ^sqlite@3.37.2%gcc@8.5.0+column_metadata+dynamic_extensions+fts~functions+rtree arch=linux-centos7-x86_64
[+]      ^util-linux-uuid@2.37.4%gcc@8.5.0 arch=linux-centos7-x86_64

Error message

Python appears to install successfully, but downstream packages fail, e.g. py-setuptools:

Error message
```
==> py-setuptools: Executing phase: 'install'
==> [2022-03-12-10:39:09.181041] '/projects/spack/opt/spack/gcc-8.5.0/python/6z6bbfg/bin/python3.9' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/projects/spack/opt/spack/gcc-8.5.0/py-setuptools/26gyn5x' '.'
Using pip 21.3.1 from /projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip (python 3.9)
Non-user install due to --prefix or --target option
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-ephem-wheel-cache-p5vr3e0t
Created temporary directory: /tmp/pip-req-tracker-iv7wygbd
Initialized build tracking at /tmp/pip-req-tracker-iv7wygbd
Created build tracker: /tmp/pip-req-tracker-iv7wygbd
Entered build tracker: /tmp/pip-req-tracker-iv7wygbd
Created temporary directory: /tmp/pip-install-2zg9v9l0
Processing /tmp/s3j/spack-stage/spack-stage-py-setuptools-59.4.0-26gyn5xre2zssmqsn2qqurxbbooxgssz/spack-src
  Added file:///tmp/s3j/spack-stage/spack-stage-py-setuptools-59.4.0-26gyn5xre2zssmqsn2qqurxbbooxgssz/spack-src to build tracker '/tmp/pip-req-tracker-iv7wygbd'
  Created temporary directory: /tmp/pip-modern-metadata-n1k48pa1
  Preparing metadata (pyproject.toml): started
  Running command /projects/spack/opt/spack/gcc-8.5.0/python/6z6bbfg/bin/python3.9 /projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpte_wdk3q
  Preparing metadata (pyproject.toml): finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
    status = run_func(*args)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 338, in run
    requirement_set = resolver.resolve(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 468, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 430, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 201, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 281, in __init__
    super().__init__(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 292, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 546, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 58, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py", line 49, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 556, in prepare_metadata
    self.metadata_directory = generate_metadata(
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_internal/operations/build/metadata.py", line 28, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir)
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py", line 188, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_vendor/pep517/wrappers.py", line 332, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/projects/spack/opt/spack/gcc-8.5.0/py-pip/m2bllhv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 89, in _build_backend
    obj = import_module(mod_path)
  File "/projects/spack/opt/spack/gcc-8.5.0/python/6z6bbfg/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 1030, in _gcd_import
  File "", line 1007, in _find_and_load
  File "", line 972, in _find_and_load_unlocked
  File "", line 228, in _call_with_frames_removed
  File "", line 1030, in _gcd_import
  File "", line 1007, in _find_and_load
  File "", line 986, in _find_and_load_unlocked
  File "", line 680, in _load_unlocked
  File "", line 850, in exec_module
  File "", line 228, in _call_with_frames_removed
  File "/tmp/s3j/spack-stage/spack-stage-py-setuptools-59.4.0-26gyn5xre2zssmqsn2qqurxbbooxgssz/spack-src/setuptools/__init__.py", line 18, in 
    from setuptools.dist import Distribution
  File "/tmp/s3j/spack-stage/spack-stage-py-setuptools-59.4.0-26gyn5xre2zssmqsn2qqurxbbooxgssz/spack-src/setuptools/dist.py", line 38, in 
    from setuptools import windows_support
  File "/tmp/s3j/spack-stage/spack-stage-py-setuptools-59.4.0-26gyn5xre2zssmqsn2qqurxbbooxgssz/spack-src/setuptools/windows_support.py", line 2, in 
    import ctypes
  File "/projects/spack/opt/spack/gcc-8.5.0/python/6z6bbfg/lib/python3.9/ctypes/__init__.py", line 8, in 
    from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'

Looking back at the python build logs, it silently hid the failure of the ctypes module:

Error message
*** WARNING: renaming "_ctypes" since importing it failed: build/lib.linux-x86_64-3.9/_ctypes.cpython-39-x86_64-linux-gnu.so: undefined symbol: ffi_prep_cif

Python build finished successfully!

The necessary bits to build these optional modules were not found:

_bz2                  _curses               _curses_panel

_lzma                 _sqlite3              _tkinter

readline

To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been

built by the Makefile instead, as configured by the Setup files:

_abc                  atexit                pwd

time


Following modules built successfully but were removed because they could not be imported:

_ctypes

Information on your system

  • Spack: 0.17.1-1532-8adc6b7e8e
  • Python: 3.9.10
  • Platform: linux-centos7-zen2
  • Concretizer: clingo

Additional information

@adamjstewart @scheibelp @skosukhin @varioustoxins

A recent build on an almost identical configuration passed, so I suspect this is due to the libffi update in #28970 .
spack-build-out.txt

General information

  • I have run spack debug report and reported the version of Spack/Python/Platform
  • I have run spack maintainers <name-of-the-package> and @mentioned any maintainers
  • I have uploaded the build log and environment files
  • I have searched the issues of this repo and believe this is not a duplicate

Metadata

Metadata

Assignees

No one assigned

    Labels

    build-error

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions