Can't compile Meson extensions against managed Python due to pkg-config #11028
Description
Summary
When compiling a Python extension using Meson's py_installation.extension_module(), Meson looks for the Python development headers using pkg-config. Because the prefix in python-build-standalone's pc file is /install, the build fails.
The sysconfig paths in PBS are fine, but for better or worse Meson tries pkg-config before sysconfig. There doesn't seem to be a universal way (e.g. one that works for subprojects) to use sysconfig before pkg-config in Meson, but even if there was, it seems like it would be preferable to fix the pc file.
Is that possible? Could uv fix the pc file after extraction? I guess this wouldn't help non-uv PBS users, however.
I realize this is documented, so apologies for raising a known issue! But it took a little sleuthing to figure out how Meson was even picking up this /install path, so I thought I'd at least share that here.
For completeness, here's the .pc file:
python-3.13.pc
❯ cat /mnt/home/lgarrison/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/pkgconfig/python-3.13.pc
# See: man pkg-config
prefix=/install
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include
Name: Python
Description: Build a C extension for Python
Requires:
Version: 3.13
Libs.private: -lpthread -ldl -lutil
Libs: -L${libdir}
Cflags: -I${includedir}/python3.13
Here's where Meson tries pkg-config before sysconfig: https://github.com/mesonbuild/meson/blob/0b9baf573bf2b1619d7822f67965a35391172625/mesonbuild/dependencies/python.py#L391
Here's the output of Meson's python_info.py script that uses sysconfig for discovery (looks fine):
python_info.py output
❯ python mesonbuild/scripts/python_info.py | jq | grep /install
"CONFIG_ARGS": "'--build=x86_64-unknown-linux-gnu' '--host=x86_64-unknown-linux-gnu' '--prefix=/install' '--with-openssl=/tools/deps' '--with-system-expat' '--with-system-libmpdec' '--without-ensurepip' '--with-readline=editline' '--enable-shared' '--with-mimalloc' '--enable-optimizations' '--with-lto' '--with-build-python=/tools/host/bin/python3.13' '--with-dbmliborder=bdb' 'build_alias=x86_64-unknown-linux-gnu' 'host_alias=x86_64-unknown-linux-gnu' 'CC=clang' 'CFLAGS= -fPIC ' 'LDFLAGS= -Wl,--exclude-libs,ALL -LModules/_hacl' 'CPPFLAGS= -fPIC '",
"INSTALL": "/usr/bin/install -c",
"INSTALL_DATA": "/usr/bin/install -c -m 644",
"INSTALL_PROGRAM": "/usr/bin/install -c",
"INSTALL_SCRIPT": "/usr/bin/install -c",
"INSTALL_SHARED": "/usr/bin/install -c -m 755",
"WASM_ASSETS_DIR": "./install",
"WASM_STDLIB": "./install/lib/python3.13/os.py",
And here's a typical compiler command that Meson emits, with -I/install/...:
Compiler command
ccache cc -I_pycorrfunc.cpython-313-x86_64-linux-gnu.so.p -I. -I../.. -I../../include -I../../lib/theory/DD -I../../lib/common
-I../../subprojects/nanobind-2.2.0/include -I../../subprojects/robin-map-1.3.0/include -I/install/include/python3.13 -fvisibility=hidden
-fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -O3 -DPYCORRFUNC_USE_DOUBLEACCUM -fPIC -fopenmp -ffunction-sections
-fdata-sections -DPYCORRFUNC_USE_DOUBLE -DNB_DOMAIN=d -funroll-loops -MD -MQ _pycorrfunc.cpython-313-x86_64-linux-gnu.so.p/lib_theory_DD_countpairs.c.o
-MF _pycorrfunc.cpython-313-x86_64-linux-gnu.so.p/lib_theory_DD_countpairs.c.o.d -o
_pycorrfunc.cpython-313-x86_64-linux-gnu.so.p/lib_theory_DD_countpairs.c.o -c ../../lib/theory/DD/countpairs.c
../../lib/theory/DD/countpairs.c:7:10: fatal error: Python.h: No such file or directory
#include <Python.h>
^~~~~~~~~~
There's a constellation of sort-of similar uv issues around sysconfig variables (e.g. those linked from #8429), but I think in this case the issue is specifically with the pkg-config file. I couldn't find an existing issue about that.
Steps to reproduce
Here is a reproducer on a small project. I'm happy to work on making it more minimal if that would be helpful.
❯ git clone https://github.com/lgarrison/pycorrfunc.git
❯ cd pycorrfunc
❯ git checkout c869af8feb936960ef3673aa0974eb332c47fc8a
❯ uv sync --python-preference=only-managed
Platform
Rocky 8 Linux
Version
0.5.24
Python version
Python 3.13