The headers property of a package defines where to look for headers for that package. So with this PR, spec['py-numpy'].headers will tell you where the headers can be found. We could define this in the package that needs to find the headers, but if there are more than one, we would need to copy-n-paste it every time. So the idea is to define it only once and let packages reuse it. It was supposed to work by defining it in the base class, but it isn't working and I don't know why.

Btw, py-petsc4py is the package that has trouble finding these headers without adding them to CPATH @balay

Ah okay, that is helpful context info. In that case, recursively searching platlib seems watertight. The numpy headers should never be landing in <prefix>/include.

@spackbot run pipeline

I've started that pipeline for you!

Btw, py-petsc4py is the package that has trouble finding these headers without adding them to CPATH @balay

Looks like petsc4py is querying numpy directly to get this info

>>> import numpy
>>> numpy.get_include()
'/usr/lib64/python3.10/site-packages/numpy/core/include'

So assuming petsc4py will be unaffected by this change..

I'm seeing this pattern used in the py-gpaw, vapor, and precice packages to get a particular numpy/core/include subdirectory of platlib, which in precice's case at least is used to populate a specific -DNumPy_INCLUDE_DIR CMake var:

Outside of py-matplotlib and py-torchvision, I wasn't able to see any packages specifically requesting py-numpy's .headers (which I guess makes sense if #28527 was just merged). The reason I mention this is: if multiple packages are trying to get at that precise subdirectory of numpy's includes, would it make sense to add that particular platlib subdirectory as a property of the py-numpy package as well here?

@adamjstewart: I don't fully understand what function that inspect.getmodule(self).include (added in #28527) is expected to invoke, and by inspecting the callstack, inspect.getmodule(self).include seemed to be calling _headers_default_handler() in spec.py (which I still do not understand whatsoever), which is why you saw both the default and the build_systems/python.py method being called in order.

However, I was regardless able to make py-scipy find the headers for py-numpy by taking the exact code you used in this PR, and instead moving it into build_systems/python.py's def headers(self) method. The diff is here adamjstewart/spack@packages/py-numpy...cosmicexplorer:spack:packages/py-numpy, but it's just one commit that does exactly that. Please let me know if this also works for you.

would it make sense to add that particular platlib subdirectory as a property of the py-numpy package as well here?

That's the point of using spec['py-numpy'].headers.directories[0], we should replace the hacks in those package with this if we can get it working.

I don't fully understand what function that inspect.getmodule(self).include (added in #28527) is expected to invoke

It's supposed to access spec['python'].package.include, but I guess it's actually named python_include, not include. Let me try that. I'll try your patch as well.

Yep, that seems to work, nice find! I'll open a separate PR for this.