BLD: enable building NumPy with Meson by rgommers · Pull Request #22663 · numpy/numpy

rgommers · 2022-11-23T23:31:00Z

This enables building with NumPy on Linux and macOS. Windows support should be complete too, but is untested as of now and may need a few tweaks. This contains:

A set of meson.build files and related code generation script tweaks, header templates, etc.
One CI job on Linux
Basic docs on using Meson to build NumPy (not yet integrated in the html docs, it's too early for that - this is for early adopters right now).

The build should be complete, with the major exception of SIMD support. The full test suite passes. See gh-22546 for the tracking issue with detailed notes on the plan for switching NumPy to Meson as its build system.

rgommers · 2022-11-24T13:24:39Z

The phase [skip azp] [skip github] somehow triggers wheel builds, that looks like a bug. Already all green once, so cancelled now.

charris · 2022-11-24T14:15:11Z

somehow triggers wheel builds

The "build" label does that.

rgommers · 2022-11-24T14:35:28Z

The "build" label does that.

argh, the #!@&$ bot defeating my labeling. thanks for pointing that out.

meson.build

melissawm · 2022-11-24T23:05:43Z

When trying to compile on windows, I get different outputs from distutils and meson.

_numpyconfig.h generated by distutils

#define NPY_SIZEOF_SHORT SIZEOF_SHORT
#define NPY_SIZEOF_INT SIZEOF_INT
#define NPY_SIZEOF_LONG SIZEOF_LONG
#define NPY_SIZEOF_FLOAT 4
#define NPY_SIZEOF_COMPLEX_FLOAT 8
#define NPY_SIZEOF_DOUBLE 8
#define NPY_SIZEOF_COMPLEX_DOUBLE 16
#define NPY_SIZEOF_LONGDOUBLE 8
#define NPY_SIZEOF_COMPLEX_LONGDOUBLE 16
#define NPY_SIZEOF_PY_INTPTR_T 8
#define NPY_SIZEOF_OFF_T 4
#define NPY_SIZEOF_PY_LONG_LONG 8
#define NPY_SIZEOF_LONGLONG 8
#define NPY_NO_SIGNAL 1
#define NPY_NO_SMP 0
#define NPY_HAVE_DECL_ISNAN
#define NPY_HAVE_DECL_ISINF
#define NPY_HAVE_DECL_SIGNBIT
#define NPY_HAVE_DECL_ISFINITE
#define NPY_USE_C99_COMPLEX 1
#define NPY_USE_C99_FORMATS 1
#define NPY_VISIBILITY_HIDDEN 
#define NPY_ABI_VERSION 0x01000009
#define NPY_API_VERSION 0x00000010

#ifndef __STDC_FORMAT_MACROS
#define __STDC_FORMAT_MACROS 1
#endif

_numpyconfig.h generated by meson

/* #undef NPY_HAVE_ENDIAN_H */

#define NPY_SIZEOF_SHORT 2
#define NPY_SIZEOF_INT 4
#define NPY_SIZEOF_LONG 4
#define NPY_SIZEOF_FLOAT 4
/* #undef NPY_SIZEOF_COMPLEX_FLOAT */
#define NPY_SIZEOF_DOUBLE 8
/* #undef NPY_SIZEOF_COMPLEX_DOUBLE */
#define NPY_SIZEOF_LONGDOUBLE 8
/* #undef NPY_SIZEOF_COMPLEX_LONGDOUBLE */
#define NPY_SIZEOF_PY_INTPTR_T 8
#define NPY_SIZEOF_OFF_T 4
#define NPY_SIZEOF_PY_LONG_LONG 8
#define NPY_SIZEOF_LONGLONG 8

/* #undef NPY_HAVE_DECL_ISNAN */
/* #undef NPY_HAVE_DECL_ISINF */
/* #undef NPY_HAVE_DECL_ISFINITE */
/* #undef NPY_HAVE_DECL_SIGNBIT */
#define NPY_USE_C99_COMPLEX 1
/* #undef NPY_HAVE_COMPLEX_DOUBLE */
/* #undef NPY_HAVE_COMPLEX_FLOAT */
/* #undef NPY_HAVE_COMPLEX_LONG_DOUBLE */
#define NPY_USE_C99_FORMATS 1

#define NPY_NO_SIGNAL 1
#define NPY_NO_SMP 0

#define NPY_VISIBILITY_HIDDEN 
#define NPY_ABI_VERSION 0x01000009
#define NPY_API_VERSION 0x00000010

#ifndef __STDC_FORMAT_MACROS
#define __STDC_FORMAT_MACROS 1
#endif

It looks like a bunch of headers are not linked correctly - I am very ignorant on windows issues but just as an experiment I edited numpy/core/meson.build so that these two files agree, skipping the check for complex types. Then, I get a bunch of other errors involving _NPY_SIGINT_BUF (which including signal.h seems to solve?) and then locale_t... I suspect some include path is not detected correctly but I don't know enough to give a full diagnostic. Let me know if I can do any further tests on this.

rgommers · 2022-11-25T03:35:08Z

Thanks @melissawm, that doesn't look too bad. The C99 complex types indeed aren't available, the headers are consistent there. A difference is that it's explicit in the meson-generated headers (shows as /* #undef HAVE_XXX */), while distutils silently drops them. The former is nicer. What seems to need fixing is the numpy-specific replacements, and then we should get back NPY_SIZEOF_COMPLEX_FLOAT & co.

I'll make the MSVC version change you pointed out, and will open a separate issue for Windows support as a follow-up to this PR.

`Python.h` must be included before including and standard library headers, if it's pulled in (which happens when you include numpy headers). Otherwise we see build warnings like: ``` 142/244] Compiling C object numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so.p/src_multiarray_textreading_field_types.c.o In file included from /opt/hostedtoolcache/Python/3.11.0/x64/include/python3.11/Python.h:86, from ../numpy/core/include/numpy/npy_common.h:5, from ../numpy/core/include/numpy/ndarraytypes.h:4, from ../numpy/core/src/multiarray/textreading/field_types.h:9, from ../numpy/core/src/multiarray/textreading/field_types.c:1: /opt/hostedtoolcache/Python/3.11.0/x64/include/python3.11/cpython/pytime.h:208:60: warning: ‘struct timespec’ declared inside parameter list will not be visible outside of this definition or declaration 208 | PyAPI_FUNC(int) _PyTime_FromTimespec(_PyTime_t *tp, struct timespec *ts); | ^~~~~~~~ /opt/hostedtoolcache/Python/3.11.0/x64/include/python3.11/cpython/pytime.h:213:56: warning: ‘struct timespec’ declared inside parameter list will not be visible outside of this definition or declaration 213 | PyAPI_FUNC(int) _PyTime_AsTimespec(_PyTime_t t, struct timespec *ts); | ^~~~~~~~ /opt/hostedtoolcache/Python/3.11.0/x64/include/python3.11/cpython/pytime.h:217:63: warning: ‘struct timespec’ declared inside parameter list will not be visible outside of this definition or declaration 217 | PyAPI_FUNC(void) _PyTime_AsTimespec_clamp(_PyTime_t t, struct timespec *ts); | ^~~~~~~~ ```

…pared

This enables building with NumPy on Linux and macOS. Windows support should be complete to, but is untested as of now and may need a few tweaks. This contains: - A set of `meson.build` files and related code generation script tweaks, header templates, etc. - One CI job on Linux - Basic docs on using Meson to build NumPy (not yet integrated in the html docs, it's too early for that - this is for early adopters right now). The build should be complete, with the major exception of SIMD support. The full test suite passes. See numpygh-22546 for the tracking issue with detailed notes on the plan for switching NumPy to Meson as its build system. Co-authored-by: Stefan van der Walt <stefanv@berkeley.edu>

rgommers · 2022-11-25T11:42:48Z

The labeler bot is messing with labels again. It should never do that after the initial run it had, this is both annoying and a huge waste of CI resources. I believe the SciPy action gets this right (we removed it once until this exact issue was fixed). Can I interest anyone in fixing that?

EDIT: I have manually disabled it via the GitHub UI for the time being.

rgommers · 2022-11-25T12:52:45Z

Okay, this is ready to go in. I'll also point out that this adds -Werror to the CI job, so we prevent new issues creeping in. We used to have one on Travis CI, but we seem to have lost that somewhere along the way.

rgommers · 2022-11-27T13:51:18Z

Okay, I'll hit the green button and will then update the meson branch to be in sync. @stefanv confirmed that this PR looks good. That keeps the ball rolling, and we can work on Windows and SIMD support in the meson branch.

seberg · 2022-11-28T21:06:32Z

Not quite sure if it is this, but locally, I am now getting:

numpy/random/_common.pyx has not changed
Cythonizing sources
Traceback (most recent call last):
  File "/Users/sebastianb/forks/numpy/setup.py", line 476, in <module>
    setup_package()
  File "/Users/sebastianb/forks/numpy/setup.py", line 468, in setup_package
    setup(**metadata)
  File "/Users/sebastianb/forks/numpy/numpy/distutils/core.py", line 126, in setup
    dist = setup(**new_attr)
  File "/Users/sebastianb/forks/numpy/numpy/distutils/core.py", line 169, in setup
    return old_setup(**new_attr)
  File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 159, in setup
    dist.parse_config_files()
  File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/setuptools/dist.py", line 868, in parse_config_files
    pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
  File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 62, in apply_configuration
    config = read_configuration(filepath, True, ignore_option_errors, dist)
  File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 126, in read_configuration
    validate(subset, filepath)
  File "/opt/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/setuptools/config/pyprojecttoml.py", line 51, in validate
    raise ValueError(f"{error}\n{summary}") from None
ValueError: invalid pyproject.toml config: `project`.
configuration error: `project` must not contain {'license-files'} properties

when using runtests.py but not when doing pip install .? Does that immediately make sense to you?

mattip · 2022-11-28T21:09:56Z

check what version of setuptools you have. We pin to setuptools<60 for a reason

rgommers · 2022-11-28T21:10:10Z

Ugh. No support for PEP 621 probably. What setuptools version in that build log where you see the failure?

but not when doing pip install .?

That is because that does an isolated build, so pinned setuptools version

rgommers · 2022-11-28T21:10:45Z

We pin to setuptools<60 for a reason

Indeed, but we can comment out these pyproject.toml entries for now to not make things harder than they have to be.

eli-schwartz · 2022-11-28T21:12:55Z

pyproject.toml

+license-files.paths = [
+    "LICENSE.txt",
+    "LICENSES_bundles.txt"
+]


Unsupported versions of setuptools that are NOT version 59 as pinned in this file, add pyproject.toml support to begin with -- and don't like this PEP 639 draft metadata key because reasons.

Basically, apparently it's setuptools' job (even if setuptools is supposed to use setup.py) to refuse to build anything at all, if a new informational metadata value is added to the standard and setuptools doesn't support it.

I guess that's sort of defensible. The spec says:

When specifying project metadata, tools MUST adhere and honour the metadata as specified in this PEP. If metadata is improperly specified then tools MUST raise an error to notify the user about their mistake.

@abravalheri was presumably relying on that wording when implementing setuptools' hard error mode there?

Hi @eli-schwartz , if setuptools is invoked in a project that contains pyproject.toml it will try to read metadata/configuration from it.

It also will try to follow the spec as close as possible.

Thanks for the explanations. I'll open a PR to comment out the problematic items.

Side note: it'd be great if PEP 639 was hurried along, it's quite annoying that PyPI doesn't properly support PEP 621 license info (mainly because it's not sufficient, bad spec), and is waiting for PEP 639 to get something sensible implemented. Right now are choices are basically to ignore any spec and do whatever setuptools does, or to get the mess that you see in the sidebar of https://pypi.org/project/scipy/. Context in mesonbuild/meson-python#129.

Done in gh-22687

seberg · 2022-11-28T21:33:03Z

I got a too new setuptools indeed, although I guess it is a mix of things since it worked before? Anyway, I am happy if/that the answer is to make sure the setuptools pin adds up again.

(I guess the hard error mode may be weird, but a setuptools thing to worry about)

See comments on numpygh-22663

rgommers added 01 - Enhancement Meson Items related to the introduction of Meson as the new build system for NumPy labels Nov 23, 2022

rgommers added this to the 1.25.0 release milestone Nov 23, 2022

github-actions bot added the 36 - Build Build related PR label Nov 23, 2022

stefanv force-pushed the build-with-meson branch 2 times, most recently from 025d655 to b3b864e Compare November 24, 2022 01:00

rgommers removed the 36 - Build Build related PR label Nov 24, 2022

rgommers force-pushed the build-with-meson branch from 52107a1 to 3f0c588 Compare November 24, 2022 14:38

github-actions bot added the 36 - Build Build related PR label Nov 24, 2022

melissawm reviewed Nov 24, 2022

View reviewed changes

meson.build Outdated Show resolved Hide resolved

rgommers force-pushed the build-with-meson branch from 51c8ab5 to 5dbac25 Compare November 25, 2022 11:25

rgommers and others added 3 commits November 25, 2022 12:33

MAINT: avoid gcc warning that signed and unsigned types are being com…

632f573

…pared

rgommers force-pushed the build-with-meson branch from 5dbac25 to 4002a7d Compare November 25, 2022 11:38

rgommers removed the 36 - Build Build related PR label Nov 25, 2022

rgommers merged commit 3b7672e into numpy:main Nov 27, 2022

rgommers deleted the build-with-meson branch November 27, 2022 13:51

rgommers mentioned this pull request Nov 27, 2022

Tracking issue: moving to Meson as the build system for NumPy #22546

Closed

eli-schwartz reviewed Nov 28, 2022

View reviewed changes

rgommers added a commit to rgommers/numpy that referenced this pull request Nov 29, 2022

BLD: revert adding PEP 621 metadata, it confuses setuptools

16be220

See comments on numpygh-22663

rth mentioned this pull request Mar 7, 2023

parallel build fails #13080

Closed

Uh oh!

Conversation

rgommers commented Nov 23, 2022 • edited by stefanv Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Nov 24, 2022

Uh oh!

charris commented Nov 24, 2022

Uh oh!

rgommers commented Nov 24, 2022

Uh oh!

Uh oh!

melissawm commented Nov 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Nov 25, 2022

Uh oh!

rgommers commented Nov 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Nov 25, 2022

Uh oh!

rgommers commented Nov 27, 2022

Uh oh!

seberg commented Nov 28, 2022

Uh oh!

mattip commented Nov 28, 2022

Uh oh!

rgommers commented Nov 28, 2022

Uh oh!

rgommers commented Nov 28, 2022

Uh oh!

eli-schwartz Nov 28, 2022

Choose a reason for hiding this comment

Uh oh!

eli-schwartz Nov 28, 2022

Choose a reason for hiding this comment

Uh oh!

abravalheri Nov 29, 2022

Choose a reason for hiding this comment

Uh oh!

rgommers Nov 29, 2022

Choose a reason for hiding this comment

Uh oh!

rgommers Nov 29, 2022

Choose a reason for hiding this comment

Uh oh!

seberg commented Nov 28, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

rgommers commented Nov 23, 2022 •

edited by stefanv

Loading

melissawm commented Nov 24, 2022 •

edited

Loading

rgommers commented Nov 25, 2022 •

edited

Loading