BLD: Meson `__config__` generation by ganesh-k13 · Pull Request #22769 · numpy/numpy

ganesh-k13 · 2022-12-10T17:45:47Z

Changes

Add functionality to autogenerate build information from Meson information
In order to add new information, do the following:

Add the information as an argument in numpy/meson.build
Modify __config__.py.in to accept the new argument

New `config.py`:

'print' mode

In [2]: np.show_config()                                                                                                                                                                          
Build Dependencies:                                                                                                                                                                               
  blas:                                                                                                                                                                                           
    found: 'True'                                                                                                                                                                                 
    includedir: /usr/include/x86_64-linux-gnu/openblas-pthread/                                                                                                                                   
    libdir: /usr/lib/x86_64-linux-gnu/openblas-pthread/                                                                                                                                           
    name: openblas                                                                                                                                                                                
    openblas config: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS= NO_LAPACK=                                                                                                            
      NO_LAPACKE=1 NO_AFFINITY=1 USE_OPENMP=0 generic MAX_THREADS=64                                                                                                                              
    pc file directory: /usr/lib/x86_64-linux-gnu/pkgconfig
    type name: pkgconfig                                                                         
    version: 0.3.20                                                                              
  lapack:                                                                                                                                                                                         
    found: 'True'                                                                                
    includedir: /usr/include/x86_64-linux-gnu/openblas-pthread/                                  
    libdir: /usr/lib/x86_64-linux-gnu/openblas-pthread/
    name: openblas                                                                               
    openblas_config: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS= NO_LAPACK=           
      NO_LAPACKE=1 NO_AFFINITY=1 USE_OPENMP=0 generic MAX_THREADS=64                             
    pc file directory: /usr/lib/x86_64-linux-gnu/pkgconfig
    type_name: pkgconfig                                                                         
    version: 0.3.20                                                                              
Compilers:                                                                                                                                                                                        
  c:                                                                                             
    commands: cc                                
    linker: ld.bfd                              
    name: gcc                                   
    version: 11.3.0                                                                              
  c++:                                          
    commands: c++                               
    linker: ld.bfd                              
    name: gcc                                   
    version: 11.3.0                                                                              
  cython:                                       
    commands: cython                                                                             
    linker: cython                                                                               
    name: cython                                                                                 
    version: 0.29.30                                                                             
Machine Information:                                                                             
  build:                                                                                         
    cpu: x86_64 
    endian: little
    family: x86_64
    system: linux
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
Python Information:
  path: /home/ganesh/os/np-test/bin/python3
  version: '3.10'

'json' mode

In [3]: np.show_config(mode='json')
Out[3]: 
{'Compilers': {'c': {'name': 'gcc',
   'linker': 'ld.bfd',
   'version': '11.3.0',
   'commands': 'cc'},
  'cython': {'name': 'cython',
   'linker': 'cython',
   'version': '0.29.30',
   'commands': 'cython'},
  'c++': {'name': 'gcc',
   'linker': 'ld.bfd',
   'version': '11.3.0',
   'commands': 'c++'}},
 'Machine Information': {'host': {'cpu': 'x86_64',
   'family': 'x86_64',
   'endian': 'little',
   'system': 'linux'},
  'build': {'cpu': 'x86_64',
   'family': 'x86_64',
   'endian': 'little',
   'system': 'linux'}},
 'Build Dependencies': {'blas': {'name': 'openblas',
   'found': 'True',
   'version': '0.3.20',
   'type name': 'pkgconfig',
   'includedir': '/usr/include/x86_64-linux-gnu/openblas-pthread/',
   'libdir': '/usr/lib/x86_64-linux-gnu/openblas-pthread/',
   'openblas config': 'USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS= NO_LAPACK= NO_LAPACKE=1 NO_AFFINITY=1 USE_OPENMP=0 generic MAX_THREADS=64',
   'pc file directory': '/usr/lib/x86_64-linux-gnu/pkgconfig'},
  'lapack': {'name': 'openblas',
   'found': 'True',
   'version': '0.3.20',
   'type_name': 'pkgconfig',
   'includedir': '/usr/include/x86_64-linux-gnu/openblas-pthread/',
   'libdir': '/usr/lib/x86_64-linux-gnu/openblas-pthread/',
   'openblas_config': 'USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS= NO_LAPACK= NO_LAPACKE=1 NO_AFFINITY=1 USE_OPENMP=0 generic MAX_THREADS=64',
   'pc file directory': '/usr/lib/x86_64-linux-gnu/pkgconfig'}},
 'Python Information': {'path': '/home/ganesh/os/np-test/bin/python3',
  'version': '3.10'}}

'print' mode no PyYaml

In [2]: np.show_config()                                                                         
/home/ganesh/os/numpy/build-install/usr/lib/python3/dist-packages/numpy/__config__.py:117: UserWarning: Install `pyyaml` for better output                  
  warnings.warn('Install `pyyaml` for better output')           
{                                               
  "Compilers": {                                
    "c": {                                      
      "name": "gcc",                            
      "linker": "ld.bfd", 
      "version": "11.3.0",     
      "commands": "cc"                                                                           
    },                                                                                           
    "cython": {                                                                                                                                                                                   
      "name": "cython",                                                                          
      "linker": "cython",                       
      "version": "0.29.30",                     
      "commands": "cython"
    },                                                                                           
    "c++": {                                    
      "name": "gcc",                            
      "linker": "ld.bfd",                       
      "version": "11.3.0",
      "commands": "c++"                         
    }                                           
  },                                            
  "Machine Information": {
    "host": {                                   
      "cpu": "x86_64",                          
      "family": "x86_64",
      "endian": "little",
      "system": "linux"                         
    },
    "build": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "linux"
    }
  },
  "Build Dependencies": {
    "blas": {
      "name": "openblas",
      "found": "True",
      "version": "0.3.20",
      "type name": "pkgconfig",
      "includedir": "/usr/include/x86_64-linux-gnu/openblas-pthread/",
      "libdir": "/usr/lib/x86_64-linux-gnu/openblas-pthread/",
      "openblas config": "USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS= NO_LAPACK= NO_LAPACKE=1 NO_AFFINITY=1 USE_OPENMP=0 generic MAX_THREADS=64",
      "pc file directory": "/usr/lib/x86_64-linux-gnu/pkgconfig"
    },
    "lapack": {
      "name": "openblas",
      "found": "True",
      "version": "0.3.20",
      "type_name": "pkgconfig",
      "includedir": "/usr/include/x86_64-linux-gnu/openblas-pthread/",
      "libdir": "/usr/lib/x86_64-linux-gnu/openblas-pthread/",
      "openblas_config": "USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS= NO_LAPACK= NO_LAPACKE=1 NO_AFFINITY=1 USE_OPENMP=0 generic MAX_THREADS=64",
      "pc file directory": "/usr/lib/x86_64-linux-gnu/pkgconfig"
    }
  },
  "Python Information": {
    "path": "/home/ganesh/os/np-test/bin/python3",
    "version": "3.10"
  }
}

TODO

related: #20939 (comment) | scipy/scipy#16803

ganesh-k13 · 2022-12-10T17:47:29Z

Hi @rgommers , I've made a POC of what I had in mind and wanted to know if this is the right approach for the requirement. I can continue based on feedback.

eli-schwartz · 2022-12-13T03:53:11Z

Since this only contains information known at the time of configuration, I wonder if it's necessary to have a generator script at all? Maybe passing a dict in meson to a configure_file() and using that to fill out a template, would work equally well?

The main difference is that it requires less work and one less build edge.

ganesh-k13 · 2022-12-13T09:22:19Z

Ah ok that makes sense. Something like this yeah?

rgommers · 2022-12-13T19:24:53Z

Ah ok that makes sense. Something like this yeah?

That example is not the best (the if-else could use a single .set10 call instead), but the basic principle is okay for a dict passed to configure_file. In NumPy itself there's a more comprehensive example, including use of a template:

numpy/numpy/core/meson.build

Line 423 in a24ddad

config_h = configure_file(

numpy/meson.build

rgommers · 2022-12-15T15:19:01Z

This looks pretty good at first glance @ganesh-k13. I added some comments, will do a proper review later.

ganesh-k13 · 2022-12-16T15:42:53Z

numpy/__config__.py.in

+            "PYTHON_PATH": "@PYTHON_PATH@",
+            "PYTHON_VERSION": "@PYTHON_VERSION@",
+        },
+        "SIMD Extensions": {


As of today it's empty and get's removed by _cleanup (including the top-level key), but should work once we add SIMD. I'll keep a look out and test it post that, or I could remove it now and add it later.

ganesh-k13 · 2022-12-16T15:49:29Z

Are we planning on using Meson Tests? I was exploring how to add UT. If it's in the plan, I can start the test framework with this module, else I'll stick with pytest and see how to test this.

eli-schwartz · 2022-12-16T15:53:01Z

If you've already heavily invested in pytest, meson test doesn't add much. Particularly, there's not a whole lot of benefit in taking one test harness/reporter (meson test) and using it to run another one (pytest).

You can do it, but it will just result in meson test reporting on a single test (that is, whether running pytest passes or fails) and then you can inspect the meson test logs to see the output of pytest.

It's really meant for cases where you don't have an existing test harness, and want to set one up. Particularly, when tests are run as a series of test programs rather than in-process setup/teardown.

You can use a TAP reporter for pytest, then tell meson to read the Meson Test via the TAP protocol, that does get you subtest reporting. But pytest is already pretty nice as is so it's work for no clear gain, and passing options from meson test to pytest at test time feels clunky.

rgommers

Hi @ganesh-k13, I did some testing. It all seems to work, my main observation is that the output is quite difficult to read. For comparison, let's have a look at show_runtime, show_config and the output of Meson itself:

In [2]: np.show_runtime()
[{'simd_extensions': {'baseline': [], 'found': [], 'not_found': []}},
 {'architecture': 'SkylakeX',
  'filepath': '/home/rgommers/mambaforge/envs/numpy-dev/lib/libopenblasp-r0.3.21.so',
  'internal_api': 'openblas',
  'num_threads': 24,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.21'}]

In [3]: np.show_config()
Out[3]: 
{'C Compiler': {'C_COMP': 'gcc',
  'C_COMP_LINKER_ID': 'ld.bfd',
  'C_COMP_VERSION': '10.4.0',
...

And the Meson configure output:

C compiler for the host machine: /home/rgommers/mambaforge/envs/numpy-dev/bin/x86_64-conda-linux-gnu-cc (gcc 10.4.0 "x86_64-conda-linux-gnu-cc (conda-forge gcc 10.4.0-19) 10.4.0")
C linker for the host machine: /home/rgommers/mambaforge/envs/numpy-dev/bin/x86_64-conda-linux-gnu-cc ld.bfd 2.39
C++ compiler for the host machine: /home/rgommers/mambaforge/envs/numpy-dev/bin/x86_64-conda-linux-gnu-c++ (gcc 10.4.0 "x86_64-conda-linux-gnu-c++ (conda-forge gcc 10.4.0-19) 10.4.0")
C++ linker for the host machine: /home/rgommers/mambaforge/envs/numpy-dev/bin/x86_64-conda-linux-gnu-c++ ld.bfd 2.39
Cython compiler for the host machine: cython (cython 0.29.32)
Host machine cpu family: x86_64
Host machine cpu: x86_64
...

The capitals, hard to understand names and returning a dict of dicts instead of actually printing all contribute to the output being tricky for the reader to parse. I suggest changing the keys of the dict being as easy as possible to understand. And then also providing an option to print to stdout rather than returning a dict of dicts.

A few examples to make that concrete:

'BLAS_VERSION': '0.3.21',

I'd make that 'version': '0.3.21' (it's already under a BLAS key).

'LAPACK_TYPE_NAME': 'pkgconfig',

I'd make this: 'detection method': 'pkgconfig',

ganesh-k13 · 2022-12-19T05:11:54Z

Yeah, that's a really good point! In fact Matti had a similar idea in the previous PR(#20939 (comment)). I'll make those changes 👍

ganesh-k13 · 2022-12-22T15:24:57Z

I have added modes, please see the issue description for examples.

numpy/meson.build

numpy/__config__.py.in

ganesh-k13 · 2022-12-24T13:43:28Z

Overall I think we can refine the keys a bit to make more sense. I'm torn between making it programmatically accessible vs human readable, for example: openblas_config vs openblas configuration. Please feel free to suggest any changes to keys or the way we are displaying them if PyYaml dump makes sense or we can write our own tabulate of sorts.

rgommers

This is getting close! Just a few minor comments, and docs left to be added.

numpy/__config__.py.in

rgommers · 2022-12-24T13:48:38Z

Overall I think we can refine the keys a bit to make more sense. I'm torn between making it programmatically accessible vs human readable, for example: openblas_config vs openblas configuration.

I think optimizing for human-readable when printing sounds fine to me. I think that's the first thing people will do. Programmatic access is important too, but it's fine is that requires say a space in the key of a dict.

numpy/tests/test_numpy_config.py

numpy/__config__.py.in

rgommers · 2022-12-24T14:47:29Z

When you think this is ready to merge, please take it out of Draft mode and let me know - happy to get this in soon:)

ganesh-k13 · 2022-12-24T15:22:12Z

I didn't realise it was draft. Yeah almost ready, I'll just restrict the tests. Does it need a release note as show_config takes args now? Or we just wait for distutils to go and add one big release note?

rgommers · 2022-12-24T15:49:15Z

distutils to go is probably ~5 months away, so let's not wait for that.

* Adds a new env in `__config__` to indicate if built with meson

* Removed `Target` from host info * Fixed warning with using compiler `path` * Added `cross-compiled` flag

ganesh-k13 · 2023-01-23T12:07:41Z

Hey @rgommers , any more changes needed on this PR?

rgommers

Thanks for the ping @ganesh-k13, I missed the last update. This looks good to go now, time to merge it. Thanks a lot for getting this over the finish line!

* Based on numpy/numpy#22769

seberg · 2023-01-26T11:44:37Z

I am running into a circular import trying to add a flag to the main namespace. I could change that design to be a getter or move the initialization around.

But I am wondering if we can move the C imports here back to be done lazily to avoid importing C until later?

seberg · 2023-01-26T11:52:51Z

Nvm, it is nice to __cpu_baseline__ defined in that scope... Maybe this should be cleaned up, but I think the solution here is to just move my thing before this init.

ganesh-k13 · 2023-01-26T12:34:17Z

We could refine the cleanup to take place in the actual function and move the import inside. I think that's cleaner as well. Shall I do that? I feel we will hit this in the future.

seberg · 2023-01-26T12:42:46Z

I think it is fine as is, thanks. I had added the _utils to move most things that may be required in C into. Just now, I thought I would have this one flag directly in the main namespace, so it needs to happen before the import.

The only thing I could think of doing would be to make it a bit more obvious where the C-module is imported the first time (which is also the place guarded by this?):

    except ImportError as e:
        msg = """Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there."""
        raise ImportError(msg) from e

I think. Maybe with a comment that things that used from C must be defined before or live live in _utils/.

* Based on numpy/numpy#22769

ganesh-k13 requested a review from rgommers December 10, 2022 17:45

ganesh-k13 marked this pull request as draft December 10, 2022 17:46

ganesh-k13 added component: build Meson Items related to the introduction of Meson as the new build system for NumPy labels Dec 10, 2022

ganesh-k13 force-pushed the enh_config branch 2 times, most recently from aa0a2f5 to 1a7680e Compare December 15, 2022 14:32

ganesh-k13 commented Dec 15, 2022

View reviewed changes

numpy/meson.build Show resolved Hide resolved

rgommers reviewed Dec 15, 2022

View reviewed changes

numpy/meson.build Outdated Show resolved Hide resolved

rgommers reviewed Dec 15, 2022

View reviewed changes

numpy/meson.build Outdated Show resolved Hide resolved

rgommers reviewed Dec 15, 2022

View reviewed changes

numpy/meson.build Outdated Show resolved Hide resolved

ganesh-k13 force-pushed the enh_config branch from 1a7680e to 942e971 Compare December 16, 2022 15:38

ganesh-k13 commented Dec 16, 2022

View reviewed changes

rgommers requested changes Dec 18, 2022

View reviewed changes

rgommers mentioned this pull request Dec 19, 2022

Update scipy/__config__.py to contain useful information scipy/scipy#16803

Closed

ganesh-k13 force-pushed the enh_config branch from 942e971 to ee6b49c Compare December 22, 2022 15:18

ganesh-k13 commented Dec 22, 2022

View reviewed changes

numpy/meson.build Outdated Show resolved Hide resolved

ganesh-k13 commented Dec 22, 2022

View reviewed changes

numpy/__config__.py.in Outdated Show resolved Hide resolved

ganesh-k13 force-pushed the enh_config branch from ee6b49c to 7875c1b Compare December 24, 2022 12:43

rgommers reviewed Dec 24, 2022

View reviewed changes

numpy/__config__.py.in Outdated Show resolved Hide resolved

numpy/__config__.py.in Outdated Show resolved Hide resolved

numpy/__config__.py.in Outdated Show resolved Hide resolved

ganesh-k13 force-pushed the enh_config branch 2 times, most recently from d15c981 to eaf2b21 Compare December 24, 2022 14:38

ganesh-k13 commented Dec 24, 2022

View reviewed changes

numpy/tests/test_numpy_config.py Show resolved Hide resolved

rgommers reviewed Dec 24, 2022

View reviewed changes

numpy/__config__.py.in Outdated Show resolved Hide resolved

DOC: Documentation for new show_config function

6935fe9

ganesh-k13 force-pushed the enh_config branch from eaf2b21 to f4db5d0 Compare December 24, 2022 16:45

ganesh-k13 added 2 commits December 24, 2022 22:22

TST: Tests for new show_config

36d865d

* Adds a new env in `__config__` to indicate if built with meson

DOC: np.show_config improvements (numpy#22769)

4f5905a

ganesh-k13 force-pushed the enh_config branch from f4db5d0 to 4f5905a Compare December 24, 2022 16:53

ganesh-k13 marked this pull request as ready for review December 24, 2022 16:58

ganesh-k13 self-assigned this Dec 24, 2022

ganesh-k13 requested a review from rgommers January 3, 2023 15:23

MAINT: Removed Target from host info

a35447f

* Removed `Target` from host info * Fixed warning with using compiler `path` * Added `cross-compiled` flag

ganesh-k13 mentioned this pull request Jan 13, 2023

ENH: add support for fujitsu C/C++ compiler and SSL2 to numpy. #22982

Merged

rgommers added this to the 1.25.0 release milestone Jan 24, 2023

rgommers approved these changes Jan 24, 2023

View reviewed changes

rgommers merged commit 04823c1 into numpy:main Jan 24, 2023

ganesh-k13 added a commit to ganesh-k13/scipy that referenced this pull request Jan 26, 2023

BLD: Added meson based config generation

8dd8d0e

* Based on numpy/numpy#22769

This was referenced Jan 26, 2023

BLD: Meson __config__ generation scipy/scipy#17862

Merged

BUG: Handle arrays in conf_data #23098

Merged

ganesh-k13 mentioned this pull request Jan 26, 2023

BUG: Importing numpy causes error when the Windows username contains a single quote #22093

Closed

ganesh-k13 added a commit to ganesh-k13/scipy that referenced this pull request Jan 28, 2023

BLD: Added meson based config generation

cca74a2

* Based on numpy/numpy#22769

Uh oh!

Conversation

ganesh-k13 commented Dec 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

New __config__.py:

TODO

Uh oh!

ganesh-k13 commented Dec 10, 2022

Uh oh!

eli-schwartz commented Dec 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ganesh-k13 commented Dec 13, 2022

Uh oh!

rgommers commented Dec 13, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rgommers commented Dec 15, 2022

Uh oh!

ganesh-k13 Dec 16, 2022

Choose a reason for hiding this comment

Uh oh!

ganesh-k13 commented Dec 16, 2022

Uh oh!

eli-schwartz commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

ganesh-k13 commented Dec 19, 2022

Uh oh!

ganesh-k13 commented Dec 22, 2022

Uh oh!

Uh oh!

Uh oh!

ganesh-k13 commented Dec 24, 2022

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rgommers commented Dec 24, 2022

Uh oh!

Uh oh!

Uh oh!

rgommers commented Dec 24, 2022

Uh oh!

ganesh-k13 commented Dec 24, 2022

Uh oh!

rgommers commented Dec 24, 2022

Uh oh!

ganesh-k13 commented Jan 23, 2023

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

seberg commented Jan 26, 2023

Uh oh!

seberg commented Jan 26, 2023

Uh oh!

ganesh-k13 commented Jan 26, 2023

Uh oh!

seberg commented Jan 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ganesh-k13 commented Dec 10, 2022 •

edited

Loading

New `config.py`:

eli-schwartz commented Dec 13, 2022 •

edited

Loading

eli-schwartz commented Dec 16, 2022 •

edited

Loading