BLD, ENH: Display compiler configuration in `show_config` by ganesh-k13 · Pull Request #20939 · numpy/numpy

ganesh-k13 · 2022-01-29T16:43:02Z

Display Compiler Configuration

Output of np.show_config truncated till needed portion::

# CFLAGS=$CFLAGS" -Wno-maybe-uninitialized" python3.10d setup.py build_ext -i --cpu-dispatch=avx -j 12 | tee log
>>> np.show_config()
...
C compiler flags used in this NumPy install:
    compiler type = unix
    compiler name = x86_64-linux-gnu-gcc
+++    build flags = ['-pthread', '-Wno-unused-result', '-Wsign-compare', '-g', '-Og', '-Wall', '-g', '-Og', '-ffile-prefix-map=/build/python3.10-18rSsq/python3.10-3.10.0~b1=.', '-fstack-protector-strong', '-Wformat', '-Werror=format-security', '-g', '-Og', '-ffile-prefix-map=/build/python3.10-18rSsq/python3.10-3.10.0~b1=.', '-fstack-protector-strong', '-Wformat', '-Werror=format-security', '-Wno-maybe-uninitialized', '-Wdate-time', '-D_FORTIFY_SOURCE=2', '-fPIC']
    linker flags = ['-pthread', '-shared', '-Wl,-O1', '-Wl,-Bsymbolic-functions', '-Wl,-Bsymbolic-functions', '-Wl,-z,relro', '-Wl,-Bsymbolic-functions', '-Wl,-z,relro', '-Wno-maybe-uninitialized', '-Wdate-time', '-D_FORTIFY_SOURCE=2']
    compiler preprocessors = ['x86_64-linux-gnu-gcc', '-pthread', '-E', '-Wdate-time', '-D_FORTIFY_SOURCE=2']
    include dirs = ['numpy/core/src/common', 'numpy/core/src', 'numpy/core', 'numpy/core/src/npymath', 'numpy/core/src/multiarray', 'numpy/core/src/umath', 'numpy/core/src/npysort', 'numpy/core/src/_simd', '/usr/include/python3.10d', 'numpy/core/src/common', 'numpy/core/src/npymath']
CPP compiler flags used in this NumPy install:
    compiler type = unix
    compiler name = x86_64-linux-gnu-g++
+++    build flags = ['-pthread', '-Wno-unused-result', '-Wsign-compare', '-g', '-Og', '-Wall', '-g', '-Og', '-ffile-prefix-map=/build/python3.10-18rSsq/python3.10-3.10.0~b1=.', '-fstack-protector-strong', '-Wformat', '-Werror=format-security', '-g', '-Og', '-ffile-prefix-map=/build/python3.10-18rSsq/python3.10-3.10.0~b1=.', '-fstack-protector-strong', '-Wformat', '-Werror=format-security', '-Wno-maybe-uninitialized', '-Wdate-time', '-D_FORTIFY_SOURCE=2', '-fPIC']
    linker flags = ['-pthread', '-shared', '-Wl,-O1', '-Wl,-Bsymbolic-functions', '-Wl,-Bsymbolic-functions', '-Wl,-z,relro', '-Wl,-Bsymbolic-functions', '-Wl,-z,relro', '-Wno-maybe-uninitialized', '-Wdate-time', '-D_FORTIFY_SOURCE=2']
    compiler preprocessors = ['x86_64-linux-gnu-gcc', '-pthread', '-E', '-Wdate-time', '-D_FORTIFY_SOURCE=2']
    include dirs = ['numpy/core/src/common', 'numpy/core/src', 'numpy/core', 'numpy/core/src/npymath', 'numpy/core/src/multiarray', 'numpy/core/src/umath', 'numpy/core/src/npysort', 'numpy/core/src/_simd', '/usr/include/python3.10d', 'numpy/core/src/common', 'numpy/core/src/npymath']

# python3.10d setup.py build_ext -i --cpu-dispatch=avx -j 12 | tee log
>>> np.show_config()
...
C compiler flags used in this NumPy install:
    compiler type = unix
    compiler name = x86_64-linux-gnu-gcc
---    build flags = ['-pthread', '-Wno-unused-result', '-Wsign-compare', '-g', '-Og', '-Wall', '-g', '-Og', '-ffile-prefix-map=/build/python3.10-18rSsq/python3.10-3.10.0~b1=.', '-fstack-protector-strong', '-Wformat', '-Werror=format-security', '-g', '-Og', '-ffile-prefix-map=/build/python3.10-18rSsq/python3.10-3.10.0~b1=.', '-fstack-protector-strong', '-Wformat', '-Werror=format-security', '-g', '-Og', '-ffile-prefix-map=/build/python3.10-18rSsq/python3.10-3.10.0~b1=.', '-fstack-protector-strong', '-Wformat', '-Werror=format-security', '-Wdate-time', '-D_FORTIFY_SOURCE=2', '-fPIC']
    linker flags = ['-pthread', '-shared', '-Wl,-O1', '-Wl,-Bsymbolic-functions', '-Wl,-Bsymbolic-functions', '-Wl,-z,relro', '-Wl,-Bsymbolic-functions', '-Wl,-z,relro', '-g', '-Og', '-ffile-prefix-map=/build/python3.10-18rSsq/python3.10-3.10.0~b1=.', '-fstack-protector-strong', '-Wformat', '-Werror=format-security', '-Wdate-time', '-D_FORTIFY_SOURCE=2']
    compiler preprocessors = ['x86_64-linux-gnu-gcc', '-pthread', '-E', '-Wdate-time', '-D_FORTIFY_SOURCE=2']
    include dirs = ['numpy/core/src/common', 'numpy/core/src', 'numpy/core', 'numpy/core/src/npymath', 'numpy/core/src/multiarray', 'numpy/core/src/umath', 'numpy/core/src/npysort', 'numpy/core/src/_simd', '/usr/include/python3.10d', 'numpy/core/src/common', 'numpy/core/src/npymath']
CPP compiler flags used in this NumPy install:
    compiler type = unix
    compiler name = x86_64-linux-gnu-g++
---    build flags = ['-pthread', '-Wno-unused-result', '-Wsign-compare', '-g', '-Og', '-Wall', '-g', '-Og', '-ffile-prefix-map=/build/python3.10-18rSsq/python3.10-3.10.0~b1=.', '-fstack-protector-strong', '-Wformat', '-Werror=format-security', '-g', '-Og', '-ffile-prefix-map=/build/python3.10-18rSsq/python3.10-3.10.0~b1=.', '-fstack-protector-strong', '-Wformat', '-Werror=format-security', '-g', '-Og', '-ffile-prefix-map=/build/python3.10-18rSsq/python3.10-3.10.0~b1=.', '-fstack-protector-strong', '-Wformat', '-Werror=format-security', '-Wdate-time', '-D_FORTIFY_SOURCE=2', '-fPIC']
    linker flags = ['-pthread', '-shared', '-Wl,-O1', '-Wl,-Bsymbolic-functions', '-Wl,-Bsymbolic-functions', '-Wl,-z,relro', '-Wl,-Bsymbolic-functions', '-Wl,-z,relro', '-g', '-Og', '-ffile-prefix-map=/build/python3.10-18rSsq/python3.10-3.10.0~b1=.', '-fstack-protector-strong', '-Wformat', '-Werror=format-security', '-Wdate-time', '-D_FORTIFY_SOURCE=2']
    compiler preprocessors = ['x86_64-linux-gnu-gcc', '-pthread', '-E', '-Wdate-time', '-D_FORTIFY_SOURCE=2']
    include dirs = ['numpy/core/src/common', 'numpy/core/src', 'numpy/core', 'numpy/core/src/npymath', 'numpy/core/src/multiarray', 'numpy/core/src/umath', 'numpy/core/src/npysort', 'numpy/core/src/_simd', '/usr/include/python3.10d', 'numpy/core/src/common', 'numpy/core/src/npymath']

Notice the extra '-Wno-maybe-uninitialized' in case of the first output with extra CFLAGS. Not sure how and why the extra flags are coming for normal compilation :)

On a side note, I tried many things, like injecting the variables into globals of __config__.py, but given there are two files and we will depend on the fact that format does not change is risky. Moving the generation to a later time seems hard, quite frankly I don't think we can. If we do, however, we can add a subclass of system_info which would be pretty.

I wanted to add FORTRAN as well but thought let's see how this goes first.

resolves: #10983

mattip · 2022-01-30T10:45:51Z

This is getting a bit unwieldy. Could we add some kind of structure to the copious output? Maybe a flag output=str (default, like today) with a output=dict non-default variant, that would then allow users to somehow filter what they get back

ganesh-k13 · 2022-01-30T10:57:32Z

Yeah completely agree. I can rework the output format for both proposed str and dict. We could add other modes like CSV as well?

mattip · 2022-01-30T11:27:19Z

Whatever makes sense. We can spread out the work over a few PRs if the choices get complicated.

ganesh-k13 · 2022-02-05T05:03:53Z

Writing to a python file does not seem ideal. Let me try dumping to a YAML and reconstructing the data. Can add flexibility in display this way. I'll try it out, I'm ok to revert back anytime. Moving to draft for now.

* Dumps compiler related configurations into an YAML file * Only C and CPP for now. WIP: Dump system info into a YAML file * This will be read later and replace the globals in `get_config` and `show` MAINT: Added generated YAMLs to .gitignore

ganesh-k13 · 2022-02-20T13:36:29Z

Sample generated YAMLs: https://gist.github.com/ganesh-k13/a7968fc287557a7fc791c9462149d2da

The plan is to remove all globals calls in __config__ and load these YAMLs. I'll pick up SIMD stuff from numpy.core._multiarray_umath and refine the display. Hoping to make it extensible and make it easy to add new display stuff.

ganesh-k13 · 2022-12-03T12:30:26Z

With Meson coming in, I'm going to try something with that. Will close this for now.

rgommers · 2022-12-04T21:39:29Z

@ganesh-k13 here is what's probably the best way to go about using Meson for this kind of info: scipy/scipy#16803 (comment)

ganesh-k13 · 2022-12-05T03:01:10Z

Ah that's pretty neat! I'll start a new PR by following that issue, always wanted to refactor the __config__. Thanks @rgommers!

github-actions bot added the 01 - Enhancement label Jan 29, 2022

ganesh-k13 marked this pull request as draft February 5, 2022 05:05

BLD: Generate compiler configuration YAML

6e39546

* Dumps compiler related configurations into an YAML file * Only C and CPP for now. WIP: Dump system info into a YAML file * This will be read later and replace the globals in `get_config` and `show` MAINT: Added generated YAMLs to .gitignore

ganesh-k13 force-pushed the enh_10983_compiler_config branch from b2ca2c0 to 6e39546 Compare February 20, 2022 13:27

ganesh-k13 added the 36 - Build Build related PR label Feb 20, 2022

ganesh-k13 changed the title ~~ENH: Display compiler configuration in show_config~~ BLD, ENH: Display compiler configuration in show_config Feb 20, 2022

ganesh-k13 mentioned this pull request May 6, 2022

ENH: Use threadpoolctl in show_runtime (a new function) #21468

Merged

ganesh-k13 closed this Dec 3, 2022

ganesh-k13 mentioned this pull request Dec 10, 2022

BLD: Meson __config__ generation #22769

Merged

8 tasks

ganesh-k13 mentioned this pull request Jan 13, 2023

ENH: add support for fujitsu C/C++ compiler and SSL2 to numpy. #22982

Merged

ganesh-k13 mentioned this pull request Jan 26, 2023

BLD: Meson __config__ generation scipy/scipy#17862

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

BLD, ENH: Display compiler configuration in `show_config`#20939

BLD, ENH: Display compiler configuration in `show_config`#20939
ganesh-k13 wants to merge 1 commit intonumpy:mainfrom
ganesh-k13:enh_10983_compiler_config

ganesh-k13 commented Jan 29, 2022

Uh oh!

mattip commented Jan 30, 2022

Uh oh!

ganesh-k13 commented Jan 30, 2022

Uh oh!

mattip commented Jan 30, 2022

Uh oh!

ganesh-k13 commented Feb 5, 2022

Uh oh!

ganesh-k13 commented Feb 20, 2022

Uh oh!

ganesh-k13 commented Dec 3, 2022

Uh oh!

rgommers commented Dec 4, 2022

Uh oh!

ganesh-k13 commented Dec 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ganesh-k13 commented Jan 29, 2022

Display Compiler Configuration

Uh oh!

mattip commented Jan 30, 2022

Uh oh!

ganesh-k13 commented Jan 30, 2022

Uh oh!

mattip commented Jan 30, 2022

Uh oh!

ganesh-k13 commented Feb 5, 2022

Uh oh!

ganesh-k13 commented Feb 20, 2022

Uh oh!

ganesh-k13 commented Dec 3, 2022

Uh oh!

rgommers commented Dec 4, 2022

Uh oh!

ganesh-k13 commented Dec 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants