Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more details about the Python build in sys.version #100086

Closed
vstinner opened this issue Dec 7, 2022 · 7 comments
Closed

Add more details about the Python build in sys.version #100086

vstinner opened this issue Dec 7, 2022 · 7 comments
Labels
type-bug An unexpected behavior, bug, or error

Comments

@vstinner
Copy link
Member

vstinner commented Dec 7, 2022

Currently, it's not easy to guess how Python was built just by looking at python -VV output or sys.version.

I propose to enhance sys.version to include more details about how Python was configured for the build:

  • debug or release build
  • "shared" or "framework" build
  • assertions enabled or not
  • address/memory/undefined behavior sanitizer
  • etc.

It should help me to more quickly identify if a buildbot was built in debug mode or release mode.

It should help to see how Python was optimized: with or without LTO and PGO optimizations?

It should help to quickly have an idea of which ABI is used: the "pystats" ABI is different than the "debug" than the "release" ABI.

Linked PRs

@vstinner vstinner added the type-bug An unexpected behavior, bug, or error label Dec 7, 2022
@ned-deily
Copy link
Member

ned-deily commented Dec 7, 2022

I don't think this change should be made. It potentially affects all users of Python, for example, the REPL users, but is of no benefit to 99% of them. All of this information is readily available via multiple other mechanisms, in particular, via python3 -m test.pythoninfo or via sysconfig.get_config_vars(). And as implemented in the PR, the information is truncated when displayed due to the string format limitation in getversion.c and can make the header unnecessarily long:

Python 3.12.0a3+ (heads/py_build_str-dirty:94b93efeed, Dec 7 2022, 16:23:18, release framework bu) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin

@vstinner
Copy link
Member Author

vstinner commented Dec 7, 2022

One motivation for me is that many users don't use Python binaries from python.org or from their Linux packaging system, but build their own binary with their custom recipe and it's not trivial to guess how the binary was built. Was it properly optimized with LTO+PGO? Was it built in debug mode?

For a long time, the most popular Docker recipe to build Python used ./configure && make: no LTO, no PGO.

Today, I cannot recall if the macOS binary is optimized with LTO or not.

Recently, @JulienPalard shared his recipe to build a custom Python with few lines of shell code: compile-python.sh. I was surprised to see --with-pydebug in the middle of the script: why building Python in debug mode? Some users may use it without paying attention to this configure flag.

in particular, via python3 -m test.pythoninfo or via sysconfig.get_config_vars()

Well, these APIs are not great :-(

test.pythoninfo is designed to be used on the command line, it doesn't have a simple API to query info about the current Python.

sysconfig.get_config_vars() is hard to use: you have to guess which variable is the best to get compiler flags: make your choice in this long list, it can depend on how Python was built, in case of doubt you might have to check multiple variables. Moreover, once you have compiler options, you have to parse this string which is not trivial (I hate parsing shell). For LTO, do you have to check compiler flags or linker flags, hum?

@vstinner
Copy link
Member Author

vstinner commented Dec 7, 2022

It potentially affects all users of Python, for example, the REPL users, but is of no benefit to 99% of them.

Ah? I'm not the 99%, I build Python on a daily basis and work on Python built in various ways. I know that :-)

I saw this "debug build" vs "release build" as an useful information, but maybe it should be omitted for the common case: "release build" string? What about "lto+pgo": is it useful for most users?

On my Fedora 37, sys.version is Python 3.11.0 (main, Oct 24 2022, 00:00:00) [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)] on linux.

String length:

  • Python version: 8% (6 characters)
  • space + Build info: 39% (30 characters)
  • space + compiler name and version: 53% (41 characters)
  • Total: 77 characters

To make this string longer, the compiler part should be shorter: [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)] on linux.

@vstinner
Copy link
Member Author

vstinner commented Dec 7, 2022

At the beginning, I started by modifying test.libregrtest to add some "build details" in the first lines of the test runner:

== CPython 3.12.0a2+ (heads/main-dirty:9dc787ea96, Dec 6 2022, 11:51:12) [GCC 9.4.0]
== Linux-5.15.0-1019-azure-x86_64-with-glibc2.31 little-endian
== cwd: /opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/build_oot/host/build/test_python_829492æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Using random seed 7151121
0:00:00 load avg: 7.01 Run tests in parallel using 2 child processes (timeout: 15 min, worker timeout: 20 min)

These lines don't say if Python was built in debug mode or not. The test.pythonfo data is not far in CIs, but it's annoying to have to reach the test.pythoninfo step, unfold it, scroll (this is especially painful in the buildbot web UI), just to check if Python was built in debug mode or not.

The first line is coming from the code:

        print("==", platform.python_implementation(), *sys.version.split())

Since sys.version is used, I thought that maybe putting the info directly in sys.version can benefit to more users.

@vstinner
Copy link
Member Author

vstinner commented Dec 7, 2022

About the REPL, python2 and pypy put the info on two lines which makes it more readable IMO:

$ python2
Python 2.7.18 (default, Aug 22 2022, 00:00:00) 
[GCC 12.2.1 20220819 (Red Hat 12.2.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

and

$ pypy3
Python 3.9.12 (2fc6706f5902, Oct 12 2022, 21:40:58)
[PyPy 7.3.9 with GCC 12.2.1 20220819 (Red Hat 12.2.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> 

The single line format (sys.version) is also used by the python -VV command:

$ python3 -VV
Python 3.11.0 (main, Oct 24 2022, 00:00:00) [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)]

@vstinner
Copy link
Member Author

vstinner commented Dec 7, 2022

I wrote PR #100093 which only changes the Python test runner (libregrtest), so it doesn't change the Python REPL nor python -VV. It doesn't change sys.version, so the platform module doesn't need to be updated.

vstinner added a commit that referenced this issue Dec 8, 2022
The Python test runner (libregrtest) now logs Python build information like
"debug" vs "release" build, or LTO and PGO optimizations.
@vstinner
Copy link
Member Author

vstinner commented Dec 8, 2022

Fixed by: 3c89202

@vstinner vstinner closed this as completed Dec 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

2 participants