When Python is configured with:

./configure --enable-optimizations

PGO is enabled but not LTO.

I recall that a few years ago, GCC with LTO had bugs. But now, GCC with LTO is reliable. I suggest to enable it by default in Python 3.11.

Or did I miss a reason to not do that?

IIRC activating lto is specially annoying on MacOS due to the need of some llvm components that are a bit hard to get. Let me dig a bit to see if I reproduce the problem

Pablo:

IIRC activating lto is specially annoying on MacOS due to the need of some llvm components that are a bit hard to get. Let me dig a bit to see if I reproduce the problem

Ah, I guess that you are referring to this requirement:
"The C compiler Clang requires llvm-ar for LTO (ar on macOS), as well as an LTO-aware linker (ld.gold or lld)."
https://docs.python.org/dev/using/configure.html#cmdoption-with-lto

Maybe configure can enable LTO on all platforms but macOS.

IIRC activating lto is specially annoying on MacOS due to the need of some llvm components that are a bit hard to get

Can you say more? We are currently using --with-lto with a vanilla Apple Command Line Tools (or Xcode) for macOS installer builds when building on macOS 10.15 High Sierra or higher. Perhaps this was just an issue on older versions.

Yeah, I had problems in the past to get llvm-ar or some other component. I still need time to reproduce and to see if this still happens on new versions.

I don't think you need llvm-ar anymore with the Apple Tool Chain but let me look into it as I have all the relevant previous macOS releases as VMs to test with.

FWIW I agree that we should try adding LTO to --enable-optimizations now.

Sorry, this slipped off my radar and I haven't gone back and checked older versions of macOS. But it certainly is true that at least with the current versions of the Apple Developer Tools (either the Command Line Tools or Xcode) for macOS 11 (Big Sur) and macOS 12 (Monterey), things just work.

Can we use --lto=thin when availabe?
And can we not use --lto when building profiling python?

One data point: when I enabled --with-lto, the cpython build task failed on my container build host due to an out of memory condition. It seems that gcc uses a lot more RAM in that case (host has 2 GB, I think). I fixed it by adding a swap file but I suspect if we make it to default on, more people will run into the issue.

@methane san cc @ned-deily

I would like to switch the default LTO policy to the thinLTO if available from Python3.12.
The thinLTO option is only introduced from Python3.11 so let's leave this option as an experimental choice for Python 3.11 :)

AS-IS

./configure --with-lto  # fullLTO
./configure --with-lto=full  # fullLTO
./configure --with-lto=thin  # thinLTO

##TO-BE

./configure --with-lto # thinLTO if possible if not fullLTO
./configure --with-lto=full  # fullLTO
./configure --with-lto=thin  # thinLTO

I like faster LTO by default. Because default LTO is slow.

Random comments:

I don't want to use LTO while profiling build (1st build in PGO).

What about GCC? GCC supports -flto=auto option since GCC 10.
https://www.phoronix.com/scan.php?page=news_item&px=GCC-flto-auto-option

I don't have an opinion yet. I plan to compare both options on macOS.

I don't want to use LTO while profiling build (1st build in PGO).

+1

What about GCC? GCC supports -flto=auto option since GCC 10

We may need to add the option for ./configure --with-lto=auto and I think that we have a chance to add it to Python 3.11
Also in that case, from Python 3.12 ./configure --with-lto should use auto option if available, if not should use the default LTO policy for GCC.

So. Can we enable LTO in Python 3.12? Is there a way to opt-out from LTO when --enable-optimizations is used? For example, does ./configure --enable-optimizations --without-lto work as expected?

Is it possible to disable LTO if it's known to not work properly / if the toolchain is too old? I'm thinking at Ned's toolchain to build Python on macOS for x86 CPU.