Building the Bootloader

Building and installing bootloader from source distribution or repository checkout

To build (and install) bootloader from source distribution or repository checkout, follow these steps:

• download and install Python, which is required for running waf,

• git clone or download the source from our GitHub repository,

• cd into the directory where you cloned the repository or unpacked the source,

• cd bootloader, and

• build the bootloader by running: python ./waf all,

• install the PyInstaller with rebuilt bootloader; first, change back to the parent (top-level) directory: cd .., and

• run pip install . or pip install -e . (for editable install)

• test the build by running (parts of) the test-suite.

This will produce the bootloader executables for your current platform (on Windows, these files will have the .exe extension):

• PyInstaller/bootloader/OS_ARCH/run,

• PyInstaller/bootloader/OS_ARCH/run_d,

• PyInstaller/bootloader/OS_ARCH/runw (macOS and Windows only), and

• PyInstaller/bootloader/OS_ARCH/runw_d (macOS and Windows only).

The bootloader’s architecture defaults to the machine’s one, but can be changed using the --target-arch option – given the appropriate compiler and development files are installed. For example, to build a 32-bit bootloader on a 64-bit machine, run:

python ./waf all --target-arch=32bit

If this reports an error, read the detailed notes that follow, then ask for technical help.

Forcing bootloader rebuild as part of package installation

Instead of performing manual steps from the previous section, it is also possible to force bootloader to be rebuild when installing PyInstaller directly through pip. This alternative might prove especially useful in setting up automated build pipelines.

To force bootloader to be rebuilt during PyInstaller package installation, set the PYINSTALLER_COMPILE_BOOTLOADER environment variable (to any value) before running pip. This will cause the installation process to (re)build the bootloader via the equivalent of running the python ./waf configure all command. You can pass additional arguments to the build process by setting the PYINSTALLER_BOOTLOADER_WAF_ARGS environment variable.

For the above to work, you also need to ensure that:

• prior installation of PyInstaller (if present) is removed: pip uninstall PyInstaller

• any cached wheels from previous PyInstaller installation attempts are removed, to prevent pip from re-using them: pip cache remove PyInstaller

• when running the pip install command, --no-binary=pyinstaller is added, to ensure PyInstaller is installed from source distribution (sdist), rather from a binary wheel

It is also recommended to use --verbose option with pip install command, so you can observe the bootloader’s build log and verify that it was, in fact, rebuild (as opposed, for example, to pip simply re-using and installing a pre-built wheel).

Therefore, the complete list of steps is as follows:

set PYINSTALLER_COMPILE_BOOTLOADER=1
set "PYINSTALLER_BOOTLOADER_WAF_ARGS=--example-option=example-value --example-flag"
python -m pip uninstall PyInstaller
python -m pip cache remove PyInstaller
python -m pip install --verbose --no-binary=PyInstaller PyInstaller

$env:PYINSTALLER_COMPILE_BOOTLOADER=1
$env:PYINSTALLER_BOOTLOADER_WAF_ARGS="--example-option=example-value --example-flag"
python -m pip uninstall PyInstaller
python -m pip cache remove PyInstaller
python -m pip install --verbose --no-binary=PyInstaller PyInstaller

export PYINSTALLER_COMPILE_BOOTLOADER=1
export PYINSTALLER_BOOTLOADER_WAF_ARGS="--example-option=example-value --example-flag"
python -m pip uninstall PyInstaller
python -m pip cache remove PyInstaller
python -m pip install --verbose --no-binary=PyInstaller PyInstaller

Building for GNU/Linux

Building for macOS

On macOS please install Xcode, Apple’s suite of tools for developing software for macOS. Instead of installing the full Xcode package, you can also install and use Command Line Tools for Xcode. Installing either will provide the clang compiler.

If the toolchain supports universal2 binaries, the 64-bit bootloaders are by default built as universal2 fat binaries that support both x86_64 and arm64 architectures. This requires a recent version of Xcode (12.2 or later). On older toolchains that lack support for universal2 binaries, a single-arch x86_64 thin bootloader is built. This behavior can be controlled by passing --universal2 or --no-universal2 flags to the waf build command. Attempting to use --universal2 flag and a toolchain that lacks support for universal2 binaries will result in configuration error.

The --no-universal2 flag leaves the target architecture unspecified letting the resultant executable’s architecture be the C compiler’s default (which is almost certainly the architecture of the build machine). Should you want to build a thin executable of either architecture, use the --no-universal2 flag and then optionally override the compiler, adding the -arch flag, via the CC environment variable.

Build a thin, native executable:

python waf --no-universal2 all

Build a thin, x86_64 executable (regardless of the build machine’s architecture):

CC='clang -arch x86_64' python waf --no-universal2  all

Build a thin, arm64 executable (regardless of the build machine’s architecture):

CC='clang -arch arm64' python waf --no-universal2 all

By default, the build script targets macOS 10.13, which can be overridden by exporting the MACOSX_DEPLOYMENT_TARGET environment variable.

export TARGET=OSX  # make the Vagrantfile build for macOS
vagrant up linux64 && vagrant halt linux64

mkdir -p ~/osxcross
tar -C ~/osxcross --xz -xf /vagrant/sdks/osx/osxcross.tar.xz
PATH=~/osxcross/bin/:$PATH
python ./waf all CC=x86_64-apple-darwin15-clang
python ./waf all CC=i386-apple-darwin15-clang

Building for Windows

The pre-compiled bootloader coming with PyInstaller are self-contained static executable that imposes no restrictions on the version of Python being used.

When building the bootloader yourself, you have to carefully choose between three options:

1. Using the Visual Studio C++ compiler.

   This allows creating self-contained static executables, which can be used for all versions of Python. This is why the bootloaders delivered with PyInstaller are build using Visual Studio C++ compiler.

   Visual Studio 2015 or later is required.

2. Using the MinGW-w64 suite.

   This allows to create smaller, dynamically linked executables, but requires to use the same level of Visual Studio [2] as was used to compile Python. So this bootloader will be tied to a specific version of Python.

   The reason for this is, that unlike Unix-like systems, Windows doesn’t supply a system standard C library, leaving this to the compiler. But Mingw-w64 doesn’t have a standard C library. Instead it links against msvcrt.dll, which happens to exist on many Windows installations – but is not guaranteed to exist.

3. Using cygwin and MinGW.

   This will create executables for cygwin, not for ‘plain’ Windows.

In all cases you may want

• to set the path to include python, e.g. set PATH=%PATH%;c:\python35,

• to peek into the Vagrantfile or ../appveyor.yml to learn how we are building.

You can also build the bootloaders for cygwin.

set PATH=C:\MinGW\bin;%PATH%

Building for AIX

• By default AIX builds 32-bit executables.

• For 64-bit executables set the environment variable OBJECT_MODE.

When creating a 64-bit build, the compiler and other AIX utilities that work with binary files (for example, the strip utility) may need to be passed the -X64 flag to force 64-bit mode. Rather than passing this flag to every command, the preferred way to provide this setting is to use the OBJECT_MODE environment variable.

Depending on whether you are using 32-bit or 64-bit Python build, you may therefore need to set or unset the OBJECT_MODE environment variable prior to running waf in order to build a matching type of the bootloader executable.

To determine whether you are using 32-bit or 64-bit Python, use the following command:

python -c "import sys; print(sys.maxsize <= 2**32)"

If the output of above command is True, your Python is 32-bit, and you should build bootloader using the following commands:

unset OBJECT_MODE
python ./waf all

Otherwise (64-bit Python), you should use the following commands:

export OBJECT_MODE=64
python ./waf all

python ./waf all --target-arch=64bit

To build the bootloader, you will need a compiler compatible (identical) with the one that was used to build Python itself.

To identify the compiler that was used to build Python, you can use the following command:

python -c "import sysconfig; print(sysconfig.get_config_var('CC'))"

If the compiler is gcc you may need additional RPMs installed to support the GNU run-time dependencies.

When the IBM compiler is used, no additional prerequisites are expected. The recommended value for CC environment variable with the IBM compiler is xlc_r.

Building for FreeBSD

A FreeBSD bootloader may be built with clang using the usual steps on a FreeBSD machine. Beware, however that any executable compiled natively on FreeBSD will only run on equal or newer versions of FreeBSD. In order to support older versions of FreeBSD, you must compile the oldest OS version you wish to support.

Alternatively, the FreeBSD bootloaders may be cross compiled from Linux using Docker and a FreeBSD cross compiler image. This image is kept in sync with the oldest non end of life FreeBSD release so that anything compiled on it will work on all active FreeBSD versions.

In a random directory:

• Start the docker daemon (usually with systemctl start docker - possibly requiring sudo if you haven’t setup rootless docker).

• Download the latest cross compiler .tar.xz image from here.

• Import the image: docker image load -i freebsd-cross-build.tar.xz. The cross compiler image is now saved under the name freebsd-cross-build. You may discard the .tar.xz file if you wish.

Then from the root of this repository:

• Run:

  docker run -v $(pwd):/io -it freebsd-cross-build bash -c "cd /io/bootloader; ./waf all"
  

Vagrantfile Virtual Machines

PyInstaller maintains a set of virtual machine description for testing and (cross-) building. For managing these boxes, we use vagrant.

All guests [3] will automatically build the bootloader when running vagrant up GUEST or vagrant provision GUEST. They will build both 32- and 64-bit bootloaders.

When building the bootloaders, the guests are sharing the PyInstaller distribution folder and will put the built executables onto the build-host (into ../PyInstaller/bootloader/).

Most boxes requires two Vagrant plugins to be installed:

vagrant plugin install vagrant-reload vagrant-scp

Example usage:

vagrant up linux64      # will also build the bootloader
vagrant halt linux64    # or `destroy`

# verify the bootloader has been rebuild
git status ../PyInstaller/bootloader/

You can pass some parameters for configuring the Vagrantfile by setting environment variables, like this:

GUI=1 TARGET=OSX vagrant up linux64

or like this:

export TARGET=OSX
vagrant provision linux64

We currently provide this guests:

linux64:

GNU/Linux (some recent version) used to build the GNU/Linux bootloaders.

• If TARGET=OSX is set, cross-builds the bootloaders for macOS (see Cross-Building for macOS).

• If TARGET=WINDOWS is set, cross-builds the bootloaders for Windows using mingw. Please have in mind that this imposes the restrictions mentioned above.

• Otherwise (which is the default) bootloaders for GNU/Linux are build.

windows10:

Windows 10, used for building the Windows bootloaders using Visual C++.

• If MINGW=1 is set, the bootloaders will be build using MinGW. Please be aware of the restrictions mentioned above.

Note

The Windows box uses password authentication, so in some cases you need to enter the password (which is Passw0rd!).

build-osxcross:

GNU/Linux guest used to build the macOS SDK and cctools as described in section Cross-Building for macOS.