What is system Python and why does it matter?
I've heard people say not to use "system Python", what is this and why not? What am I supposed to do instead?
2 answers
The system Python is for use by your system and it's better to not use it directly. It won't be good for you or the system.
Most distros will install Python in a system-wide location so that packages that need Python can work. You're not supposed to pip install stuff on this Python. Instead, if an additional Python package is needed, your package manager is expected to create the same files that pip install would, via packages named like python-requests.
If you run pip install on the system Python:
- When a new minor version of Python comes out all those packages will "disappear". Say you have Python 3.12,
pip installon system Python will put the files in/lib/python3.12/site-packages/. When Python 3.13 comes out and your package manager updates it, it will set up/lib/python3.13/site-packages/and move all thepython-package files there. But it won't move yourpip installpackages, because it doesn't know about them. You'll have to do it manually. Python releases minor versions frequently enough that this will create busywork over time. - Your packages will conflict with the package manager. Say you
pip install requestsand the files go in/lib/python3.12/site-packages/. Later, your package manager tries to installpython-requestswhich goes in the same place. You will get errors about "file already exists". - There's probably safeguards like forcing you to pass
--break-system-packagesto do it, so it will be tedious.
Luckily you don't have to deal with this stuff at all. Set up a tool like https://github.com/pyenv/pyenv and you will still have commands like python (which now, for you, point to an isolated Python and not the system one). Your Python packages won't be affected by system upgrades. You can easily run multiple versions side by side and switch them.
You can also manually create a Python venv somewhere like your home directory. This doesn't have all the conveniences of a tool like pyenv though, so usually isn't worth the effort.
0 comment threads
Overview
Modern Linux distributions generally come with Python pre-installed; but fundamentally this copy of Python is not provided for experimentation or general software development, but rather to support other applications and tools in the distro.
As a developer, it won't meet your needs as-is unless all your dependencies are already available as system packages, and this route is still probably only a good idea if you have the means to distribute a system package. (In fact, some of these distributions of Python are even missing parts of the standard library.) As an end user, while Python is generally cross-platform, if you want to install code not packaged by your distro, you will have to take extra steps.
There are really two separate concerns here: the Python executable itself, and the system Python environment — meaning, basically, the path(s) where system packages will install Python code, and where Python will look for libraries by default at startup.
The executable
Generally, there's no problem with using the provided python executable (or any symlinks etc. it may have). In fact, the most common strategies for dealing with environment problems depend on this. However, attempting to upgrade this Python, or to install an additional version in the usual place, may cause serious breakages. If for example you build a separate version of Python from source, it will be better to ./configure it to use a different install prefix (such as /opt or /usr/local, rather than the default /usr) and make altinstall to put the installed build there instead. You may also want to rename the executable, or otherwise ensure that it doesn't shadow the system Python: make sure that system scripts with a #!/usr/bin/python or #!/usr/bin/python3 shebang will run with the original system Python executable.
The environment
Installer tools like pip often default to installing in a folder relative to the target Python, which Python automatically configures at startup to be searched for third-party code. This can still work as an unprivileged user (and you should never use sudo with these tools), installing into a user-specific folder; but when Python (and therefore these system scripts) run as you, they will pick up that user-specific folder, which can cause interference.
Modern Linux distros therefore implement an additional protection, implemented as of the Python 3.11 release in 2022. This represents cooperation between third-party Python tools (pip is not made by the core Python development team) and Linux distros: the distro places a marker file to signal that a system package manager is responsible for packages installed there, and Python package tools respect that by default.
Circumventing this protection is possible, but strongly discouraged. In the worst case, interference with the system Python environment could break (among other things) your written-in-Python system package manager, making recovery difficult. Also, if you happen to install malware into the system environment, ordinary system tools could be tricked into running that malware, when it might otherwise sit inactive until you explicitly imported some code from your own project.
How to avoid the system environment
The standard isolation technique is the virtual environment. There are many tools you can use to create these, manage them, control when they're used, etc.; but fundamentally they all work in the same way. A separate folder hierarchy is created similar to the one for system packages, and some symlinks to the "base" Python are put in appropriate places in this hierarchy. Additionally, a short pyvenv.cfg config file is added, as well as some "activation" scripts that allow for some convenient (and generally optional!) manipulation of environment variables[1]. When Python starts from the symlink, it looks for the corresponding config file; if it's found and validated, Python will configure its default search path for Python code, to use the virtual environment's folders instead of the system's.
The Python standard library provides a venv module (on Debian and derivatives, you will likely need to install a separate package for this, even though it's supposed to be part of the standard library) that creates virtual environments for you. This includes a simple command-line interface: e.g. python -m venv my-venv. By default, these virtual environments will include a bootstrapped installation of pip; the bootstrap process may take several seconds and use ~15 megabytes of additional disk space, per environment. You can avoid this by passing --without-pip to the command, but then you'll need a separate copy of pip (or another installer) to install into that environment. With pip, pass the --python argument (with the path to the virtual envrionment's symlink) to choose the target installation environment.[2]
-
Pedantically, a virtual environment doesn't need to contain the activation scripts to be valid — but all the standard ways to set up a virtual environment will create them. ↩︎
-
This requires pip version 22.3 or later, and requires that your copy of pip is compatible with the version of Python that the virtual environment uses. For most people this will rarely if ever cause a problem. Using pip with
--pythonis also slightly slower, since the program will basically restart itself in a new process under the specified Python. Other installation tools may avoid issues like these. ↩︎
0 comment threads
0 comment threads