Where Are Python Packages Installed in Linux? An In-Depth Guide for Developers

As Python continues its rapid growth as one of the world‘s most popular programming languages, Linux has cemented itself as the go-to development environment for Python programmers. With its open source ethos, excellent terminal access, variety of text editors, and easy environment configuration, Linux offers unparalleled flexibility for Python developers.

However, one key aspect of working in this ecosystem is properly understanding how Python packages are installed and managed in a Linux environment. Having clarity on where your packages reside and how they integrate with the operating system will enable smoother development workflows and quicker debugging when issues arise.

In this comprehensive guide, you will gain expert insight into Python package management within Linux. We will cover:

• Prerequisite steps for package installation
• The key directories where Python packages are stored
• How these locations differ based on install method and Linux distribution
• Portable techniques for finding packages independent of environment
• Accessing available Python modules
• Best practices for Python virtual environments

Along the way, you will gain hands-on examples and a full-stack perspective on working with packages effectively as a Python developer within Linux. Let‘s dig in!

Prerequisites: Install pip and check Python versions

Before installing external Python packages, you should ensure pip is available in your environment. Pip is the de facto standard package installer for Python. To install it:

sudo apt install python3-pip python-pip

This will install pip for both Python 2.x and Python 3.x if you have multiple versions.

With pip installed, check what Python executables are present on your system:

ls -1 /usr/bin/python*

Take note of the major.minor versions found, as we will reference them moving forward.

Source Installs, setup.py, Pip, and System Package Manager

There are four main methods of installing Python packages within Linux:

1. Installing from source: Manually compiling from downloaded source code
2. Using a package‘s setup.py script
3. The pip installer
4. A distribution‘s system package manager (apt, dnf, etc.)

The directory your packages are installed into differs based on the method used:

Install Method	Directory	Example
From source / setup.py	/usr/local/lib/pythonX.Y/	/usr/local/lib/python3.8/site-packages
pip	/usr/local/lib/pythonX.Y/dist-packages	/usr/local/lib/python3.7/dist-packages
System package manager	/usr/lib/pythonX.Y/	/usr/lib/python2.7/dist-packages

Where X.Y refers to your Python version.

These paths can vary slightly between Linux distributions, but this breakdown covers the standard conventions.

Now let‘s look at some examples of identifying package locations based on install method.

Seeing Source or setup.py-Installed Packages

If we installed the popular data analysis library Pandas from source or using its setup.py file, it would land under the default source path:

/usr/local/lib/python3.8/site-packages

Listing this directory shows pandas installed:

ls /usr/local/lib/python3.8/site-packages

Output:
pandas

So Pandas lives at:

/usr/local/lib/python3.8/site-packages/pandas

This directory remains consistent even when switching Linux distributions, owing to its independence from system package managers.

Viewing pip-Installed Packages

To display packages installed with pip, use:

pip list

Here we see numpy, scipy, matplotlib and other common data science packages:

These are installed under the pip dist-packages path:

/usr/local/lib/python3.7/dist-packages 

So the numpy module resides at:

/usr/local/lib/python3.7/dist-packages/numpy

Again this path persists across Linux distributions thanks to pip consistency.

Checking System Package Manager-Installed Packages

To see Python packages installed via apt on Ubuntu Linux, list the apt packages path:

ls /usr/lib/python3/dist-packages

The certifi SSL certificate module is installed here under /usr/lib/python3/ so we can find it at:

/usr/lib/python3/dist-packages/certifi

Other distributions have similar default system paths for Python packages.

Portable Method: Finding Packages with find

Instead of memorizing the various locations packages can be installed to, we can use the find command to search for packages based on their .py file extension.

find / -type f -name "*.py"

This recursively searches the entire Linux file system and prints out filenames ending in .py.

In the truncated output we can see the different site-packages paths identified:

Using find allows you to locate installed Python packages regardless of how they were installed or which distribution you are using.

Checking Available Python Modules

In addition to finding package installation locations, you can also check what Python modules are available to import within your programs.

In the Python interpreter:

import sys
help("modules")

The output displays all built-in and third-party modules that can be imported:

Reviewing these modules helps identify what functionality you have access to.

Best Practice: Leverage Virtual Environments

While understanding global package installation paths is useful, best practice for Python development is using virtual environments.

Virtualenvs allow you to isolate all dependencies on a per-project basis, removing issues of package version conflicts across projects.

They also allow you to establish a consistent, portable environment across different machines. This enhances reproducibility and team collaboration when developing projects.

I recommend always developing Python applications within virtual environments. The virtualenv and pipenv packages are excellent options for managing isolated, self-contained dev environments.

Key Takeaways and Next Steps

We covered several key topics regarding Python packages within the Linux landscape:

• The major installation paths packages can land in
• How locations differ based on install method
• Using find for portable package discovery
• Checking available modules
• Leveraging virtual environments

With this solid conceptual foundation, you are equipped to effectively work with Python packages across projects. Some recommended next steps:

• Audit your current projects‘ dependencies using pip freeze and pip list
• Standardize your future projects within isolated virtual environments
• Use python -m site to bring deeper insight into your site-packages directories
• Learn to compile any C-based extensions needed for your packages

Understanding Python package management will accelerate your productivity and proficiency in harnessing the vast breadth of Python‘s ecosystem across any Linux environment.