When would the -e, --editable option be useful with pip install?

Concrete example of using --editable in development

If you play with this test package as in:

cd ~
git clone https://github.com/cirosantilli/vcdvcd
cd vcdvcd
git checkout 5dd4205c37ed0244ecaf443d8106fadb2f9cfbb8
python -m pip install --editable . --user

it outputs:

Obtaining file:///home/ciro/bak/git/vcdvcd
Installing collected packages: vcdvcd
  Attempting uninstall: vcdvcd
    Found existing installation: vcdvcd 1.0.6
    Can't uninstall 'vcdvcd'. No files were found to uninstall.
  Running setup.py develop for vcdvcd
Successfully installed vcdvcd-1.0.6

The Can't uninstall 'vcdvcd' is normal: it tried to uninstall any existing vcdvcd to then replace them with the "symlink-like mechanism" that is produced in the following steps, but failed because there were no previous installations.

Then it generates a file:

~/.local/lib/python3.8/site-packages/vcdvcd.egg-link

which contains:

/home/ciro/vcdvcd
.

and acts as a "symlink" to the Python interpreter.

So now, if I make any changes to the git source code under /home/ciro/vcdvcd, it reflects automatically on importers who can from any directory do:

python -c 'import vcdvcd'

Note however that at my pip version at least, binary files installed with --editable, such as the vcdcat script provided by that package via scripts= on setup.py, do not get symlinked, just copied to:

~/.local/bin/vcdcat

just like for regular installs, and therefore updates to the git repository won't directly affect them.

By comparison, a regular non --editable install from the git source:

python -m pip uninstall vcdvcd
python -m pip install --user .

produces a copy of the installed files under:

~/.local/lib/python3.8/site-packages/vcdvcd

Uninstall of an editable package as done above requires a new enough pip as mentioned at: How to uninstall editable packages with pip (installed with -e)

Tested in Python 3.8, pip 20.0.2, Ubuntu 20.04.

Recommendation: develop directly in-tree whenever possible

The editable setup is useful when you are testing your patch to a package through another project.

If however you can fully test your change in-tree, just do that instead of generating an editable install which is more complex.

E.g., the vcdvcd package above is setup in a way that you can just cd into the source and do ./vcdcat without pip installing the package itself (in general, you might need to install dependencies from requirements.txt though), and the import vcdvcd that that executable does (or possibly your own custom test) just finds the package correctly in the same directory it lives in.

From Working in "development" mode:

Although not required, it’s common to locally install your project in “editable” or “develop” mode while you’re working on it. This allows your project to be both installed and editable in project form.

Assuming you’re in the root of your project directory, then run:

pip install -e .

Although somewhat cryptic, -e is short for --editable, and . refers to the current working directory, so together, it means to install the current directory (i.e. your project) in editable mode.

Some additional insights into the internals of setuptools and distutils from “Development Mode”:

When creating a Python project, developers usually want to implement and test changes iteratively, before cutting a release and preparing a distribution archive.

In normal circumstances this can be quite cumbersome and require the developers to manipulate the PYTHONPATH environment variable or to continuously re-build and re-install the project.

To facilitate iterative exploration and experimentation, setuptools allows users to instruct the Python interpreter and its import machinery to load the code under development directly from the project folder without having to copy the files to a different location in the disk. This means that changes in the Python source code can immediately take place without requiring a new installation.

You can enter this “development mode” by performing an editable installation inside of a virtual environment, using pip’s -e/--editable flag, as shown below:

$ cd your-python-project
$ python -m venv .venv
# Activate your environment with:
#      `source .venv/bin/activate` on Unix/macOS
# or   `.venv\Scripts\activate` on Windows

$ pip install --editable .

# Now you have access to your package
# as if it was installed in .venv
$ python -c "import your_python_project"

An “editable installation” works very similarly to a regular install with pip install ., except that it only installs your package dependencies, metadata and wrappers for console and GUI scripts. Under the hood, setuptools will try to create a special .pth file in the target directory (usually site-packages) that extends the PYTHONPATH or install a custom import hook.

It is important to note that pip uninstall can not uninstall a module that has been installed with pip install -e. So if you go down this route, be prepared for things to get very messy if you ever need to uninstall. A partial solution is to (1) reinstall, keeping a record of files created, as in sudo python3 -m setup.py install --record installed_files.txt, and then (2) manually delete all the files listed, as in e.g. sudo rm -r /usr/local/lib/python3.7/dist-packages/tdc7201-0.1a2-py3.7.egg/ (for release 0.1a2 of module tdc7201). This does not 100% clean everything up however; even after you've done it, importing the (removed!) local library may succeed, and attempting to install the same version from a remote server may fail to do anything (because it thinks your (deleted!) local version is already up to date).

As suggested in previous answers, there is no symlinks that are getting created. How does '-e' option work? -> It just updates the file "PYTHONDIR/site-packages/easy-install.pth" with the project path specified in the 'command pip install -e'. So each time python search for a package it will check this directory as well => any changes to the files in this directory is instantly reflected.

The chief reason this is useful is because it frees your development from having to bother with all the repeated pip installs from local sources.

Imagine you have to fix a bug that requires an API change to three packages. You edit one of them, pip install it locally, test the other one on these new changes to the first, edit the second and likewise install it locally and go to edit the third package find that it has broke between the two edits and you need to go back and edit the first again, pip install etc etc. That’s a lot of development friction to have to bother with all these intermediate pip install steps.

What is more desirable in this type of multi package dependency package development is to instead install them all in editable mode. Then you can freely edit and make changes and test without ever needing to do the intermediate pip installs to get the packages working off your latest draft. You do the upfront editable install from source, get to hacking and testing and when everything is proven you can commit the changes to all packages and ship a parallel release for all packages.

Also another reason is it helps with code navigation in IDEs where you always want to navigate to dependency packages in a repo, especially a monorepo, where multiple packages live. You can enumerate in configurations many includes folders for the IDE to know of the many locations where a package can live or the interpreter that is selected in the IDE can have editable installs set to the clone of your repo and likewise be able to navigate to folders especially if those package folders do aliasing in their setup.py logic.

It's simple: --editable is for developers of the/that package. Normal users never need it!

With -e of --editable, the package is not installed as a typical package; with a copy in (e.g.) .../site-packages. Instead, only some administrative files are placed there. They direct Python to read the files in the original "source" location.

So, when those sources are edited, they are used (directly) -- without reinstalling them (again)! Very convenient for package developers; probably too risky for normal users -- as changing/deleting a file has immediate effect