Import Your Own Python Code Without Pythonpath Tricks
With a single one-liner and proper code structure
“I can’t even import my own scripts!”
“But it worked on my laptop!”
Yes, the god of Suffering has many faces. The last one is my favorite.
Let me explain the issue.
Your colleague has given you these wonderful, brand-new Python scripts to end all the injustices in the world. You are ready to go, eager to run them, but you can’t import any of their functions and classes.
Not only you can’t import them from one script to another, but you can’t import them from your running Python interpreter as well.
Or maybe you can, but only if Jupyter and Saturn are aligned. In that same folder. Or like a PyCharm. And nobody else can do the same.
So you (or your kind colleague for you) have chosen the patchy way.
The Way of Suffering.
You have modified your PYTHONPATH. Or similar. You were forced to do so. And it’s a mess.
The import chaos for your code
As you might have noticed, I’m talking about the chaos that Python imports can generate. Or, better, the chaos that messy configuration can generate.
Because imports have no faults on their own, poor little creatures (that’s not true, I do have some things to say on that… but not now).
The real issue is the way they are used or supposed to be used.
From Tricks to Ghosts
I’ve seen several hacks to avoid this import issue.
Just some special mentions here, to underline why they are bad in general and what risks they pose.
If you just want to know the final working solution, without knowing why it is better than these tricks, skip this part!
The Ghost of PYTHONPATH
The first one is the quickest: altering the PYTHONPATH environment variable.
Basically, it works like the system path variable. It tells Python where to search for additional scripts and modules. You have just to add the path of your containing folder in its list of values, and you are ready to go.
Nice trick.
But then you have to share your code, even with just your collaborators, or you have to move it onto other machines for some reason. And nothing works anymore.
Every single time, you have to set the PYTHONPATH manually. You have to write this warning everywhere, in caps lock, within your README.
This Ghost will haunt you forever.
Furthermore, you can’t create a generally-working script to avoid this, because you don’t know where your other colleagues could put this code, so you don’t know the absolute path to specify.
But ok, maybe it’s my DevOps side to talk, or my what’s-this-please-automate-everything paranoia. I admit it.
There’s another and worse issue, too. Maybe you exported that PYTHONPATH definition in your config files because you have no intention to set it every time you open a shell or IDE.
However, now you are working with several, distinct projects and the definitions start to conflict. There is no separation.
Ouch.
I’m not saying it’s totally wrong (or I am? — cough).
It surely works in simple cases, but isn’t there another solution working in general? Maybe something that could be automated and could run everywhere?
Besides, did you notice the “additional” definition above?
“It tells Python where to search for additional scripts and modules.”
It comes from Python documentation.How could my code be an addition? It is the main code to run!
Not to mention that, on that very page, there is also a warning about different behaviors of Python’s alternative implementations — altough I never noticed distinctions in this particular case.
The Master-Ghost of sys.path
“But I can automate this configuration! I can modify sys.path in my scripts!”
But… Why?
Besides having almost the same issues as the previous Ghost, this solution is mixing application code with system configuration code.
When something smells bad, it’s often rotten. This case isn’t different. Mixing scopes is always a bad idea.
Furthermore, you are playing with environment variables at runtime. Why waiting so long to set up your code? Setup is the first thing to do, not the last one! It makes you aware of errors as soon as possible, to exclude the following components from the list of possible causes.
Get Federico Pugliese’s stories in your inbox
Join Medium for free to get updates from this writer.
And what happens when you have to fork your process, or when you use a library that does it without warning you? Is it inheriting these environment variables? Do you have to worry about it every single time?
A Solution in Plain Sight
If you pay close attention, you will see it.
See the way you have to import functions and classes from other libraries, I mean. Besides the fact they work seamlessly, they work everywhere, with no PYTHONPATH changes.
So why would you use a different mechanism?
They work because these packages are put under the site-packages folder, which is inspected by Python by default.
Now the point is: who the hell is moving them there?
The answer is simple. It is your pip install command (or equivalent).
Yes, I’ll focus on the pip case where you also have a virtual environment. But of course it is the same for conda envs.
This is not the right place to list the advantages of having a virtual environment, right?
Installing your own code
So the idea is to install your code along with your other packages. This means that your code has to be structured as a package as well.
Do not despair, it is very easy! Just keep in mind two things.
(1) You just have to collect your code in at least a folder with an empty __init__.py file.
These files are essential to mark their folders as Python modules. In this way, they are recognized for the installation (and therefore imports). They are mandatory.
Suppose your package is justice:
justice\
...
innerfolder\
__init__.py
some_script.py
...
...
__init__.py
one_of_your_scripts.py
setup.pySo, an __init__.py for each folder.
(2) There is also another essential piece: the setup.py file.
The setup.py file is searched and used by the pip install <package> command by default. It tells several things: metadata about your package, where to find scripts and (optional but suggested) what are the requirements.
from setuptools import setup, find_packages# List of requirements
requirements = [] # This could be retrieved from requirements.txt# Package (minimal) configuration
setup(
name="justice",
version="1.0.0",
description="End world injustices",
packages=find_packages(), # __init__.py folders search
install_requires=requirements
)
Writing the requirements there makes your pip install <package> command install your package dependencies as well. Yes, just with a single command! No need to run pip install -r requirements.txt too. Double benefit!
There is a lot to say here, also about distributions (source, wheel). Maybe in the next article!
So everything you need to do now is to install your code by using pip install <path_to_setup.py_folder>, or in this case: pip install .
Wow. Is everything alright? Just with that?
Yeah, almost.
Because, actually, you can’t do that for your code. This installation would copy your scripts into the site-packages folder, and that’s it. What if your code changes? It is in development, so this is very likely. You would have to install it every time. Not handy at all.
That’s why you should use the single, one-liner savior:
pip install -e .The development mode install
The -e option tells pip to install it in “development mode”. Basically, it puts just a reference to your package in the site-packages folder. So you can change it without worrying about installation anymore!
You just have to do this once and for all, even if you change your code.
If you installed your package in the virtual environment, you can just import your code from anywhere, and from any shell, provided you activated that virtualenv.
For instance:
from justice import one_of_your_scripts
one_of_your_scripts.a_function()And most importantly, you can run that exact installation command anywhere. Because it works for your colleague, too. You do not have to worry about the path of the installation. That means easy automation.
Quick, clean, general.
Everything a real solution should be.
