Python packaging, pytest, click, VS.Code

I’m working on my blog engine. I want to package it nicely as a Python package. I also want to be able to run my code from a command line using the click library, and write and run tests using pytest, and be able to do all this with the debugger in VS.Code. I’ve never made my peace with Python packaging, for many many years now I’ve avoided it. Because it’s a mess.

Anyway I just tried again here in April 2024 and more or less succeeded in just an hour or so. This is where I would write clear docs with a github repo with the code but.. nah. I’m not confident it’s right and I don’t understand it well enough to present coherently. But I’ll write up what I learned.

I mostly followed the Packaging Python Projects tutorial which explains the creation of this sample project. Following that gets you a basic Python project which you can install with pip install -e. The key thing here is everything’s driven by the pyproject.toml file. If you are reading any docs that talk about setup.py they are old and outdated; the TOML is the new way.

Some subtleties in the pyproject.toml:

• My project has dependencies that I specified in the TOML. It depends on pytest and click.
• I had to pick a build backend; I’m using setuptools as the old non-fancy default.
• My project has console scripts. You can specify these in entry points or in a scripts section.

Getting click working was not hard. There’s docs for click + setuptools. Note these are in terms of the old setup.py style but translating to pyproject.toml isn’t too hard.

Here’s my source tree for a toy package called “whir”

• pyproject.toml in the root directory
• Python code in sys/whir/.
• src/whir/__init__.py that imports all the public functions, particularly the click commands.
• src/whir/lib.py is simple library code that other stuff imports.
• src/whir/main.py contains a main_cli() function that’s a click command. __init__.py imports it so it can be used as a script in pyproject.toml. It also imports the lib code via import whir.lib, not a relative import.
• tests/test_lib is a pytest function that tests the lib code. It imports it with from whir import lib.

Getting pytest working was not hard either. The default setup worked just fine from a shell. Getting it to work inside VS.Code’s extension for it required adding something to pyproject.toml to get pytest to use importlib, as described here. But now I can just press a button the IDE GUI and all my tests run with a nice display. Yay!

Getting the VS.Code debugger to work with click was tricky. AFAICT there’s no easy way to get VS.Code to invoke a click command. But there’s a hack where you call the click command with string command line arguments inside a if __name__ == '__main__'; then VS.Code can be used to just debug the file and it will work.

Edit: thanks to Claude I’ve now found two more ways to invoke a click command from a function. The intended way to just invoke it is to use main_cli().callback(args): the click decorator adds a callback function to the command function you can use. There’s also a whole library for testing invocation of Click commands, CliRunner, that will mimic a command line call. It sets up a special environment that captures stdout and stderr, can mock up a filesystem, etc. It’s for using in a unit test framework.

Ranting reflection

It all worked pretty straightforwardly! But I have no idea at all how it works. If something goes wrong I won’t understand what is happening or how to fi x it. And the tools are creating little files and shims that get between me and my code. It’s always like this with packaging and build systems and I hate it.