Knowledge Bits - Python

Python Asynchronous Programming Fundamentals

2022-05-29T00:00:00-04:00

Python introduced asynchronous programming capabilities in version 3.4 in 2014, with further notable improvements in almost every minor version since. However, to many Python programmers, this area of the language remains esoteric, misunderstood, and underutilized. This article aims to elucidate the fundamental concepts of asynchronous programming as part of the first step towards mastery.

There won’t be many code samples in this article, but reading it should make it easier to grok the asyncio documentation and figure out how to piece things together.

Contents

High-Level Overview
Definitions
Syntax
Running More than One Coroutine at Once
Exception Handling
Example Code
Asynchronous Programming vs. Threads
Async Version History
Alternative Async Implementations

High-Level Overview

Asynchronous programming provides a way to interleave execution of multiple functions (coroutines) at once; at any given point, only one of the functions is actually doing operations, while the others are waiting for things like blocking I/O to complete (or, if the I/O has already completed, they’re waiting for the current coroutine to be suspended so that they get a chance to resume).

Note

In the Python standard library and all third-party async libraries that I am aware of, the kinds of low-level operations that one can ultimately suspend execution waiting for are almost exclusively I/O-related. If you’re seeking to add concurrency to CPU-bound code (e.g., anything involving number crunching), you are likely to be better off using multiprocessing instead.

Specifically, whenever execution of a currently-running coroutine reaches an await expression, the coroutine may be suspended, and another previously-suspended coroutine may resume execution if what it was suspended on has since returned a value. Suspension can also happen when an async for block requests the next value from an asynchronous iterator or when an async with block is entered or exited, as these operations use await under the hood.

Note that, although multiple coroutines can be processed at once, effectively finishing in any order, operations within a single coroutine continue to be executed in the order they’re written. For example, given the following code:

async def f():
    ...

async def g():
    ...

async def amain():
    await f()
    await g()

asyncio.run(amain())

When await f() is reached, amain() will be suspended until f() is finished, and only after that will execution proceed to the next line, starting coroutine g(). If you want to execute f() and g() concurrently, you need to use asyncio.create_task(), asyncio.gather(), or similar. See this article for more details.

In general, coroutines can only be called or scheduled by other coroutines. To run a “top-level” coroutine from inside synchronous code (i.e., either inside a non-coroutine function or at module level), the simplest & preferred way is to use the asyncio.run() function introduced in Python 3.7. Code meant to run on older Pythons must use the lower-level loop.run_until_complete() instead, along with other loop methods for cleanup.

Definitions

Asynchronous programming features the execution of multiple tasks concurrently, with one task being run while waiting for others to complete. Asynchronous programming in Python is an example of cooperative multitasking [wiki], as it requires the running coroutines to cooperate and yield control on their own; if the current coroutine goes for a long time without calling await, execution will remain on that coroutine all the while, and all other coroutines in the thread will remain in a suspended state. This is contrast to the preemptive multitasking [wiki] of multithreaded and multiprocess programs, where the Python interpreter or OS scheduler decides on its own when to switch between running contexts, with the points at which switches can occur being difficult or impossible to predict in general.

A coroutine function is a function defined with async def instead of just def. (In the context of asynchronous programming, a function defined with just def is called a synchronous function.) Only coroutine functions can contain await, async for, and async with constructs, and, as of Python 3.10, they cannot contain yield from constructs.

Note that just calling a coroutine function does not cause it to start running; you need to either schedule it for concurrent execution with asyncio.create_task() or similar or else await on it directly.

A coroutine object is the result of calling a coroutine function. This is not the value returned (or raised) by the function; rather, it is a pending computation that can be suspended & resumed at any point that the coroutine function uses await, async for, or async with. The actual return value or exception is obtained either by awaiting on the coroutine object (possibly wrapped in a task and/or something like asyncio.gather()) from within another coroutine function or by running it as a “top-level” entry point using asyncio.run() within synchronous code.

Asynchronous generators — coroutine functions that use yield — are a bit of an exception. You do not await the result of the function; instead, you iterate through it using either async for ... in ...: or await anext(...).
Confusingly, both coroutine functions and coroutine objects can be referred to as just “coroutines.”

The actual scheduling of coroutine execution is managed by an event loop. An event loop is created by asyncio.run() or asyncio.new_event_loop(), handed one or more coroutines and/or synchronous callbacks, and then set off to run either forever or until completion of a “top-level” coroutine. It’s the event loop’s job to execute the current coroutine until it suspends on an await, after which it looks to see if any suspended coroutines are now done with their suspension and either picks one to resume or, if none are ready, waits until one is.

An awaitable is any value that the await keyword can be applied to; this includes coroutine objects, futures, future-likes, and tasks (see below). Awaiting on an awaitable causes the current coroutine to be suspended until the awaitable is ready to provide a value or raise an exception.

A future (class asyncio.Future) is a low-level container for the result of a computation (a value or exception) that starts out empty and is assigned a value or exception later. Awaiting on a future will suspend the current coroutine until something else either stores a result in the future or cancels it.

You may already be familiar with futures in the form of the Future class from the concurrent.futures module, which provides access to the results of operations evaluated in other threads or processes. The asyncio.Future class is similar in spirit, but has a different API.

A future-like is an object with an __await__() method, which must return an iterator. Awaiting on a future-like causes the current coroutine to be suspended until the iterator is exhausted, at which point the value attribute of the terminating StopIteration exception is returned as the result of the await expression.

A task (class asyncio.Task) represents a running coroutine. Creating a task from a coroutine object with asyncio.create_task() will cause the coroutine to be scheduled for execution concurrently with other running coroutines. The task instance can later be awaited on to suspend the current coroutine until the wrapped coroutine finishes executing, returning its result.

A running task can be cancelled by calling its Task.cancel() method. This will cause the underlying coroutine to receive an asyncio.CancelledError exception the next time it awaits, likely putting an end to the task’s execution.

Syntax

Asynchronous programming in Python takes place inside coroutines, functions defined using async def instead of just def. Within a coroutine, the await keyword can be applied to any awaitable expression (such as a call to another coroutine) to suspend execution of the coroutine until the awaitable has a value or exception ready, at which point the coroutine is resumed and the await expression returns that value or raises that exception.

Here’s a basic example you’ve seen in all the tutorials:

import asyncio

async def waiter():
    print("Before sleep")
    await asyncio.sleep(5)
    print("After sleep")

asyncio.run(waiter())

This code prints a message, waits for a five-second sleep to elapse, and then prints another message. As written, it’s rather pointless; we’re only running one coroutine at once, so there’s no advantage over using a synchronous function with time.sleep(). Here’s something slightly more involved:

import asyncio

async def operate(time, result):
    print(f"Spending {time} seconds doing operations ...")
    await asyncio.sleep(time)
    print(f"Operations done after {time} seconds!")
    return result

async def amain():
    x, y = await asyncio.gather(operate(5, 42), operate(2, 23))
    print(f"Got {x=}, {y=}")
    assert x == 42
    assert y == 23

asyncio.run(amain())

This code mocks spending time on two blocking operations in parallel. If you run the script (Python 3.8+ required) and time it, you’ll see that it only takes about 5 seconds in total, and the 2-second task completes three seconds before the 5-second one. After both tasks are done, the final “Got x=42, y=23” message is printed.

Besides await, there are two other syntactical constructs specific to coroutines: async for ... in ...: (for iterating over asynchronous iterables) and async with ...: (for entering & exiting asynchronous context managers). These work the same way as their non-async counterparts, except that the iterables and context managers in question need to support asynchronous usage; for example, an async for cannot iterate over a list, and an async with cannot operate on an ordinary filehandle. Similarly, a regular for cannot be applied to an asynchronous iterator, and a regular with cannot be applied to, say, asyncio.Lock.

Speaking of asynchronous iteration, this works pretty much how you’d expect: by using yield inside a coroutine, it becomes an asynchronous generator that can be iterated over with async for or await anext(...). Note that, in contrast to non-generator coroutines, you do not apply await to an asynchronous generator. For example, given this function:

async def aiterator():
    for i in range(5):
        await asyncio.sleep(i)
        yield i

you use it like this:

async for x in aiterator():
    print(x)

No await anywhere in the async for loop.

Note that there is no way to get a value out of a coroutine without awaiting on it (either directly or via something like asyncio.gather()); if a coroutine is never awaited and never converted into a task, asyncio will complain when it is garbage-collected. Moreover, await (and async for and async with) cannot be used outside of a coroutine; in order to start the awaiting on a “top-level” coroutine, you need to use asyncio.run().

Note that the body of a coroutine isn’t required to contain any awaits or similar, though if it doesn’t, there often isn’t much point in making it a coroutine in the first place. An exception is the __aenter__() special method of asynchronous context managers; usually, the body will just be return self, but it’s still required to define the method with async def.

Type Annotations

Adding type annotations to asynchronous code works the same way as for synchronous code. If an asynchronous func() take an integer x and returns a string, you write its annotated signature as async def func(x: int) -> str. However, if you pass around an unawaited coroutine object (not always the best idea), you annotate it as Awaitable[T], where T is the return type of the coroutine.

Async callables do not have their own type; they are instead annotated as Callable[..., Awaitable[T]], where T is the return type of the coroutine function.

Asynchronous iterators, iterables, and context managers, though, do get their own types: AsyncIterator, AsyncIterable, and typing.AsyncContextManager/contextlib.AbstractAsyncContextManager.

Special Methods

__aiter__() and __anext__()

These methods are used to implement asynchronous iterator & iterable classes as an alternative to writing asynchronous generator functions, analogously to how a class can be defined with __iter__() and __next__() methods to implement a synchronous iterator as an alternative to writing a generator function.

__aiter__() must be a synchronous function that returns an object with __aiter__() and __anext__() methods. __anext__() must be a coroutine that either returns a value or raises a StopAsyncIteration exception.

__aenter__() and __aexit__()

These methods are used to implement asynchronous context managers. They are defined the same way as the __enter__() and __exit__() methods of synchronous context managers, except that the asynchronous versions must be coroutines.

__await__()

This method is used to create a future-like class that can be awaited directly. There is generally very little need to implement this, but we include it here for completeness.

__await__() must be a synchronous function that returns an iterator. This iterator will be advanced each time the event loop checks to see if the future-like is ready to return a value. If the future-like is not ready, the iterator must yield a special value (see below). If the future-like is ready, it must either return a result or raise an exception; this result or exception will then be the result of the await expression acting on the future-like.

The values that the iterator yields depend on what async implementation the code is running under (See “Alternative Async Implementations” below). When using the Python standard library’s asyncio, you generally want to yield None. When using trio, you need to yield an instance of one of several private classes internal to trio. When using curio, you need to yield a ("trap_name", *trap_args) tuple instructing the kernel to invoke a special “trap” function; yielding ("trap_sleep", 0) instructs the kernel to do nothing special.

For example, if you want to implement your own Future class for use with asyncio, you might start out writing it like so:

class MyFuture:
    def __init__(self):
        self.value = None
        self.is_set = False

    def set_value(self, value):
        self.value = value
        self.is_set = True

    def __await__(self):
        while not self.is_set:
            yield
        return self.value

Historical Note: Generator-Based Coroutines

When asyncio was first introduced in Python 3.4, the async and await keywords were not yet present. Instead, coroutine functions were created by applying the @asyncio.coroutine decorator to a normal generator function, and awaiting was done using yield from. There were no asynchronous iterators or asynchronous context managers in 3.4, either. Even after async and await were introduced in Python 3.5, the older generator-based coroutines could not use them.

This style of writing coroutines was deprecated in Python 3.8 and removed entirely in Python 3.11.

Running More than One Coroutine at Once

Now for the part you’ve been waiting for, and the part that makes asynchronous programming worth it: actually running multiple functions concurrently.

The simplest way to start running another coroutine concurrently is to pass a coroutine object to asyncio.create_task(); this schedules the coroutine for execution, but it won’t actually start running until the next time the current coroutine calls await (and maybe not even then). asyncio.create_task() returns an asyncio.Task object that can be used to query the state of the coroutine or cancel it; awaiting on the task object will cause the current coroutine to be suspended until the task is complete, at which point the return value of the task’s underlying coroutine is returned.

If you create multiple tasks and then await on them one by one, a given await will not return a result until the task in question is done; if task B finishes running while you’re still awaiting on task A, the coroutine doing the awaiting will continue to be suspended until A is done, and when it then later awaits on B, it will get back B’s return value immediately, because B is already done. For example, the following code:

import asyncio
from time import strftime

def hms():
    return strftime("%H:%M:%S")

async def operate(time, result):
    print(f"{hms()}: Spending {time} seconds doing operations ...")
    await asyncio.sleep(time)
    print(f"{hms()}: Operations done after {time} seconds!")
    return result

async def amain():
    task1 = asyncio.create_task(operate(5, 42))
    task2 = asyncio.create_task(operate(2, 23))
    r1 = await task1
    print(f"{hms()}: task1 returned {r1}")
    r2 = await task2
    print(f"{hms()}: task2 returned {r2}")

asyncio.run(amain())

outputs something like the following:

17:12:56: Spending 5 seconds doing operations ...
17:12:56: Spending 2 seconds doing operations ...
17:12:58: Operations done after 2 seconds!
17:13:01: Operations done after 5 seconds!
17:13:01: task1 returned 42
17:13:01: task2 returned 23

If you need to await on multiple coroutines but don’t care about the exact order in which they finish, you can use asyncio.gather(), asyncio.as_completed(), or asyncio.wait() to await on them together; this article gives a good overview and explanation of the differences between the functions.

You don’t have to await on a task if you don’t need its return value or don’t need to be assured that it ever finishes, but if such a “background task” raises an uncaught exception, asyncio will complain. One way to address this is to attach a synchronous callback function to the task with Task.add_done_callback() that retrieves any uncaught exceptions with Task.exception(), like so:

import asyncio
from time import strftime

def hms():
    return strftime("%H:%M:%S")

async def bg_task():
    print(f"{hms()}: In the background")
    raise RuntimeError("Ouch")

def done_callback(task):
    try:
        if e := task.exception():
            print(f"{hms()}: Task <{task.get_name()}> raised an error: {e}")
        else:
            print(f"{hms()}: Task <{task.get_name()}> finished successfully")
    except asyncio.CancelledError:
        print(f"{hms()}: Task <{task.get_name()}> was cancelled!")

async def fg_task():
    task = asyncio.create_task(bg_task(), name="bg_task")
    task.add_done_callback(done_callback)
    print(f"{hms()}: Now we sleep and let bg_task do its thing")
    await asyncio.sleep(2)
    print(f"{hms()}: I'm awake!")

asyncio.run(fg_task())

The output from the above will look like:

17:16:33: Now we sleep and let bg_task do its thing
17:16:33: In the background
17:16:33: Task <bg_task> raised an error: Ouch
17:16:35: I'm awake!

Exception Handling

Whenever an exception occurs inside a coroutine, it propagates upwards to whatever’s awaiting on it; if unhandled, it will propagate all the way out through the asyncio.run() call, at which point all still-running tasks are cancelled. If there is no chain of awaits leading to the “top-level” coroutine (say, because you did asyncio.create_task() and then didn’t await on the result, letting it run in the background), asyncio will end up complaining when the coroutine is eventually garbage-collected. See the passage above on using Task.add_done_callback() to handle such errors.

For the specific case of KeyboardInterrupt, the exception is raised in whatever coroutine the main thread is currently running at the time.

Example Code

This gist provides an example of using asynchronous programming in Python to download assets for one or more releases of a GitHub repository in parallel. Try it out with an invocation like:

python download-assets.py --download-dir jq stedolan/jq jq-1.5 jq-1.6

The script requires Python 3.8 or higher and the ghrepo and httpx packages on PyPI to run.

Asynchronous Programming vs. Threads

Asynchronous programming does not use threads; by default, all coroutines and their operations are run in whichever thread called asyncio.run(). The exception is when asyncio.to_thread() or loop.run_in_executor() is used to run a synchronous function in a separate thread (or, with the latter function, even a separate process), returning an object that can be awaited on to receive the function’s result.

If multiple threads call asyncio.run() separately, each thread will get its own event loop and collection of coroutines.

Note that each thread in a Python process has at most one event loop at a time, and an event loop can only belong to one thread. An important consequence of this is that, if you have a synchronous function foo() that calls asyncio.run() on some coroutine, then foo() cannot be called by another coroutine, because that would lead to two event loops in the same thread, which doesn’t work.

Compared to threads, asynchronous programming provides the following advantages:

In asynchronous programming, the coroutine being executed can only change when the current coroutine uses await or similar. This allows the programmer to be assured that, between awaits in the same coroutine, operations will not be interfered with and data will not be modified by other coroutines.

When using threads, on the other hand, the running thread [1] can change at almost any point as chosen by the interpreter, which necessitates careful programming and copious use of locks in order to ensure that variables are not modified by one thread “behind the back of” another thread that’s also using them.
If you’ve done serious work with threads, you’ve likely encountered the fact that you cannot “kill” a thread in the middle of its execution unless the “killable” thread is deliberately programmed to allow for this by, say, regularly checking some flag and exiting if it’s true. Asynchronous programming, on the other hand, makes it possible to cancel a running coroutine via the asyncio.Task.cancel() method; once a coroutine is cancelled, the next time the event loop checks on it while it’s suspended on an await or similar, the coroutine will be resumed, but instead of receiving the value it was awaiting for, an asyncio.CancelledError will be raised at the await expression, likely putting an end to the coroutine’s execution.

[1]	Recall that, due to Python’s global interpreter lock (GIL), regardless of how many threads a Python program uses or how many cores your machine has, only one thread will be executing Python bytecode at any moment.

Async Version History

Here follows a list of the notable developments & changes in asynchronous programming across Python versions.

Python 3.4

The asyncio module is added, implementing PEP 3156. This enables the creation of coroutines via the @asyncio.coroutine decorator; within a coroutine, awaiting is performed with yield from. (Support for creating coroutines in this way would later be deprecated in Python 3.8 and removed in Python 3.11.)

Most of the functionality added in this version is now categorized as the “low level” part of asyncio.

Python 3.5

PEP 492 implemented:

It is now possible to define coroutines via async def and await with await.
- In Python 3.5, yield cannot be used inside the body of an async def coroutine function.
Asynchronous iteration with async for is now possible.
Asynchronous context managers are now supported via async with.
__await__(), __aiter__(), __anext__(), __aenter__(), and __aexit__() special methods added
Originally, __aiter__() methods were expected to be coroutines (or anything else returning an awaitable) resolving to asynchronous iterators. This was changed in 3.5.2 to have __aiter__() instead return an asynchronous iterator directly. Returning an awaitable from __aiter__() produces a PendingDeprecationWarning starting in 3.5.2, a DeprecationWarning starting in 3.6, and a RuntimeError starting in 3.7.

Python 3.6

yield can now be used in the body of an async def coroutine function, thereby enabling asynchronous generators (PEP 525). (yield from remains prohibited, though.)
async for can now be used in list, set, & dict comprehensions and in generator expressions
await expressions can now be used in any comprehension
Using async or await as an identifier now generates a DeprecationWarning

Python 3.7

async and await are now reserved keywords
asyncio.run() added
asyncio.create_task() added

Python 3.8

Running python -m asyncio now starts an async REPL
@asyncio.coroutine() is now deprecated
Passing a loop parameter is now deprecated for most of asyncio’s high-level API
asyncio.CancelledError now inherits directly from BaseException instead of Exception

Python 3.9

asyncio.to_thread() added

Python 3.10

aiter() and anext() functions added
The loop parameter (deprecated in Python 3.8) is now removed from most of asyncio’s high-level API

Python 3.11

asyncio.TaskGroup added
@asyncio.coroutine (deprecated in Python 3.8) is now removed

Alternative Async Implementations

While all the code we’ve shown so far uses the Python standard library’s asyncio module, it’s not required to use this to work with coroutines. Alternative async library implementations exist that define their own event loops and primitive operations. The more notable implementations include:

trio seeks to enable structured concurrency [wiki] in asynchronous code. In trio, a collection of tasks are run concurrently by grouping them together under a nursery (also known as a task group); if one of the tasks in a nursery raises an error, all the other tasks in the same nursery are automatically cancelled.
curio is an asyncio alternative featuring a more streamlined API and intended to be easier to reason about

In general, different async implementations are incompatible, and features from different implementations cannot be used in the same code unless you make careful use of whatever compatibility facilities they may provide.

In fact, just being able to use the same code unmodified regardless of whether using implementation A or implementation B is tricky, as all implementations use different primitives. Libraries to help you with that include:

sniffio can be used to detect which async library is in use
anyio provides a common API (based on trio) that can be used to run the same code under both asyncio and trio (and previously curio, until it & anyio parted ways in anyio 3.0)

Creating Multi-Value Enums in Python

2022-01-14T00:00:00-05:00

Say you’re creating an enum type in Python and you want each enum member to have one or more additional constant attributes alongside value, but these attributes can’t be easily derived from value (and thus don’t make for good @propertys). The Python docs provide an example involving planets similar to what you want, but on closer inspection, you see it won’t do — the example code produces an enum whose members’ values are tuples of the other attributes, whereas you want or need value to be something else. Is there a way to define enum members with a combination of a “main” value and a set of “extra” values?

Yes. Yes, there is.

For a specific example, let’s say you want to define an enum for cardinal and ordinal directions where the members’ values are human-readable names used in some data source you have to parse and each member should additionally have x and y attributes giving the deltas to add to a coordinate’s components in order to move a step in the respective direction. That is, you want the following to evaluate as shown:

>>> Direction("north-west")
<Direction.NORTH_WEST: 'north-west'>
>>> Direction.NORTH_WEST.value
'north-west'
>>> Direction.NORTH_WEST.x
-1
>>> Direction.NORTH_WEST.y
1

The solution is to define each enum member with a tuple of the value and extra attributes, and then write a __new__ method that assigns the elements of the tuple (which are passed as arguments to __new__) appropriately, like so:

from enum import Enum

class Direction(Enum):
    NORTH      = ("north", 0, 1)
    NORTH_EAST = ("north-east", 1, 1)
    EAST       = ("east", 1, 0)
    SOUTH_EAST = ("south-east", 1, -1)
    SOUTH      = ("south", 0, -1)
    SOUTH_WEST = ("south-west", -1, -1)
    WEST       = ("west", -1, 0)
    NORTH_WEST = ("north-west", -1, 1)

    def __new__(cls, value, x, y):
        obj = object.__new__(cls)
        obj._value_ = value
        obj.x = x
        obj.y = y
        return obj

# Check that `value` and other attributes are what they're supposed to be:
for d in Direction:
    print(f"{d!r} {d.name=} {d.value=} {d.x=} {d.y=}")

# Member access by value uses only the "main" value:
print(f"{Direction('north')=}")

Output from the above script:

<Direction.NORTH: 'north'> d.name='NORTH' d.value='north' d.x=0 d.y=1
<Direction.NORTH_EAST: 'north-east'> d.name='NORTH_EAST' d.value='north-east' d.x=1 d.y=1
<Direction.EAST: 'east'> d.name='EAST' d.value='east' d.x=1 d.y=0
<Direction.SOUTH_EAST: 'south-east'> d.name='SOUTH_EAST' d.value='south-east' d.x=1 d.y=-1
<Direction.SOUTH: 'south'> d.name='SOUTH' d.value='south' d.x=0 d.y=-1
<Direction.SOUTH_WEST: 'south-west'> d.name='SOUTH_WEST' d.value='south-west' d.x=-1 d.y=-1
<Direction.WEST: 'west'> d.name='WEST' d.value='west' d.x=-1 d.y=0
<Direction.NORTH_WEST: 'north-west'> d.name='NORTH_WEST' d.value='north-west' d.x=-1 d.y=1
Direction('north')=<Direction.NORTH: 'north'>

Note

If you’re inheriting a concrete non-Enum type in addition to Enum (whether explicitly or via IntEnum or StrEnum), you may need to replace the obj = object.__new__(cls) line in the __new__ method with an invocation of the concrete type’s __new__. For example, if inheriting from (str, Enum) or Python 3.11’s StrEnum, this line should become obj = str.__new__(cls, value).

Now, if you’re type-checking with mypy (and you really should be), you’ll likely hit some errors on the above code, even after adding annotations to __new__. Specifically, as of mypy 0.931, there are two outstanding mypy issues that affect the above snippet:

Issue #1021 means that mypy ignores attribute assignments in __new__, so it won’t recognize x and y as attributes of Direction, leading to print(f"… {d.x=} {d.y=}") being seen as erroneous. There are two ways to make mypy aware of the extra-value attributes: either add annotations for them in the main body of the class, like so:

class Direction(Enum):
    NORTH      = ("north", 0, 1)
    # [snip]
    NORTH_WEST = ("north-west", -1, 1)

    # Add these two lines:
    x: int
    y: int

    def __new__(cls, value: str, x: int, y: int) -> Direction:
        ...

… or else move the extra-value assignments to an __init__ method, like so:

class Direction(Enum):
    NORTH      = ("north", 0, 1)
    # [snip]
    NORTH_WEST = ("north-west", -1, 1)

    def __new__(cls, value: str, x: int, y: int) -> Direction:
        obj = object.__new__(cls)
        obj._value_ = value
        return obj

    def __init__(self, value: str, x: int, y: int) -> None:
        self.x = x
        self.y = y

The second mypy problem is issue #10573: mypy is not aware of the Enum class’s trickery with its constructor, so when it sees Direction('north') on the last line, having already seen a Direction.__new__ that takes three arguments, it thinks there are two arguments missing from the Direction() call. Unfortunately, at time of writing, there does not seem to be a way around this other than just slapping a “# type: ignore[call-arg]” comment on each call to a multi-value enum type.

Skipping Pytest Tests Unless an Option is Given

2021-12-05T00:00:00-05:00

When testing Python code with pytest, you may occasionally write tests that you only want to run under special circumstances, such as long-running tests that should only be run under continuous integration and not when invoking pytest locally. The naïve way to accomplish this is to decorate the tests in question with a pytest mark like @pytest.mark.slow and then specify -m "not slow" when running pytest locally, but then you have to remember to pass this option every time, and if you hardcode it into your tox.ini or pytest configuration, you’ll need something else to remove it when testing under CI. Fortunately, there are better ways to make pytest skip tests by default.

In this article, we will add a “--run-slow” option to pytest and configure selected “slow” tests to be skipped unless this option is given on the command line.

Option 1: Use a Hook to Attach a `skip` Marker to Marked Tests

One way to disable selected tests by default is to give them all some mark and then use the pytest_collection_modifyitems hook to add an additional pytest.mark.skip mark if a certain command-line option was not given. We do this by adding the following to our conftest.py file:

import pytest

def pytest_addoption(parser):
    parser.addoption(
        "--run-slow",
        action="store_true",
        default=False,
        help="Run slow tests",
    )

def pytest_collection_modifyitems(config, items):
    if not config.getoption("--run-slow"):
        skipper = pytest.mark.skip(reason="Only run when --run-slow is given")
        for item in items:
            if "slow" in item.keywords:
                item.add_marker(skipper)

and then decorate all of our slow tests with @pytest.mark.slow, like so:

import pytest

@pytest.mark.slow
def test_something_slow():
    ...

Now, when we run pytest, the marked tests will be skipped unless the --run-slow option is passed on the command line.

Option 2: Use `@pytest.mark.skipif`

Did you know? The condition passed to @pytest.mark.skipif doesn’t have to be a Python boolean expression; it can instead be a condition string containing a Python expression, and condition strings are evaluated in a namespace that includes the pytest config object. This lets us write skipif decorators that skip tests based on whether or not certain command-line options were given.

We start out by defining a command-line option in conftest.py that will be used to tell pytest to run otherwise-skipped tests:

def pytest_addoption(parser):
    parser.addoption(
        "--run-slow",
        action="store_true",
        default=False,
        help="Run slow tests",
    )

We then decorate the tests we want to skip by default like so:

import pytest

@pytest.mark.skipif(
    "not config.getoption('--run-slow')",
    reason="Only run when --run-slow is given",
)
def test_something_slow():
    ...

If there are multiple tests to apply this condition to, we can assign the skipif decorator to a variable that is then used to decorate each one, like so:

import pytest

slow_test = pytest.mark.skipif(
    "not config.getoption('--run-slow')",
    reason="Only run when --run-slow is given",
)

@slow_test
def test_something_slow():
    ...

@slow_test
def test_something_very_slow():
    ...

Warning

You may see some old guides on the internet that instead write the skipif decorator with an actual boolean condition defined in terms of pytest.config, e.g.:

import pytest

# Outdated!
@pytest.mark.skipif(
    not pytest.config.getoption("--run-slow"),
    reason="Only run when --run-slow is given",
)
def test_something_slow():
    ...

However, this no longer works; the pytest.config global variable was removed from pytest in version 5.1.0.

Option 3: Use a Pre-Existing Plugin

As is the case for many, many things in Python, other people have already done the job of generalizing the above and publishing it on PyPI. I am aware of two pytest plugin projects for skipping tests unless a command-line option is given: pytest-explicit and pytest-optional-tests. I have not used either, but if they work as advertised, they should prove helpful.

Notable Features Introduced in Each Python Version

2021-07-11T00:00:00-04:00

The following is a list of all of the notable or interesting (in my biased opinion) features introduced in each major release of Python, starting with Python 3.0.

Contents

Python 3.0 (2008-12-03), Relative to Python 2.6
Python 3.1 (2009-06-26)
Python 3.2 (2011-02-20)
Python 3.3 (2012-09-29)
Python 3.4 (2014-03-17)
Python 3.5 (2015-09-13)
Python 3.6 (2016-12-23)
Python 3.7 (2018-06-27)
Python 3.8 (2019-10-14)
Python 3.9 (2020-10-05)
Python 3.10 (2021-10-04)
Python 3.11 (2022-10-03)
Python 3.12 (2023-10-02)

Python 3.0 (2008-12-03), Relative to Python 2.6

Release notes: [link]

print statement replaced with a print() function
exec keyword replaced with an exec() function, which does not take a stream argument
dict.keys(), dict.items() and dict.values() now return “views” instead of lists
dict.iterkeys(), dict.iteritems(), and dict.itervalues() removed
map() now returns an iterator, stops when the shortest iterable is exhausted, and cannot take None as a function
filter() now returns an iterator
Python 2’s xrange() is Python 3’s range(); Python 2’s range() is no more
Python 2’s raw_input() is Python 3’s input(); Python 2’s input() is no more
Python 2’s long is Python 3’s int
- The repr() of a (long) integer no longer includes a trailing L
Python 2’s unichr is Python 3’s chr
zip() now returns an iterator
Order comparison operators now raise TypeError when given operands of different types
sorted() and list.sort() no longer accept the cmp parameter
cmp() and __cmp__() removed
Dividing two integers with / now returns a float
sys module:
- stdin, stdout, and stderr are now text streams
- maxint, exitfunc(), exc_clear(), exc_type, exc_value, and exc_traceback removed
itertools module:
- imap(), ifilter(), and izip() removed
- ifilterfalse() renamed to filterfalse()
- izip_longest() renamed to zip_longest()
math module: ceil() and floor() now return integers
array module:
- array.read() and array.write() removed
- c typecode removed
operator module: sequenceIncludes() and isCallable() removed
random module: jumpahead() API removed
os module: tmpnam(), tempnam(), and tmpfile() removed
string module: letters, lowercase, and uppercase removed
__builtin__ module renamed to builtins
Octal literals now use the prefix 0o instead of just 0
Python 2’s str is Python 3’s bytes; Python 2’s unicode is Python 3’s str
All string literals are now text/Unicode strs
The u prefix for text string literals is no longer supported (later reintroduced in Python 3.3)
Mixing byte strings and text strings now produces a TypeError
\uXXXX and \UXXXXXXXX escapes in raw string literals are now interpreted literally instead of being converted to Unicode characters
- As a consequence of this, the ur string prefix is no longer supported
basestring removed
The repr() of a string now longer escapes non-ASCII characters
The default Python source encoding is now UTF-8
Non-ASCII letters are now allowed in identifiers
StringIO and cStringIO removed; use io.StringIO and io.BytesIO instead
Function arguments and return values can now be annotated
Functions now have an __annotations__ attribute
Named parameters after *args or a bare * in a function signature are now keyword-only
Keyword arguments are allowed after the list of base classes in a class definition
New keyword: nonlocal
Extended iterable unpacking: Statements like a, b, *rest = some_sequence are now supported
Dictionary comprehensions added: {k: v for k, v in stuff}
Set literals and set comprehensions added: {1, 2} and {x for x in stuff}
New builtin functions: ascii() and next()
intern() moved to sys.intern()
Builtins apply(), buffer(), callable() (later reintroduced in Python 3.2), coerce(), execfile(), and file removed
reduce() moved to functools.reduce()
reload() moved to imp.reload()
dict.has_key() removed
round()’s rounding strategy and return type changed
True, False, and None are now reserved words

The syntax for using a metaclass has changed from:

class C:
    __metaclass__ = M
    ...

to:

class C(metaclass=M):
    ...

Special method __prepare__ on metaclasses added
Module-global __metaclass__ variable no longer supported
List comprehensions of the form [... for var in item1, item2, ...] must now be written [... for var in (item1, item2, ...)] instead.
The loop control variables of list comprehensions are no longer leaked into the surrounding scope
... (the ellipsis) can now be used as an expression anywhere, and it can no longer be spelled . . .
Tuple unpacking in function parameters no longer supported
Calling repr() via backticks no longer supported
<> removed
Trailing l or L on integer literals no longer supported
from module import * is no longer allowed inside functions
All module imports where the module name does not start with a period are now treated as absolute imports
Classic classes removed; all classes are now new-style
The following modules are removed: audiodev, Bastion/rexec, bsddb185, bsddb3, Canvas, cfmfile, cl, commands, compiler, dircache, dl, fpformat, gopherlib, htmllib, ihooks, imageop, imputil, linuxaudiodev, md5, mhlib, mimetools, MimeWriter, mimify, multifile, mutex, new, popen2, posixfile, pure, rfc822, sets, sgmllib, sha, sre, statvfs, stringold, sunaudio, sv, test.testall, thread, timing, toaiff, user, UserDict (moved to collections), UserList (moved to collections), and UserString (moved to collections)
All IRIX-specific, Mac-specific, and Solaris-specific modules removed
_winreg module renamed to winreg
ConfigParser module renamed to configparser
copy_reg module renamed to copyreg
Queue module renamed to queue
SocketServer module renamed to socketserver
cPickle module renamed to _pickle
cProfile module renamed to _profile
repr module renamed to reprlib
test.test_support module renamed to test.support
The modules anydbm, dbhash, dbm, dumbdm, gdbm, and whichdb have been combined into a new dbm module
HTMLParser module renamed to html.parser
htmlentitydefs module renamed to html.entities
httplib module renamed to http.client
The modules BaseHTTPServer, CGIHTTPServer, and SimpleHTTPServer have been combined into a new http.server module
Cookie module renamed to `http.cookies
cookielib module renamed to http.cookiejar

tkinter modules renamed as follows:

Old Name	New Name
Dialog	tkinter.dialog
FileDialog	tkinter.filedialog
FixTk	tkinter._fix
ScrolledText	tkinter.scrolledtext
SimpleDialog	tkinter.simpledialog
Tix	tkinter.tix
Tkconstants	tkinter.constants
Tkdnd	tkinter.dnd
Tkinter	tkinter.__init__
tkColorChooser	tkinter.colorchooser
tkCommonDialog	tkinter.commondialog
tkFileDialog	tkinter.filedialog
tkFont	tkinter.font
tkMessageBox	tkinter.messagebox
tkSimpleDialog	tkinter.simpledialog
turtle	tkinter.turtle

urllib2 module split into urllib.request and urllib.error modules
urlparse module renamed to urllib.parse
urllib module split into urllib.parse, urllib.request, and urllib.error modules
robotparser module renamed to urllib.robotparser
xmlrpclib module renamed to xmlrpc.client
The modules DocXMLRPCServer and SimpleXMLRPCServer have been combined into a new xmlrpc.server module
Exceptions must now inherit from BaseException
StandardError removed
Exceptions no longer behave like sequences; use the args attribute instead
except exc, var is now written except exc as var
The variable used to catch an exception is now deleted when the except block is left
raise Exception, args is now written raise Exception(args)
Raising an exception inside an except or finally block now causes implicit exception chaining
Explicit exception chaining can be done with raise SecondaryException() from primary_exception
__getslice__(), __setslice__(), and __delslice__() removed
Special method next() renamed to __next__()
__oct__() and __hex__() removed
Removed support for __members__ and __methods__
Function attributes of the form func_X renamed to __X__
Special method __nonzero__() renamed to __bool__()
super() can now be invoked without arguments inside an instance method

Python 3.1 (2009-06-26)

Release notes: [link]

Multiple context managers can now be used in a single with statement
Directories & zip archives containing a __main__.py can now be executed by passing their path to the interpreter
Packages containing a __main__ submodule can now be executed with python -m and runpy
New bytes and bytearray method: maketrans()
The repr()s of floats are now shorter
New int method: bit_length()
The fields in strings formatted with str.format() can now omit numbering in order to be automatically numbered, as in 'Sir {} of {}'.format('Gallahad', 'Camelot')
The format specification mini-language now includes a thousands separator specifier
round(x, n) now returns an integer if x is an integer
New modules: importlib and tkinter.ttk
collections module:
- Counter and OrderedDict added
- namedtuple() now accepts a rename parameter
contextlib module: nested() is now deprecated
decimal module: Decimal.from_float() added
io module: SEEK_SET, SEEK_CUR, and SEEK_END added
itertools module:
- combinations_with_replacement() and compress() added
- count() now accepts a step parameter
json module: Decoders now accept an object_pairs_hook parameter
logging module: NullHandler added
re module: sub(), subn(), and split() now accept a flags parameter
string module: maketrans() is now deprecated

Python 3.2 (2011-02-20)

Release notes: [link]

New modules: argparse, concurrent.futures, html, and sysconfig
.pyc files are now stored in __pycache__/ directories
Modules now have a __cached__ attribute
New str method: format_map()
The str() of a float or complex is now the same as its repr()
New range methods: index() and count()
callable() function from Python 2 restored
ResourceWarning warning type added
abc module: abstractclassmethod() and abstractstaticmethod() added
ast module: literal_eval() now supports set & bytes literals
collections module:
- Counter.subtract() added
- OrderedDict.move_to_end() added
- deque.count() and deque.reverse() added
compileall command-line interface: -i and -b options added
configparser module:
- ConfigParser class replaced with SafeConfigParser, which is now customizable
- New API added based on the mapping protocol
contextlib module:
- ContextDecorator added
- nested() removed
csv module:
- unix_dialect dialect (dialect name "unix") added
- DictWriter.writeheader() added
datetime module:
- timezone added
- timedelta instances can now be multiplied by floats and divided by floats & ints
- date.strftime() now supports years from 1000 through 9999
decimal module:
- The Decimal constructor now accepts floats
- Decimal instances can now be compared with float and fractions.Fraction instances
- Context.clamp added
email.parser module: BytesFeedParser, BytesParser, message_from_bytes() and message_from_binary_file() added
email.generator module: BytesGenerator added
fractions module: The Fraction constructor now accepts floats and decimal.Decimals
functools module: cmp_to_key(), lru_cache(), and total_ordering() added
gzip module: compress() and decompress() added
hashlib module: algorithms_available and algorithms_guaranteed added
Various ABCs added to importlib.abc
inspect module: getgeneratorstate() and getattr_static() added
io module: BytesIO.getbuffer() added
itertools module: accumulate() added
json module: The indent parameter to dumps() etc. can now be a string
logging module:
- basicConfig() now accepts a style parameter
- If a logging event occurs before any explicit configuration is set up, a default configuration (available in lastResort) is now enabled
- Python callables returning booleans can now be used as filters
logging.config module: dictConfig() added
math module: isfinite(), expm1(), erf(), erfc(), gamma(), and lgamma() added
os module: fsencode(), fsdecode(), supports_bytes_environ, getenvb(), and environb added
shutil module:
- copytree() now accepts ignore_dangling_symlinks and copy_function parameters
- make_archive(), unpack_archive(), etc. added
site module: getsitepackage(), getuserbase(), and getusersitepackages() added
smtplib module: SMTP.send_message() added
ssl module:
- SSLContext and match_hostname() added
- Server Name Indication (SNI) is now supported when linked against recent versions of OpenSSL
string module: maketrans() removed
sys module: hash_info added
tarfile module: TarFile.add() now accepts a filter parameter, and the exclude parameter is now deprecated
tempfile module: TemporaryDirectory added
threading module: Barrier added
unicodedata updated to Unicode 6.0.0
urllib.parse module: urlparse() now supports IPv6 addresses

Python 3.3 (2012-09-29)

Release notes: [link]

New modules: faulthandler, ipaddress, lzma, unittest.mock, and venv
Support for implicit namespace packages (directories without an __init__.py) added
All Unicode codepoints, from U+0000 to U+10FFFF, are now always supported; there is no longer a distinction between “narrow” and “wide” builds
Multiple exception types have been merged into OSError, which now has various subclasses for common error conditions
Delegating to a subgenerator/subiterator with yield from is now possible
Chained exception context can be suppressed with raise e from None
The u"unicode" syntax for Unicode strings from Python 2 is now supported again
Functions & classes now have a __qualname__ attribute
inspect module: signature(), Signature, Parameter, and BoundArguments added
sys module: implementation added
types module: SimpleNamespace added
importlib module: Various classes & functions added
Modules now have a __loader__ attribute
"\N{...}" can now take name aliases
unicodedata updated to UCD 6.1.0
New list and bytearray methods: copy() and clear()
Raw bytes literals can now be written rb"..." in addition to br"..."
open() now accepts an opener parameter
print() now accepts a flush parameter
Hash randomization with hash() is now enabled by default
New str method: casefold()
abc:
- It is now possible to combine abstractmethod with property, classmethod, or staticmethod
- abstractproperty, abstractclassmethod, and abstractstaticmethod are now deprecated
- ABCMeta.register() can now be used as a class decorator
array module: long long type now supported
base64 module: Decoding functions now accept ASCII-only strs
binascii module: The a2b_* functions now accept ASCII-only strs
bz2 module:
- open() added
- BZ2File() now accepts arbitrary file-like objects and implements most of the io.BufferedIOBase API
collections module:
- ChainMap added
- ABCs moved to collections.abc; aliases are still present in collections itself, but importing them is deprecated
- Counter now supports +, -, +=, -=, |=, and &=
contextlib module: ExitStack added
datetime module: New datetime methods: timestamp(), strftime(), and astimezone()
email module: Policy framework added
email.parser module: BytesHeaderParser added
email.utils module: format_datetime(), parsedate_to_datetime(), and localtime() added
functools module: lru_cache() now accepts a typed parameter
hmac module: compare_digest() added
http.client module: HTTPResponse.readinto() added
html.parser module: HTMLParser can now parse broken markup without errors
html.entities: html5 added
inspect module: getclosurevars() and getgeneratorlocals() added
io module:
- x mode added to open() function
- The TextIOWrapper constructor now accepts a write_through parameter
itertools module: accumulate() now accepts a func parameter
logging module: basicConfig() now accepts a handlers parameter
math module: log2() added
multiprocessing module:
- The Process constructor now accepts a daemon parameter
- Process.sentinel added
multiprocessing.connection module: wait() added
multiprocessing.pool module: New Pool methods starmap() and starmap_async() added
os module:
- fwalk(), pipe2(), sendfile(), getpriority(), setpriority(), replace(), get_terminal_size(), getxattr(), listxattr(), removexattr(), setxatter(), sync(), and others added
- Various functions now accept dir_fd and/or follow_symlinks parameters
- Various functions can now take file descriptors as path arguments
- stat(), fstat(), lstat(), and utime() now support timestamps with nanosecond precision
re module: str regular expressions now support \uXXXX and \UXXXXXXXX escapes
pipes.quote() moved to shlex
shutil module:
- disk_usage(), chown(), and get_terminal_size() added
- Several functions now accept a symlink parameter
stat module: filemode() added
struct module: size_t and ssize_t now supported
subprocess:
- Command strings can now be bytes on POSIX
- DEVNULL added
sys module: thread_info added
textwrap module: indent() added
time module: get_clock_info(), monotonic(), perf_counter(), process_time(), clock_getres(), clock_gettime(), clock_settime(), and CLOCK_* constants added
types module: MappingProxyType, new_class(), and prepare_class() added
urllib.request module: The Request constructor now accepts a method parameter
array module: The u format is now deprecated

Python 3.4 (2014-03-17)

Release notes: [link]

New modules: asyncio, ensurepip, enum, pathlib, selectors, statistics, and tracemalloc
codecs module: encode() and decode() are now documented
unicodedata updated to UCD 6.3
min() and max() now accept a default parameter
New special method: __length_hint__()
abc module: ABC and get_cache_token() added
argparse module: FileType now accepts encoding and errors parameters
base64 module:
- Encoding & decoding functions now accept any bytes-like object
- a85encode(), a85decode(), b85encode(), and b85decode() added
contextlib module: suppress() and redirect_stdout() added
doctest command-line interface: -o and -f options added
email module:
- as_string() now accepts a policy argument
- as_bytes() method added
- EmailMessage and MIMEPart added as part of new API
- contextmanager submodule added
filecmp module: clear_cache() and DEFAULT_IGNORES added
functools module: partialmethod() and singledispatch() added
glob module: escape() added
hashlib module: pbkdf2_hmac() added
html module: unescape() added
html.parser module: HTMLParser constructor now accepts a convert_charrefs parameter, and the strict argument is now deprecated
http.server command-line interface: --bind option added
imp.reload() moved to importlib
importlib module: InspectLoader.source_to_code() added
importlib.util module: MAGIC_NUMBER, cache_from_source(), source_from_cache(), and decode_source() added
importlib.machinery module: ExtensionFileLoader.get_filename() added
runpy and python -m can now be used with namespace packages
inspect module:
- Command-line interface added
- unwrap() added
ipaddress module: is_global property added
json module: dumps() etc. will now automatically set separators to (",", ": ") when indent is non-None
multiprocessing module:
- Start methods spawn and forkserver added
- Contexts added
- get_all_start_methods(), get_start_method(), set_start_method(), and get_context() added
operator module: length_hint() added
os module:
- cpu_count() added
- open() now supports the O_PATH and O_TMPFILE flags
pdb module: print command removed
pickle module: Protocol 4 added
plistlib module: load(), dump(), loads(), and dumps() added
pprint module: PrettyPrinter, pformat(), and pprint() now accept a compact parameter
re module: fullmatch() and regex.fullmatch() added
resource module: prlimit() added
shutil module: copyfile() now raises a SameFileError when the source and destination are the same file
ssl module:
- create_default_context() and get_default_verify_paths() added
- New SSLContext methods: cert_store_stats(), get_ca_certs(), and load_default_certs()
stat module: S_IFDOOR, S_IFPORT, and S_IFWHT added
struct module: iter_unpack() and Struct.iter_unpack() added
subprocess module:
- check_output() now accepts an input parameter
sys module: getallocatedblocks() and __interactivehook__ added
tarfile module: Command-line interface added
textwrap module:
- The TextWrapper constructor now accepts max_lines and placeholder parameters
- shorten() added
threading module: main_thread() added
traceback module: clear_frames() added
types module: DynamicClassAttribute() added
urllib.request module:
- data: URLs now supported
- Request objects are now reusable
urllib.error module: HTTPError.headers added
venv module: The EnvBuilder constructor and create() now accept a with_pip parameter
importlib module: A number of methods & functions are deprecated
The imp module is now deprecated and will be removed in Python 3.12
The formatter module is now deprecated and will be removed in Python 3.10
The U mode of various “open” functions is now deprecated and will be removed in Python 3.11

Python 3.5 (2015-09-13)

Release notes: [link]

Coroutine functions (async def), awaitable objects (await and __await__()), asynchronous iteration (async for, __aiter__(), and __anext__()), and asynchronous context managers (async with, __aenter__(), and __aexit__()) added
@ operator (with __matmul__() etc. special methods) for matrix multiplication added
Multiple * and/or ** unpackings can now be used in a single function call
Tuple, list, set, & dictionary displays may now contain * or ** unpackings (as appropriate)
Percent-formatting of bytes and bytearray objects with % added
New modules: typing and zipapp
System calls are now retried when interrupted by a signal
from __future__ import generator_stop added to cause StopIteration exceptions raised inside generators to be transformed into RuntimeExceptions, which becomes the default in Python 3.7
- Without the __future__ import, such exceptions generate PendingDeprecationWarnings.
RecursionError exception type (a subclass of RuntimeError) added
.pyo files eliminated; optimized bytecode is now stored in .pyc files with opt- tags in their name
"namereplace" error handler added
Various additions & improvements to the asyncio module
cmath module: isclose() added
collections module:
- New deque methods: index(), insert(), and copy()
- deque now supports + and *
collections.abc module: Generator, Awaitable, Coroutine, AsyncIterator, and AsyncIterable added
configparser module: ConfigParser can now take a dictionary of converters, and subclasses can define additional converters as methods
contextlib module: redirect_stderr() added
enum module: The Enum callable now accepts a start parameter
glob module: glob() and iglob()` now support the ``** pattern
http module: HTTPStatus added
importlib.util module: module_from_spec() added
inspect module:
- BoundArguments.apply_defaults() added
- Signature.from_callable() added
- signature() now accepts a follow_wrapped parameter
- iscoroutine(), iscoroutinefunction(), isawaitable(), getcoroutinelocals(), and getcoroutinestate() added
io module: new BufferedIOBase method: readinto1()
ipaddress module:
- The IPv4Network and IPv6Network constructors now accept an (address, netmask) argument
- New IPv4Network and IPv6Network attribute: reverse_pointer
json module: JSON decoding errors now raise JSONDecodeError
json.tool command-line interface: The input order of keys is now preserved on output; the --sort-keys option will sort the keys instead
linecache module: lazycache() added
locale module: delocalize() added
logging module: Logging methods now accept exception instances as exc_info arguments
math module: isclose(), gcd(), inf, and nan added
fractions module: gcd() is now deprecated and will be removed in Python 3.9
operator module: matmul() and imatmul() added
os module: scandir() added
os.path module: commonpath() added
pathlib module:
- New Path methods: samefile(), expanduser(), write_text(), read_text(), write_bytes(), and read_bytes()
- Class method Path.home() added
- Path.mkdir() now accepts an exist_ok parameter
readline module: append_history_file() added
selectors module: DevpollSelector added
shutil module: move() now accepts a copy_function argument
signal module: SIG* constants have been converted to enums
socket module: socket.sendfile() added
ssl module: SSLObject added
subprocess: run() added
sys module: set_coroutine_wrapper(), get_coroutine_wrapper(), and is_finalizing() added
time module: monotonic() is now always available
timeit command-line interface: --unit option added
traceback module: TracebackException, StackSummary, FrameSummary, walk_stack(), and walk_tb() added
types module: CoroutineType and coroutine() added
unicodedata updated to Unicode 8.0.0
unittest command-line interface: --locals option added
unittest.mock module: Mock.assert_not_called() added
urllib.request module: HTTPPasswordMgrWithPriorAuth added
platform module: dist() and linux_distribution() are now deprecated
html.parser module: The convert_charrefs parameter to the HTMLParser constructor now defaults to True

Python 3.6 (2016-12-23)

Release notes: [link]

Formatted string literals (“f-strings”)
Variables can now be annotated by following the name of the variable with a colon and the annotation
Underscores can now be used in numeric literals
await and yield can now be used in the same function, thereby enabling asynchronous generators
async for can now be used in list, set, & dict comprehensions and in generator expressions
await expressions can now be used in any comprehension
Special methods __init_subclass__() and __set_name__() added
os.PathLike, the __fspath__() method, and os.fspath() added
- Relevant file functions now accept os.PathLike objects
ModuleNotFoundError exception type (a subclass of ImportError) added
datetime module:
- fold attribute added to datetime and time for denoting the second instance of a time duplicated due to DST
- The strftime() method of date and datetime now supports %G, %u, and %V
- datetime.astimezone() can now be called on naïve instances
The file system and console encodings on Windows are now both UTF-8
A class’s __dict__ now preserves the order in which the attributes were defined
**kwargs now preserves insertion order
dicts are now implemented in such a way that they preserve insertion order
New module: secrets
Various additions & improvements to the asyncio module
cmath module: tau, inf, nan, infj, and nanj added
collections.abc: Collection, Reversible, and AsyncGenerator added
enum module: auto, Flag, and IntFlag added
json module: load() and loads() now support binary input in UTF-8, UTF-16, and UTF-32
math module: tau added
random module: choices() added
re module:
- Modifier spans (e.g., as in '(?i)g(?-i:v)r') are now supported in regular expressions
- Match objects can now be indexed to access groups
statistics module: harmonic_mean() added
The smtpd module is now deprecated and will be removed in Python 3.12
subprocess module:
- encoding and errors arguments added to Popen and the wrappers around it
- The args parameter to Popen and the wrappers around it can now be a path-like object or sequence of path-like objects on POSIX systems
time module: The tm_gmtoff and tm_zone attributes of struct_time are now available on all platforms
typing module:
- Generic type aliases like Dict[str, Tuple[S, T]] are now supported
- TYPE_CHECKING added
- ClassVar added
- NewType() added
- NamedTuple now supports variable annotation syntax
unittest.mock module: New Mock methods assert_called() and assert_called_once() added
venv command-line interface: --prompt option added
Using async or await as a name will now generate a DeprecationWarning
StopIteration exceptions raised inside generators now generate DeprecationWarnings
Invalid escape sequences now generate a DeprecationWarning
The asynchat and asyncore modules are now deprecated and will be removed in Python 3.12
The pyvenv script for creating venvs is now deprecated and will be removed in Python 3.8
unicodedata updated to Unicode 9.0.0

Python 3.7 (2018-06-27)

Release notes: [link]

from __future__ import annotations added to enable postponed evaluation of annotations
dicts are now guaranteed to preserve insertion order
async and await are now reserved keywords
New modules: contextvars, dataclasses, importlib.resources
New builtin function: breakpoint()
The interpreter now coerces ASCII locales to UTF-8 under certain circumstances on non-Windows OSes
__aiter__() methods can no longer be asynchronous
__getattr__() and __dir__() can now be defined on modules
time module: variants of the timer functions added that return a number of nanoseconds as an integer
Special methods __class_getitem__() and __mro_entries__() added
Python Development Mode added
New str, bytes, and bytearray method: isascii()
argparse module: ArgumentParser.parser_intermixed_args() added
Various additions & improvements to the asyncio module
- asyncio.run() added
collections module: defaults argument added to namedtuple()
contextlib module: nullcontext(), asynccontextmanager(), and AsyncExitStack added
datetime module:
- datetime.fromisoformat() added
- The "%z" format of the strptime() methods now accepts timezone offsets containing colons as well as a timezone specifier of “Z”.
enum module: Support for the _ignore_ class property added to Enum
functools module: singledispatch() now recognizes type annotations
http.server command-line interface: --directory option added
ipaddress module: subnet_of() and supernet_of() methods added to IPv4Network and IPv6Network
locale module: format() is now deprecated and will be removed in Python 3.12
math module: remainder() added
pathlib module: Path.is_mount() added
queue module: SimpleQueue added
subprocess module:
- capture_output argument added to run()
- text argument added to run() and the Popen constructor
Removed the fpectl module
StopIteration exceptions raised inside coroutines and generators are now transformed into RuntimeExceptions
unicodedata updated to Unicode 11

Python 3.8 (2019-10-14)

Release notes: [link]

Assignment expressions: := (the “walrus operator”) can now be used to assign a value to a variable in the middle of an expression, e.g.:
```
if m := re.search(r'\d+', s):
    x = int(m.group())
```
Function parameters can now be made positional-only by placing a / after them in the argument list
One can now write f"{var=}" to get f"{var}={repr(var)}"
continue is now allowed in finally: clauses
New int, bool, and fractions.Fraction method: as_integer_ratio()
\N{name} escapes are now allowed in regular expressions
dicts and dictviews can now be passed to reversed()
Generalized iterable unpacking in yield and return statements no longer requires parentheses; e.g., one can now write return foo, *bar instead of having to do return (foo, *bar)
Missing commas between tuples in a list now generate a SyntaxWarning with a suggestion as to what went wrong
For integer arguments, the three-argument form of pow() can now take a negative exponent when the base is coprime to the modulus, in which case the modular multiplicative inverse (or a power thereof) is calculated
New modules: importlib.metadata and multiprocessing.shared_memory
Running python -m asyncio now starts an async REPL
asyncio module:
- CancelledError now inherits directly from BaseException instead of Exception
- coroutine() is now deprecated and will be removed in Python 3.11
- Passing a loop parameter is now deprecated for most of asyncio’s high-level API; it will be removed entirely in Python 3.10
- Explicitly passing coroutines to wait() is now deprecated
datetime module: New date and datetime method: fromisocalendar()
functools module: cached_property() and singledispatchmethod() added
io module: open_code() (the verified open hook) added
itertools module: accumulate() now accepts an initial parameter
json.tool command-line interface: --json-lines option added
math module:
- comb() added
- dist() added
- hypot() can now take multiple arguments
- isqrt() added
- perm() added
- prod() added
pathlib module:
- Path.link_to() added
- Path.rename() and Path.replace() now return the new path
- Path.unlink() now accepts a missing_ok argument
pickle module: Protocol 5 (with support for out-of-band buffers) added
platform module: dist() and linux_distribution() removed
shlex module: join() added
statistics module: NormalDist, fmean(), geometric_mean(), multimode(), and quantiles() added
subprocess module: The args parameter to Popen and the wrappers around it can now be a path-like object or sequence of path-like objects on Windows systems in addition to POSIX
sys module:
- audit() and addaudithook() (for audit hooks) added
- pycache_prefix added
- unraisablehook() and __unraisablehook__ added
typing module: TypedDict, Literal, Final, final(), Protocol, SupportsIndex, get_origin(), and get_args() added
unicodedata module:
- Updated to Unicode 12.1.0
- is_normalized() added
unittest.mock module:
- AsyncMock added
- call objects now have args and kwargs properties
zipfile module: Path added
Removed:
- macpath module (deprecated since Python 3.7)
- time.clock() (deprecated since Python 3.3)
- the pyvenv script for creating venvs (Use pythonX.Y -m venv instead)
- sys.set_coroutine_wrapper() and sys.get_coroutine_wrapper() (deprecated since Python 3.7)
Using is or is not with strings, numbers, and certain other literals now produces a SyntaxWarning

Python 3.9 (2020-10-05)

Release notes: [link]

dicts can now be merged & updated using the | and |= operators
Any valid expression can now be used as a decorator
New str, bytes, and bytearray methods: removeprefix() and removesuffix()
Built-in collection types like list and dict can now be used as generic types; e.g., List[str] can now be written list[str]
- This also applies to collections ABCs; e.g., typing.Sequence[str] can now be written collections.abc.Sequence[str].
- Importing the old List, Sequence, etc. types from typing is now deprecated but does not generate DeprecationWarningss at this time. The deprecated names will be removed from typing in the first Python release five years after 3.9.
New modules: graphlib and zoneinfo
asyncio module: to_thread() added
functools module: cache() added
importlib.resources module: files() added (introduced in importlib-resources v1.1.0)
Aligned importlib.metadata with importlib-metadata v1.6.1
ipaddress module now supports IPv6 scoped addresses
math module:
- gcd() can now take multiple arguments
- lcm(), nextafter(), and ulp() added
pathlib module:
- Path.readlink() added
- PurePath.is_relative_to() added
- PurePath.with_stem() added
random module: randbytes() added
typing module: Annotated added
The binhex, parser, and symbol modules are now deprecated
Using NotImplemented in a boolean context is now deprecated and will produce a TypeError in a future version of Python
Removed:
- fractions.gcd() (deprecated since Python 3.5)
- encoding parameter of json.loads() (deprecated & ignored since Python 3.1)
- asyncio.Task.current_task() and asyncio.Task.all_tasks() (deprecated since Python 3.7); use asyncio.current_task() and asyncio.all_tasks() instead
with (await asyncio.lock): and with (yield from asyncio.lock): statements are no longer supported; use async with lock instead. Likewise for asyncio.Condition and asyncio.Semaphore.
lib2to3 now emits a PendingDeprecationWarning and will be removed in Python 3.13
unicodedata updated to Unicode 13.0.0

Python 3.10 (2021-10-04)

Release notes: [link]

Pattern matching!

match status:
    case 400:
        return "Bad request"
    case 401 | 403 | 404:
        return "Not allowed"
    case _:
        return "Something's wrong with the Internet"

Context managers in with statements can now be enclosed in parentheses, e.g.:
```
with (CtxManager1() as example,
      CtxManager2()):
```
Assignment expressions can now be used unparenthesized within set literals, set comprehensions, and sequence indexes (but not slices)
Numeric literals immediately followed by keywords (e.g., 0in x) now generate a deprecation warning. Future Python versions will change this to a syntax warning and then a syntax error.
Common syntax errors now have better error messages
AttributeError and NameError error messages now include suggestions as to what you might have meant
Union[X, Y] can now be written X | Y
Dictionary views returned by dict.keys(), dict.values(), and dict.items() now have mapping attributes wrapping the original dict
The second argument of isinstance() and issubclass() can now be a Union
New int method: bit_count()
open() and friends can now be passed encoding="locale" in order to explicitly use the current locale’s encoding
zip() now has a strict parameter for requiring that all input iterables have the same length
New builtin functions: aiter() and anext()
EncodingWarning warning type added
The loop parameter (deprecated in Python 3.8) is now removed from most of asyncio’s high-level API
base64 module: b32hexencode() and b32hexdecode() added
bisect module: The functions now accept a key argument
codecs module: unregister() added
Collections ABCs can no longer be imported from collections; import them from collections.abc instead
collections module: Counter.total() added
contextlib: aclosing() added
dataclasses:
- The dataclass decorator now accepts a slots argument
- Fields can now be made keyword-only
distutils is now deprecated and will be removed in Python 3.12
glob module: glob() and iglob() now accept root_dir and dir_fd arguments
Aligned importlib.metadata with importlib-metadata v4.6
- entry_points() and package_distributions() added
inspect module: get_annotations() added
io module: text_encoding() added
itertools module: pairwise() added
os.path.realpath() now has a strict parameter for erroring when a path doesn’t exist or a symlink loop is encountered
pathlib module:
- Slice and negative indexing support added to Path.parents
- Path.hardlink_to() added, superseding Path.link_to(), which is now deprecated and will be removed in Python 3.12
- Path.stat() and Path.chmod() now have a follow_symlinks argument
platform module: freedesktop_os_release() added
statistics module: covariance(), correlation(), and linear_regression() added
sys module: orig_argv and stdlib_module_names added
traceback module: format_exception(), format_exception_only(), and print_exception() can now take just an exception argument as a positional-only parameter
types module: EllipsisType, NoneType, and NotImplementedType added
typing module: Concatenate, ParamSpec, TypeAlias, TypeGuard, and is_typeddict() added
Removed the formatter and parser modules (deprecated in Python 3.4 and 3.9, respectively)

Python 3.11 (2022-10-03)

Release notes: [link]

Exception groups
- ExceptionGroup and BaseExceptionGroup exception types added
- try: ... except* ...: ... syntax (note asterisk) for matching exception groups
Traceback messages now highlight the exact expression that caused the error
Starred expressions can now be used in for statements (e.g., for x in *lst1, *lst2:)
- This was enabled by the grammar rewrite in Python 3.9 but was not noticed or documented until 3.11.
Octal escapes in string & bytes literals with values greater than 0o377 (e.g., \477) now generate a DeprecationWarning
The various “open” functions no longer accept the U mode
New object method: __getstate__()
New BaseException method: add_note()
Chaining @classmethod decorators and using them to wrap other decorators (such as @property) is now deprecated and will be removed in Python 3.13
Format specification mini-language: A z option can now be used with floating-point types in order to coerce negative zero to “regular” zero after rounding.
New modules: tomllib and wsgiref.types
asyncio module:
- @coroutine (deprecated since 3.8) and asyncio.coroutines.CoroWrapper removed
- Barrier, Runner, TaskGroup, and timeout() added
binascii module: a2b_hqx(), b2a_hqx(), rlecode_hqx(), and rledecode_hqx() removed
binhex module removed
contextlib module: chdir() added
datetime module:
- UTC added
- The fromisoformat() methods now accept most ISO 8601 strings
enum module:
- EnumCheck, FlagBoundary, ReprEnum, StrEnum, @global_enum(), @member(), @nonmember(), @property, show_flag_values(), and @verify added
- EnumMeta renamed to EnumType; the old name is currently as an alias
functools module: singledispatch now supports union types
hashlib module: file_digest() added
inspect module:
- getmembers_static() and ismethodwrapper() added
- formatargspec() and getargspec() removed
- Frame-related functions now return objects instead of tuples (though the new objects are backwards-compatible with the tuple interface)
locale module:
- getdefaultlocale() is now deprecated and will be removed in Python 3.13
- getencoding() added
logging module:
- getLevelNamesMapping() added
- createSocket() method added to logging.handlers.SysLogHandler
math module: cbrt() and exp2() added
operator module: call() added
pathlib module: The glob() and rglob() methods now return only directories if given a pattern ending with the pathname separator
re module:
- (?>...), *+, ++, ?+, and {m,n}+ can now be used in regular expressions
- NOFLAG added
string module: get_identifiers() and is_valid() methods added to string.Template
sys module: exception() added
typing module:
- LiteralString, Never, Self, TypeVarTuple, Unpack, assert_never(), assert_type(), clear_overloads(), get_overloads(), @dataclass_transform, and reveal_type() added
- Text is now deprecated
- Individual fields in a TypedDict may now be annotated as Required or NotRequired
- The keyword-argument syntax for constructing TypedDict types is now deprecated and will be removed in Python 3.13
unicodedata updated to Unicode 14.0.0
zipfile module:
- metadata_encoding argument added to ZipFile constructor
- ZipFile.mkdir() added
- stem, suffix, and suffixes attributes added to zipfile.Path
The following modules are now deprecated and are planned to be removed in Python 3.13: aifc, audioop, cgi, cgitb, chunk, crypt, imghdr, mailcap, msilib, nis, nntplib, ossaudiodev, pipes, sndhdr, spwd, sunau, telnetlib, uu, and xdrlib

Python 3.12 (2023-10-02)

Release notes: [link]

New syntax for type-annotating generic classes, generic functions, and type aliases:

Generic class:

Old syntax:

from typing import Generic, TypeVar

T_str = TypeVar("T_str", covariant=True, bound=str)

class ClassA(Generic[T_str]):
    def method1(self) -> T_str:
        ...

New syntax:

# No Generic or TypeVar

class ClassA[T: str]:
    def method1(self) -> T:
        ...

Generic function:

Old syntax:

from typing import TypeVar

T = TypeVar("T")

def func(a: T, b: T) -> T:
    ...

New syntax:

# No TypeVar

def func[T](a: T, b: T) -> T:
    ...

Type alias:

Old syntax:

from typing import TypeAlias

IntOrStr: TypeAlias = int | str

New syntax:
```
type IntOrStr = int | str
```

Generic type alias:

Old syntax:

from typing import TypeAlias

T = TypeVar("T")

ListOrSet: TypeAlias = list[T] | set[T]

New syntax:

# No TypeVar

type ListOrSet[T] = list[T] | set[T]

Variadic type alias:

Old syntax:

from typing import TypeAlias, TypeVarTuple

Ts = TypeVarTuple("Ts")

IntEtc: TypeAlias = tuple[int, *Ts]

New syntax:
```
type IntEtc[*Ts] = tuple[int, *Ts]
```

Type alias with a parameter spec:

Old syntax:

from collections.abc import Callable
from typing import ParamSpec, TypeAlias

P = ParamSpec("P")

IntMaker: TypeAlias = Callable[P, int]

New syntax:

from collections.abc import Callable

type IntMaker[**P] = Callable[P, int]

Types of individual fields in a **kwargs function parameter can now be annotated with typing.Unpack applied to a TypedDict, like so:

from typing import TypedDict, Unpack

class Movie(TypedDict):
    name: str
    year: int

def foo(**kwargs: Unpack[Movie]):
    # kwargs["name"] must be passed a `str`, and kwargs["year"] must be
    # passed an `int`.
    ...

{..} placeholders inside f-strings may now contain any valid Python expression
- Quotes from the enclosing string may now be reused:
```
>>> songs = ['Take me back to Eden', 'Alkaline', 'Ascensionism']
>>> f"This is the playlist: {", ".join(songs)}"
'This is the playlist: Take me back to Eden, Alkaline, Ascensionism'
```
- Expressions inside f-strings may now span multiple lines and contain inline comments.
- Expressions inside f-strings may now contain backslashes.
The buffer protocol is now accessible in Python code:
- Special methods __buffer__() and __release_buffer__() added
- collections.abc module: Buffer added
- inspect module: BufferFlags added
Source code containing NUL bytes now results in a SyntaxError
Invalid escape sequences now generate a SyntaxWarning
Octal escapes in string & bytes literals with values greater than 0o377 (e.g., \477) now generate a SyntaxWarning
Applying ~ (bitwise inversion) to a bool is now deprecated and will become an error in Python 3.14.
Removed modules:
- asynchat (deprecated in Python 3.6)
- asyncore (deprecated in Python 3.6)
- distutils (deprecated in Python 3.10)
- imp (deprecated in Python 3.4)
- smtpd (deprecated in Python 3.6)
array module: The array class is now subscriptable and thus properly generic
ast module: Accessing or using any of Bytes, Ellipsis, NameConstant, Num, and Str now generates a DeprecationWarning. They are planned to be removed in Python 3.14.
asyncio module:
- as_completed() and wait() now accept generators yielding tasks
- create_eager_task_factory() and eager_task_factory() added
- run() now takes an optional loop_factory parameter
calendar module: Day and Month added
collections.abc module: ByteString is now deprecated and will be removed in Python 3.14
configparser module: ConfigParser.readfp(), ParsingError.filename, and SafeConfigParser removed (deprecated in Python 3.2)
datetime module: datetime.utcnow() and datetime.utctimestamp() are now deprecated
fraction module: Fraction instances now support float-style formatting
importlib.abc module: ResourceReader, Traversable, and TraversableResources are now deprecated and will be removed in Python 3.14
importlib.resources: as_file() now supports resource directories
inspect module: getasyncgenlocals(), getasyncgenstate(), and markcoroutinefunction() added
itertools: batched() added
locale module: format() removed (deprecated in Python 3.7)
math module:
- nextafter() now takes an optional steps parameter
- sumprod() added
os module:
- DirEntry.is_junction() added
- listdrives(), listmounts(), listvolumes(), pidfd_open(), setns(), and unshare() added
- Calling fork() or forkpty() in a multithreaded program now generates a DeprecationWarning
os.path module: isjunction() and splitroot() added
pathlib module:
- Path classes can now be subclassed
- PurePath.with_segments() added
- Path.is_junction() and Path.walk() added
- PurePath.match(), Path.glob(), and Path.rglob() now take an optional case_sensitive parameter
- PurePath.relative_to() now takes an optional walk_up parameter
pkgutil module: find_loader() and get_loader() are now deprecated and will be removed in Python 3.14
random module:
- binomialvariate() added
- The lambd argument to expovariate() now defaults to 1.0
shutil module:
- rmtree() now takes an optional onexc parameter. The onerror parameter is deprecated and will be removed in Python 3.14.
- unpack_archive() now takes an optional filter parameter for limiting what gets extracted from tar archives
sqlite3 module:
- Command-line interface added
- connect() now takes an optional autocommit parameter
- Connection.autocommit, Connection.getconfig(), and Connection.setconfig() added
- The default adapters and converters are now deprecated
ssl module: match_hostname() and wrap_socket() removed (deprecated in Python 3.7)
statistics module: correlation() now takes an optional method parameter that can be either "linear" (the default) or "ranked"
sys module:
- activate_stack_trampoline(), deactivate_stack_trampoline(), is_stack_trampoline_active(), and last_exc added
- last_type, last_value, and last_traceback are now deprecated
sys.monitoring added
tarfile module: The TarFile.extract() and TarFile.extractall() methods now take an optional filter parameter for limiting what gets extracted in order to avoid security issues. Calling one of these methods without setting a filter will generate a DeprecationWarning until Python 3.14, in which the default filter will become "data".
tempfile module:
- NamedTemporaryFile now takes an optional delete_on_close parameter
- mkdtemp() now always returns an absolute path
types module: get_original_bases() added
typing module:
- override added
- dataclass_transform() now takes an optional frozen_default parameter
- Hashable and Sized are now deprecated
unicodedata updated to Unicode 15.0.0
unittest command-line interface: --durations option added
uuid module: Command-line interface added
venv module: setuptools is no longer automatically installed in virtual environments

Using Package Data in Python Projects with Setuptools

2020-12-29T00:00:00-05:00

When creating a Python project, you may want to include a number of non-Python files in the project that the code can then access at runtime, such as templates, images, and data. These files are called package data [1], and they are included in & accessed from your project as follows.

Contents

Including Package Data in Projects
Accessing Package Data at Runtime
- Installing & Importing importlib-resources
- The importlib-resources API
Footnotes

Including Package Data in Projects

Note

The following section only pertains to projects built with setuptools. Users of flit, Poetry, and other tools must consult their respective backend’s documentation to see how to include package data.

The first requirement for being able to include package data in your package [2] is for the files to be located somewhere inside the actual package directory. If your project is a single-file .py module, you must convert it to a directory/package layout before you can store package data in it.

Once your package data is inside the package, you must tell setuptools to include the files in both the project’s wheel (so that the files will be installed when your package is installed) and sdist (so that the files can be included in the wheel when building from an sdist). There are two different major ways to do this: a blanket “include all” method that uses a MANIFEST.in file and a fine-grained method that configures one package & subpackage at time via the setup.py/setup.cfg file.

Important

In addition to the configurations described here, you also need to ensure that all folders within your package that contain package data (even if they contain no Python source code) are listed in the packages option in setup.py/setup.cfg. This can be done automatically by using the setuptools.find_namespace_packages() function instead of find_packages(). If you don’t do this, setuptools will emit a warning message starting in version 62.3.0, and later versions will cease to include such packages in your distribution at all.

Including Package Data via `MANIFEST.in`

There are two steps to this method:

Create or edit your project’s MANIFEST.in file with the proper commands to tell setuptools to include all of your package data file in the project’s sdists. See the Python Packaging User Guide for information on how to do this.

In the simplest case — in which you want all non-Python files in your package to be treated as package data — you can simply include your entire package in your sdist with the MANIFEST.in command “graft packagename” (or “graft src” if you’re using a src/ layout). If you use this command, you should also follow it with “global-exclude *.py[cod]” so that compiled Python bytecode files are not included in the sdist.
In your setup.py, pass include_package_data=True to the setup() function. If you are placing your project configuration in setup.cfg instead, set include_package_data = True in the [options] section of setup.cfg.

Doing this tells setuptools that any non-Python files found inside your package that are included in the sdist should also be included in the wheel.

Including Package Data via `setup.py`/`setup.cfg`

This method allows you to specify package data file patterns on a per-package basis through the package_data argument to setup(). package_data must be a dict mapping package or subpackage names (or the empty string, to specify all packages & subpackages) to lists of glob patterns indicating which files within those packages to include in the sdist and wheel.

A sample package_data looks like this:

package_data={
    # If any package or subpackage contains *.txt or *.rst files, include
    # them:
    "": ["*.txt", "*.rst"],
    # Include any *.msg files found in the "hello" package (but not in its
    # subpackages):
    "hello": ["*.msg"],
    # Include any *.csv files found in the "hello.utils" package:
    "hello.utils": ["*.csv"],
    # Include any *.dat files found in the "data" subdirectory of the
    # "mypkg" package:
    "mypkg": ["data/*.dat"],
}

If you are placing your project configuration in setup.cfg, you must instead specify package_data via an [options.package_data] section in which the keys are the package & subpackage names — using * instead of the empty string to signify all packages — and the values are comma-separated glob patterns. The above setup.py sample translates to setup.cfg as follows:

[options.package_data]
# If any package or subpackage contains *.txt or *.rst files, include them:
* = *.txt, *.rst
# Include any *.msg files found in the "hello" package (but not in its
# subpackages):
hello = *.msg
# Include any *.csv files found in the "hello.utils" package:
hello.utils = *.csv
# Include any *.dat files found in the "data" subdirectory of the "mypkg"
# package:
mypkg = data/*.dat

Note that glob patterns only select files located directly within the given package (or in the given subdirectory of the package, if the pattern includes a directory path); e.g., "hello": ["*.msg"] selects *.msg files in the hello package but not in any of its subpackages. To select files in subpackages, you must either include an entry for each subpackage or else use the empty string key (or asterisk key in setup.cfg) to specify a pattern for all packages & subpackages.

If a pattern contains any directory components, the forward slash (/) must be used as the directory separator, even on Windows.

If a package data file is located in a directory that does not have an __init__.py file (say, a data/ directory inside package.subpackage), that directory does not count as a package, and the file must be listed in package_data in the form "package.subpackage": ["data/PATTERN"].

Warning

If you use both include_package_data and package_data, files specified with package_data will not be automatically included in sdists; you must instead list them in your MANIFEST.in.

Excluding Package Data via `setup.py`/`setup.cfg`

The exclude_package_data argument to setup() can be used in conjunction with either of the above methods to prevent one or more files from being treated as package data. exclude_package_data takes a dict with the same structure as package_data, and any matched files are excluded from wheels. Matched files are also excluded from sdists if they are not already matched by the project’s MANIFEST.in.

In a setup.cfg, exclude_package_data becomes an [options.exclude_package_data] section whose contents have the same structure as [options.package_data].

Including Package Data via Setuptools Plugins

As an alternative to the above methods, you can use a plugin for setuptools that automatically recognizes & includes package data in sdists & wheels, usually based on what files in the project directory are under version control. One example of such a plugin is setuptools_scm, which automatically finds all files under version control in a Git or Mercurial repository and augments the project’s MANIFEST.in (if any) with the found files. This eliminates the need to write a MANIFEST.in manually (unless there are files under version control that you want to exclude from sdists or wheels), though you still need to set include_package_data to True for files in your package directory to be included in wheels.

Accessing Package Data at Runtime

There have been multiple ways to access package data over the years, from pkg_resources’ ResourceManager API to pkgutil.get_data(), but the most recent and currently-recommended way is with the importlib-resources package.

Installing & Importing `importlib-resources`

There are two versions of importlib-resources available:

The one on PyPI that is installed with pip install importlib-resources and imported with import importlib_resources (note underscore)
The one in the Python standard library starting with Python 3.7 that is imported with import importlib.resources (note period)

Development of the PyPI version tends to be ahead of whatever’s in the latest Python version. In particular, the new files()-based API described here was only introduced in version 1.1.0 of the PyPI project and was only added to the Python standard library in Python 3.9. In order to be guaranteed a version of importlib-resources that supports this API, you should add the following to your project’s install_requires:

importlib-resources>=1.1.0; python_version < '3.9'

and import importlib-resources in your code as follows:

import sys

if sys.version_info < (3, 9):
    # importlib.resources either doesn't exist or lacks the files()
    # function, so use the PyPI version:
    import importlib_resources
else:
    # importlib.resources has files(), so use that:
    import importlib.resources as importlib_resources

The `importlib-resources` API

To access a package data file in your project, start by calling importlib_resources.files() on the name of your package:

pkg = importlib_resources.files("packagename")
# The argument can optionally refer to a subpackage in the form
# "packagename.subpackage".

This gives you a Traversable object that acts like a limited pathlib.Path object for traversing package data files. To refer to a data.csv file in a data/ directory in your package, write:

pkg_data_file = pkg / "data" / "data.csv"

So now that we’ve got a reference to the package data file, how do we get anything out of it?

To open the file for reading, call the open() method:

with pkg_data_file.open() as fp:
    # Do things with fp

To get the file’s contents as bytes, call the read_bytes() method:
```
b = pkg_data_file.read_bytes()
```
To get the file’s contents as a str, call the read_text() method, optionally with an encoding argument:
```
s = pkg_data_file.read_text(encoding="utf-8")
```
To get the path to the file, call importlib_resources.as_file() on it and use the return value as a context manager:
```
with importlib_resources.as_file(pkg_data_file) as path:
    # Do things with the pathlib.Path object that is `path`
```
The use of context managers allows importlib-resources to support packages stored in zipfiles; when a path is requested for a package data file in a zipfile, the library can extract the file to a temporary location at the start of the with block and remove it at the end of the block.
To iterate through a directory (either a package or a non-package directory), use the iterdir() method. You can test whether a resource is a directory or a file with the is_dir() and is_file() methods, and you can get a resource’s basename via the name property:
```
for entry in (pkg / "data").iterdir():
    if entry.is_dir():
        print(entry.name, "DIR")
    else:
        print(entry.name, "FILE")
```

Footnotes

[1]	Specifically, package data files are files that are stored in a Python project’s package directory next to the Python source files. An alternative to package data is data files, which are files installed elsewhere on the file system. This article only deals with the former.

[2]	Throughout this article, the term “package” is used in the sense of a directory of `.py` files and other packages (a.k.a. an “import package”), not in the sense of a “distribution package” (i.e., an sdist or wheel).

Common Python Packaging Mistakes

2020-08-22T00:00:00-04:00

I think we can all agree that packaging a Python project is harder than it should be. With numerous guides & tutorials out there, people still make mistakes. Some of these mistakes break a project, some just make it less attractive, and some even cause a project to step on the toes of other projects.

As the admin of the wheel-analysis and -browsing site Wheelodex, I see a number of poorly-built wheels each morning as I peruse the day’s new entries. This eventually motivated me to create check-wheel-contents — a program for scanning a wheel for many of the below problems plus several others — in an attempt to get people to clean up their wheels, yet still the poorly-packaged projects persist.

In yet another attempt to get people to fix their broken packages, here now are some of the more frequent types of mistakes I see — along with advice on how to avoid & correct them — in no particular order.

Note

Unless otherwise specified, references to packaging configurations assume that the project is using setuptools and that configuration is being placed in setup.py instead of setup.cfg. Consult the appropriate documentation if your project is structured differently.

Contents

Avoiding General Mistakes
- Look at Your Built Distributions
- Installations are Not Namespaced
Top-Level tests/ Directory in Wheel
Top-Level README or LICENSE File in Wheel
Project Description Doesn’t Render
Project Description Not Included
Python Package Not Included in Wheel
Subpackages Not Included in Wheel
Package Data Not Included in Wheel
*.pyc Files Included in Wheel
Rebuilding Wheels without Deleting build/
Pinning Project Requirements to Exact Versions
Conclusion
Footnotes

Avoiding General Mistakes

First, some general advice that will help you avoid (or at least detect) the vast majority of packaging errors.

Look at Your Built Distributions

Before you upload your sdist and wheel (especially if it’s your first release for the project in question), actually take a look at what files are inside and make sure that everything you want to include is in there. An sdist is either a tarball (*.tar.gz) or a zipfile (*.zip), so its contents can be listed with one of these two commands:

# Tarball
$ tar ztf projectname-version.tar.gz

# Zipfile
$ zipinfo projectname-version.zip

At time of writing, the exact layout of an sdist has yet to be standardized, but if you’re building with a recent version of setuptools, the contents are structured as follows:

Everything in the sdist is inside a top-level directory named {projectname}-{version}/. This directory contains a copy of your package code, the project’s setup.py/pyproject.toml file, and various other files from your project directory; see here for more information on what gets included by default.
There exists a PKG-INFO file containing the project metadata. For historical reasons, this does not include project dependencies.
Next to your Python package, there is a {projectname}.egg-info/ directory containing more metadata, including a copy of PKG-INFO, a SOURCES.txt file listing the files in the sdist, and a requires.txt file listing your project’s dependencies. Other files may be present depending on what features your project uses.

If your sdist is missing some files from your project directory or contains some files that you don’t want in there, then (assuming you’re building your project with setuptools), you can adjust what gets included via a MANIFEST.in file and rebuild.

Note

Exactly what files should and should not be included in an sdist is, for the most part, largely a matter of opinion, but your sdist needs to include your Python code and anything from your project directory that’s needed to built a complete wheel. The files that setuptools includes by default should generally be left in there, and most people will recommend also including tests and documentation. Things that should generally be left out include *.pyc files, repository metadata like .gitignore and .hgtags, and (except in special circumstances) anything that you wouldn’t commit to version control.

Wheels (*.whl), meanwhile, are just zipfiles with a funny extension, so you can list their contents with zipinfo. The basic layout of a wheel is as follows:

Your Python package is located at the root of the wheel, rather than inside a directory.
There exists a {projectname}-{version}.dist-info/ directory containing metadata: a METADATA file describing the project (similar to an sdist’s PKG-INFO, but including dependencies), a WHEEL file describing the wheel version and tags, and a RECORD file listing the files in the wheel and their hashes. Licenses included in the wheel with the license_files setting also end up in this directory. Other files may also be present depending on what features your project uses and the versions of setuptools and wheel used to build the wheel.
If your project includes any files that are installed outside of site-packages — headers, scripts, data files (not to be confused with package data), etc. — they are stored in a {projectname}-{version}.data/ directory. Files in this directory are organized into subdirectories named after the distutils scheme keys (purelib, platlib, headers, scripts, or data) that map to the files’ install locations.

Note

Aside from the *.dist-info/ and *.data/ directories, a wheel should only contain Python packages & modules, consisting of code and data files. Files like your project’s setup.py, pyproject.toml, setup.cfg, etc. do not belong in a wheel.

Controlling what gets included in a wheel is more involved than for an sdist (when using setuptools, at least); consult documentation elsewhere on how to do this.

Warning

Do NOT try to fix your sdists or wheels by manually adding, removing, or editing the files inside, as this is likely to make the sdist/wheel invalid. Instead, change your project configuration and create new built distributions until you get what you want — and be sure to delete the build/ directory in between builds!

Installations are Not Namespaced

A key thing to understand about how Python packages are installed is that (almost) all of the files in a wheel are simply placed directly in site-packages/; the only subdirectories present will be the directories that are already in the wheel. This means that, if your wheel has a foo/ directory at the top level containing bar.py, then bar.py will be installed to site-packages/foo/bar.py; nothing is added to the path to separate it from other packages’ foo/bar.py files. Properly namespacing your files must be done by putting everything under a directory (normally your top-level Python package) with a name the same as or similar to the name of your project — which is the standard practice anyway. Where problems arise is when a top-level file or directory in a wheel has a name that other projects are also likely to use, in which case files end up overwritten with the wrong content and bugs result.

See pip issue #4625 for pip’s attempts at handling file collisions whenever they arise.

Top-Level `tests/` Directory in Wheel

The first (and probably most common) Python packaging mistake occurs when you put your tests in a tests/ directory at the root of your project (outside of your Python package) and then include this directory in your project’s wheels. The tests/ directory then ends up placed at the top-level of your wheel’s filesystem, and, as stated above, this means that it will be installed at site-packages/tests/. The problem comes from the fact that “tests/” is a name that everybody uses for their tests and too many other projects also include a top-level tests/ directory in their wheels. As a result, site-packages/tests/ becomes a mish-mash of code from different packages, some files even overwriting each other, and if someone tries to run those tests, chaos will ensue. (And if you don’t expect people to be running your tests from your installed project, why are you including tests in the wheel in the first place?)

The most common reason why tests/ ends up included in wheels is because the project’s author used find_packages() in their setup.py but didn’t use the function’s exclude argument. find_packages() works by searching for directories in the project root (or in the directory passed as the where argument) that contain an __init__.py file, and then it searches those directories for any subdirectories that also contain an __init__.py file, and so on until it runs out of directories with __init__.py files. Sometimes, people put an __init__.py file in tests/ (Whether this is necessary depends on the test framework being used), and so find_packages() with the default arguments picks it up and adds it to the project’s list of packages, resulting in it being included in the wheel.

To avoid this, you have five options:

Remove the __init__.py files from your tests/ directory and its subdirectories. Whether this is doable depends on your test framework.
Use find_packages()’s exclude argument to exclude tests/ and its subdirectories like so:
```
packages=find_packages(exclude=["tests", "tests.*"])
```
Note that we list both "tests" and "tests.*". Listing just "tests" would exclude tests/ but not its subdirectories, so we need to also list "tests.*" in order to exclude everything.
Use find_packages()’s include argument to include only your Python package and its subpackages like so:
```
packages=find_packages(include=["packagename", "packagename.*"])
```
As with exclude, we list both the package name and the package name followed by “.*” so that all subpackages of the package will be matched & included.
Move your tests/ directory inside your Python package directory so it’s no longer at the top level.
Switch your project to a src/ layout, where your Python package directory is located inside a directory named src/ and everything else — including tests/ — is outside of src/. With this layout, simply write your packages line as packages=find_packages("src"), and find_packages() will only look at what’s in src/.

Note that you will also need to add package_dir={"": "src"} to your setup() arguments in order for setuptools to grok your layout. More information about the src/ layout can be found here and here.

The second most common reason why tests/ ends up in wheels is that the project author used the exclude argument to find_packages() but listed only "tests" and not "tests.*", and so the subdirectories of tests/ (inside an otherwise-empty tests/ directory) ended up in the wheel. Both "tests" and "tests.*" need to be included in the exclude list in order to exclude the entire tests/ hierarchy.

Besides tests/, it is also a problem to include a top-level directory named test/ (singular), docs/, examples/, data/, or similar, as such directories are also often included in wheels despite the clashes that will result.

Do note that, when it comes to sdists, it’s perfectly fine to have a tests/ etc. directory at the base of your project, as sdists themselves are not installed, they’re just used to build wheels, which are what actually get installed.

Top-Level `README` or `LICENSE` File in Wheel

Similarly to the above mistake involving tests/, it is also a bad idea to include your project’s README.rst/README.md or LICENSE file (or CHANGELOG or really anything that’s not a Python module or *.pth file) at the root of your wheel, as it will collide with the READMEs and LICENSEs of other projects that do the same thing.

This mistake is particularly common among projects built using Poetry, where simple usage of the include option adds files directly into both the sdist and wheel. To include a file in only the sdist, one needs to change the include option from this form:

[tool.poetry]
include = ["CHANGELOG.md"]

to this form:

[tool.poetry]
include = [
    { path = "CHANGELOG.md", format = "sdist" }
]

If you do want to include your README or LICENSE in your wheel, the correct way is as follows:

For README, the file’s contents should already be used as the project’s (long) description, in which case the contents are already included in the project metadata, which is stored in PKG-INFO (for sdists) or *.dist-info/METADATA (for wheels), and thus there is no need to include the README as a separate file. If you need to be able to retrieve the README’s contents at runtime, this can be done by using importlib.metadata or similar to fetch the project’s description.
Licenses and related files belong inside a wheel’s *.dist-info directory. If using setuptools with wheel 0.32 or higher, licenses can be placed there by passing them to the [metadata]license_files option in setup.cfg; see the wheel documentation for more information.

At time of writing, Poetry does not support adding license files to a wheel’s *.dist-info directory, but PR #1367 would change that.

Project Description Doesn’t Render

The Python Package Index (PyPI) supports project (long) descriptions written in three possible formats: reStructuredText (the default if no format is specified), Markdown (either GitHub Flavored Markdown or CommonMark), and plain text. Markdown and plain text are lenient formats; anything you write in them is valid. However, documents written in reStructuredText can be malformed, producing errors & warning messages when rendered. When a project with a malformed reStructuredText description (either because it uses reStructuredText incorrectly or because it’s actually Markdown that wasn’t declared as Markdown) is uploaded to PyPI, PyPI does one of the following two things:

If the project does not declare a Content-Type for its malformed description, PyPI will fall back to displaying the source of the description as though it were plain text.
If the project explicitly declares the malformed description’s Content-Type as reStructuredText (i.e., as the MIME type text/x-rst), PyPI will reject the upload.

Neither situation is desirable, but at least the latter gives you the chance to correct your project description before it’s released on PyPI, while the former situation means your project’s PyPI page shows an ugly, unprofessional-looking description until you make a new release.

Note

When using setuptools, you may find that your project’s long description has been mangled somewhat, with a bunch of “Field: Value” entries added to the bottom and various information missing from the listing on the left side of the PyPI project page. This happens whenever you include a newline in your project’s summary/short description, thereby triggering setuptools bug #1390. Always make sure that no newlines end up passed to the description argument of setup()!

There are two things you can do to avoid uploading a project with a malformed description to PyPI:

Set your description’s Content-Type appropriately. If you’re using reStructuredText, this will cause PyPI to reject any uploads with malformed project descriptions. If you’re not using reStructuredText, setting the Content-Type is necessary in order for your description to be rendered properly.

The content types for the supported formats are as follows:

reStructuredText: text/x-rst

Markdown (GitHub Flavored Markdown): text/markdown or text/markdown; variant=GFM

Markdown (CommonMark): text/markdown; variant=CommonMark

Plain text: text/plain

If your project is built using setuptools, you set the description’s Content-Type by setting the long_description_content_type argument to setup() to the appropriate value from the above table. Note that this requires setuptools 36.4.0 or higher in order to work (or 38.3.0 or higher if you’re setting it in setup.cfg).
Run the twine check command from twine on your sdist and wheel before uploading them. This command checks whether your project description can be rendered on PyPI before you actually upload it.

reStructuredText:	`text/x-rst`
Markdown (GitHub Flavored Markdown):	`text/markdown` or `text/markdown; variant=GFM`
Markdown (CommonMark):	`text/markdown; variant=CommonMark`
Plain text:	`text/plain`

Project Description Not Included

It’s just embarrassing when this happens. A project without a long description just looks completely pointless; how am I supposed to know what it does or how to use it? Sadly, too many projects on PyPI lack long descriptions. Did the developer not care enough to write even a README? Did the developer forget to use the README as the long description or not know they had to?

If your project’s got a README — and really, a project that doesn’t have one isn’t ready to be released — and it’s written in reStructuredText, Markdown, or plain text (a safe bet), you can (and should) use it as your project’s long description by adding the following or similar to your setup.py:

with open("README.extension", encoding="utf-8") as fp:
    long_description = fp.read()

setup(
    ...
    long_description = long_description,
    ...
)

If your project isn’t in reStructuredText, you’ll also need to set long_description_content_type to the appropriate value in the table above so that the description renders properly on PyPI.

Python Package Not Included in Wheel

If not having a description is embarrassing, not having any code in your wheel is crippling. With a wheel like this, when people install your project, they get nothing! That’s certainly not what you want, is it?

Possible reasons why this can happen include:

You’re using find_packages() to autolocate your project’s packages, but you failed to add an __init__.py file to the top-level package (and possibly also some subpackages). Solution: Add that __init__.py.
- If your intention is to leave out the __init__.py file in order to create a namespace package, you’ll need to use find_namespace_packages() instead.
Your project’s code is a single Python module (as opposed to a directory of modules) and you’re using the packages argument to setup() and/or find_packages() in an attempt to declare the module to setuptools. This is wrong. When your project is a single Python module, instead of the packages argument, you need to use the py_modules argument. Set py_modules to a list of strings where each string is the name of a top-level Python module without the “.py” extension. (Usually, you’ll just have one module to list here.) You can’t use find_packages() for this.

If your project includes any tests (which it should), you can implicitly test that your wheel contains your project code by testing against the installed version of your project instead of the copy in your repository. To do this, pip-install your package (ideally in a virtualenv, and not in development/editable mode!) before running the tests and ensure that the directory containing the repository copy of your code is not in sys.path when the tests run. Tox can help with the first part. The second part depends in part on your test framework, but you can guarantee your tests aren’t picking up the local copy by switching to a src/ layout (see above). With these two things in place, your tests will be forced to import your package from site-packages, where it’s in a form determined by the contents of the project’s wheel. If your wheel is missing code and your tests try to import that code, you’ll get an error when the tests run, and you’ll know that you need to fix something.

Subpackages Not Included in Wheel

Sometimes, a project’s top-level package directory and the files within get included in a wheel, but the subdirectories and their contents get left out. Admittedly, I don’t know how common this is, as you can’t determine whether a wheel is missing subpackages just by looking at its contents unless you also know what’s in the project’s repository. However, it’s an easy thing to mess up, and various packaging articles I’ve read frequently make reference to this problem, so it can’t be that uncommon.

There are two major reasons why one or more of your Python package’s subpackages might be omitted from wheels:

You’re passing a list of packages to the packages argument to setup() and the list fails to include every package & subpackage in your project. If your project’s top-level package is named “foo” and it contains two subdirectories named “bar” and “baz” that contain (directly or indirectly) Python source files, then bar and baz are subpackages of foo, and they all need to be included in the packages list:
```
packages=["foo", "foo.bar", "foo.baz"]
```
If baz contains another directory named “glarch” that contains more Python source files, then "foo.baz.glarch" needs to be included in the list as well, and so on.

Note that directories that only contain data files and no Python source files do not count as packages and should not be passed to the packages argument. They are instead package data directories; see below for advice on dealing with them.

Of course, a simple alternative to listing every package explicitly is to just use the find_packages() function, which brings us to cause #2 …
You’re using find_packages() to autolocate your project’s packages, but you failed to add an __init__.py file to one or more subpackages. find_packages() only counts something as a package if it contains an __init__.py file, so you need to include that file in any subdirectory of your Python package that contains Python source files or contains a directory that contains Python source files.

As with omitting the package entirely from the wheel, proper testing practices can let you know when this happens in advance of a release.

Package Data Not Included in Wheel

Sometimes, you want to include non-Python data or resource files inside a Python package so that they can be used at runtime, but sometimes those files fail to end up in the final wheel. Like the omission of subpackages, it’s hard to know just how common this is, but even experienced Python programmers have made mistakes with package data configurations on occasion. This also happens to be yet another situation where testing the installed version of your code will help you out.

Setuptools provides two ways to specify package data. The first way is to configure MANIFEST.in so that the desired package data files are included in the sdist and then pass include_package_data=True to setup() so that all files inside the Python package that are included in the sdist are also included in the wheel. Pretty much the only way to make a mistake here is by not matching all of the files you want with MANIFEST.in commands; consult this reference if you run into problems.

The second way to specify package data is with the package_data argument to setup(). This argument takes a dict mapping package & subpackage names to lists of glob patterns defining what package data files to include in sdists & wheels. The biggest gotcha with this method is the fact that each glob pattern is only applied to the corresponding package and not any of its subpackages. This means that, with a package_data like this:

package_data={
    "package": ["*.txt"],
}

*.txt files in package will be recognized as package data and included in the sdist & wheel, but *.txt files in package.subpackage will not. To include *.txt files in package.subpackage, you’ll need to either add a "package.subpackage": ["*.txt"] entry to package_data or else include all *.txt files in all packages & subpackages by using the empty string as a key: "": ["*.txt"].

No matter which method you choose, be sure to exclude *.pyc files from consideration as package data; see the next section for why.

Note that if you combine the two ways to specify package data by setting include_package_data=True while also using package_data, then the files matched by package_data will not be included in the sdist unless they’re already included by MANIFEST.in. Getting this wrong can cause wheels built from an sdist to lack package data files.

See “Including Data Files” in the setuptools documentation for more information.

`*.pyc` Files Included in Wheel

When a Python source file is imported into a Python process, a *.pyc file containing compiled bytecode is created and (in Python 3) stored in a __pycache__/ directory so that future imports of the same file will be faster. These *.pyc files use a format that is specific to the OS, Python implementation, and Python version, and so it is pointless to share them. They do not belong in wheels (especially considering that pip already generates a host-appropriate set of *.pyc files when it installs a wheel), and yet too often people distribute wheels with *.pyc files in them.

Probably the most common reason why *.pyc files end up in wheels is that the project’s MANIFEST.in file contains “graft packagename”, “graft src”, or a similar line and include_package_data=True is passed to setup(). With this configuration, all files in the Python package directory when the wheel is built are added to the wheel. To prevent *.pyc files from being added, “global-exclude *.pyc” or similar needs to be added to the MANIFEST.in, ideally at the end of the file.

Alternatively, if the project specifies its package data with the package_data argument, including a "*" pattern in the package_data mapping is liable to cause *.pyc files to be included in the wheel. They should be excluded from package data by setting exclude_package_data to a dict that maps the appropriate keys to ["*.pyc"].

Rebuilding Wheels without Deleting `build/`

You should have noticed when building your project’s wheels that, in addition to creating a dist/ directory containing the output wheel, setuptools also creates a build/ directory containing a couple directories and a copy of your code. This build/ directory is an intermediate stage in the process of assembling a wheel; you should exclude it from version control and feel free to delete it at any time. In fact, it’s a good idea to delete it before running the command to create a wheel, especially if you’ve moved or renamed any files or directories in your code since the last time you built a wheel.

Consider the following scenario:

You build a wheel for your project, and you leave the build/ directory lying around afterwards.
You move, rename, and/or delete some files in your Python package, perhaps even renaming the package itself.
You build the wheel again — and when you do so, setuptools copies your new package tree into build/. Files that existed the last time the wheel was built overwrite their old copies in build/ successfully, but any old paths that have since been removed remain in build/.
As a result, your wheel ends up containing a mixture of your new and old code. In the case where you renamed your package, the wheel will contain both the pre-rename package and the post-rename package next to each other in their entirety, so you wheel has double the code with half of it under the wrong name.

This is clearly not desirable. The solution is to always delete the build/ directory before building a wheel, such as by cleaning your repository with git clean or similar, or by running python setup.py clean --all [1].

An even worse situation occurs if your setup.py uses find_namespace_packages() without any arguments. In this case, if you rebuild your package without first deleting the build/ directory, find_namespace_packages() will notice the .py files in build/ and assume that build/ is a namespace package, and so it’ll include build/ in your wheels — which means that build/ gets copied into build/, resulting in multiple package hierarchies in your wheels, with the problem compounding the more times you build your project without deleting the build/ directory. This particular problem can be mitigated by using the where, exclude, and/or include arguments to find_namespace_packages(), which have the same meaning as for find_packages().

Pinning Project Requirements to Exact Versions

There are a number of projects on PyPI where the dependencies are all of the form “foo == 1.2.3”, as opposed to “foo >= 1.2.3”, “foo >= 1.2, < 2”, or just “foo”. This is called pinning requirements. This makes sense when you’re developing a Python application that will be the primary project in its environment (in which case you often won’t be uploading it to PyPI), but it doesn’t make sense when you’re distributing a library for others to use alongside other arbitrary libraries. For one thing, your library is almost certainly going to work just as well with version 1.2.4 of foo [2], so why leave it out? For another thing, if someone wants to use your library with its pinned foo requirement alongside other libraries, sooner or later they’ll run into a situation where they’re installing both it and another project that requires a different version of foo (maybe even differing by one micro version!), and then problems ensue [3]. True, clashes between version dependencies in disparate projects can’t be avoided 100%, but they can be made to occur far less often if projects require generous version ranges instead of specific versions.

A general way to construct a decent version range for a requirement is to first determine the lowest version of the dependency that has all of the features you need and then use this version as the requirement’s lower bound. If the dependency follows or approximates semantic versioning, use the next major version (or the next minor version, if pre-v1) as the (exclusive) upper bound. If the dependency uses something like calendar versioning instead, things are less clear, but my preference is to leave out the upper bound and afterwards keep abreast of any future changes to the dependency. If any versions of the dependency inside the requirement’s bounds have known bugs that interfere with your project’s behavior, feel free to exclude them by adding specifiers of the form != X.Y.Z to the version range.

Conclusion

I’m very disappointed in all of you for making these mistakes so often, and I hope this article makes at least one Python package less broken. (I’d prefer it if all broken packages were less broken, but I know not to get my hopes up.)

Admittedly, most of these mistakes are due to users not using or understanding setuptools properly (aside from a Poetry anti-pattern that sneaked in at #2). Though flit and Poetry may promise to fix setuptools’ usability issues, people keep on using setuptools, and it keeps on outsmarting them. Hopefully sites like the Python Packaging User Guide eventually expand & become mature enough in the near future to cover — if not all the edge cases — at least the best practices that avoid them.

Footnotes

[1]	Setuptools is currently trying to get people to move away from `setup.py` commands, so `setup.py clean` will be discouraged — and probably deprecated — at some indeterminate point in the future. Until that happens, though, don’t feel bad about using it if you need to.

[2]	Unless `foo` is an unpredictable, compatibility-breaking mess, in which case you should probably reconsider depending on it.

[3]

Currently, pip handles conflicting version requirements with a warning and picking one requirement to follow, but pip’s new dependency resolver due out in October 2020 (already available if you pass the right flag to pip) will react to such situations by searching for older versions of the installation candidates with non-conflicting requirements, and if it can’t find any, it errors out without installing anything.

Setting Default Option Values from Config Files with Click

2020-07-17T00:00:00-04:00

When developing a command-line application in Python with Click, you may find yourself wanting to use a configuration file to set the parameters’ default values at runtime and to be able to specify such a file with a --config option. There are various libraries of varying quality that provide such functionality in different ways; however, implementing such functionality on one’s own is simple enough (as this document will show), and knowing the basics will help you develop more advanced implementations when the libraries out there don’t meet your needs.

All code in this document has been tested against Click version 7.1.2.

`default_map`

The key feature of Click that will allow us to set parameter values from a configuration file is the default_map attribute of click.Context, documented at [1]. Any values set in this dict before parameters are processed will become the parameters’ new default values. Any extra keys that do not correspond to defined parameters are ignored.

Moreover, Click will map values in default_map through the parameters’ normal type conversion & validation routines. This means that, if you have an option --foo defined with type=int, you can set ctx.default_map["foo"] = "5", and the value will be converted to an int by the time it’s passed to the command callback. This even works for boolean flags: a value set for a boolean flag option will be processed by the click.BOOL type, which maps boolean-like strings to bool values. This eases your workload when reading from a configuration file type like INI where values don’t come with type information; just pass the values straight to Click, and it’ll do the conversion for you the same as it does for values on the command line.

Areas to be careful in include parameters defined with multiple=True. The default value for such parameters (whether declared with default= in the parameter’s decorator or set in default_map) must be a list or tuple; setting such a default to a string will cause the string to be interpreted as a list of single-character strings. Also requiring special attention are options defined with nargs or with a tuple of types; correct handling for all of these is left as an exercise for the reader.

Implementing a `--config` option

In order to define a --config FILE option that reads from FILE and sets other parameters’ default values, the option first of all needs to be eager so that it can modify ctx.default_map before the other options read it, and it needs to be defined with a callback that does the actual work. Everything else after that is straightforward.

Here is a sample Python script with a --config option that reads from a given config file (or from config.ini in the current directory if no config file is given). The config file must be an INI file, and the values for the options are read from the [options] section. The command callback simply dumps out its arguments so you can see what’s being passed to it.

from   configparser import ConfigParser
import json
import click

DEFAULT_CFG = 'config.ini'

def configure(ctx, param, filename):
    cfg = ConfigParser()
    cfg.read(filename)
    try:
        options = dict(cfg['options'])
    except KeyError:
        options = {}
    ctx.default_map = options

@click.command()
@click.option(
    '-c', '--config',
    type         = click.Path(dir_okay=False),
    default      = DEFAULT_CFG,
    callback     = configure,
    is_eager     = True,
    expose_value = False,
    help         = 'Read option defaults from the specified INI file',
    show_default = True,
)
@click.option('--integer', type=int, default=42)
@click.option('--flag/--no-flag', default=False)
@click.option('--str', default='foo')
@click.option('--choice', type=click.Choice(['red', 'green', 'blue']))
def main(**kwargs):
    print(json.dumps(kwargs, sort_keys=True, indent=4))

if __name__ == '__main__':
    main()

If we run this script with no options when config.ini does not exist or is empty, we get the parameters’ built-in default values:

$ python3 config01.py
{
    "choice": null,
    "flag": false,
    "integer": 42,
    "str": "foo"
}

That’s boring! Try populating example.ini with the below text:

[options]
integer = 23
flag = yes
str = bar
choice = green

… and then run with --config example.ini:

$ python3 config01.py --config example.ini
{
    "choice": "green",
    "flag": true,
    "integer": 23,
    "str": "bar"
}

Note that the values set for the flag and integer options have been converted to their appropriate types.

Of course, options set in the config file are overridden by command-line options, no matter where the options occur in relation to --config:

$ python3 config01.py --integer 17 --config example.ini --str glarch
{
    "choice": "green",
    "flag": true,
    "integer": 17,
    "str": "glarch"
}

What if a value in the config file is invalid? Try saving the following text to bad.ini:

[options]
choice = mauve

The script will then error when passed this config file:

$ python3 config01.py --config bad.ini
Usage: config01.py [OPTIONS]
Try 'config01.py --help' for help.

Error: Invalid value for '--choice': invalid choice: mauve. (choose from red, green, blue)

Not the best possible error message (It doesn’t tell us the bad value was in the config file), but it’s better than a stack trace.

Note that, with this code, parameters in the config file must be named using the same name & spelling as the parameter’s corresponding argument to the command callback. For example, the --integer option must be written integer, not --integer or -i or i; any entries in the config file with an invalid spelling will be ignored. For options with medial hyphens on the command line, like --log-level, the hyphens must become underscores in the configuration file, like log_level. If you want to support the spelling log-level as well, insert the following line after cfg = ConfigParser() to make the ConfigParser object convert hyphens in option names to underscores:

cfg.optionxform = lambda s: s.replace('-', '_')

Configuring command groups

default_map supports passing values to subcommands in command groups in a very simple way: if the main command has a subcommand named “foo”, then ctx.default_map["foo"] can be set to a dict of parameter names & values for foo. For example, the following assignment:

ctx.default_map = {
    "color": "red",
    "foo": {
        "speed": "high",
    },
    "bar": {
        "speed": "low",
        "baz": {
            "time": "late",
        },
    },
}

sets the default value for the main command’s --color option to red, the default value of the foo subcommand’s --speed option to high, the default value of the bar subcommand’s --speed option to low, and the default value of the bar baz sub-subcommand’s --time option to late. As you can see, this comes with one major drawback: a command can’t have a subcommand with the same name as one of its parameters.

Here is a sample Python script with command groups that reads configuration from an INI file. Settings in the [options] section are applied to the top-level command, settings in the [options.CMD] section are applied to the subcommand CMD, settings in [options.CMD1.CMD2] are applied to the CMD2 sub-subcommand of the CMD1 subcommand, and so forth. As above, each command prints out the parameters it receives.

from   configparser import ConfigParser
import json
import click

DEFAULT_CFG = 'config.ini'

def configure(ctx, param, filename):
    cfg = ConfigParser()
    cfg.read(filename)
    ctx.default_map = {}
    for sect in cfg.sections():
        command_path = sect.split('.')
        if command_path[0] != 'options':
            continue
        defaults = ctx.default_map
        for cmdname in command_path[1:]:
            defaults = defaults.setdefault(cmdname, {})
        defaults.update(cfg[sect])

@click.group(invoke_without_command=True)
@click.option(
    '-c', '--config',
    type         = click.Path(dir_okay=False),
    default      = DEFAULT_CFG,
    callback     = configure,
    is_eager     = True,
    expose_value = False,
    help         = 'Read option defaults from the specified INI file',
    show_default = True,
)
@click.option('--integer', type=int, default=42)
@click.option('--flag/--no-flag', default=False)
@click.option('--str', default='foo')
@click.option('--choice', type=click.Choice(['red', 'green', 'blue']))
def main(**kwargs):
    print('* main')
    print(json.dumps(kwargs, sort_keys=True, indent=4))

@main.command()
@click.option('--speed', type=click.Choice(['low', 'medium', 'high', 'ludicrous']), default='medium')
def foo(**kwargs):
    print('* foo')
    print(json.dumps(kwargs, sort_keys=True, indent=4))

@main.group(invoke_without_command=True)
@click.option('--speed', type=click.Choice(['low', 'medium', 'high', 'ludicrous']), default='medium')
def bar(**kwargs):
    print('* bar')
    print(json.dumps(kwargs, sort_keys=True, indent=4))

@bar.command()
@click.option('--time', type=click.Choice(['early', 'late', 'exact']), default='early')
def baz(**kwargs):
    print('* baz')
    print(json.dumps(kwargs, sort_keys=True, indent=4))

if __name__ == '__main__':
    main()

Set config.ini to the following:

[options]
integer = 23
flag = yes
str = bar
choice = green

[options.foo]
speed = high

[options.bar]
speed = low

[options.bar.baz]
time = late

… and then invoke some commands to see the results:

$ python3 config02.py
* main
{
    "choice": "green",
    "flag": true,
    "integer": 23,
    "str": "bar"
}
$ python3 config02.py foo
* main
{
    "choice": "green",
    "flag": true,
    "integer": 23,
    "str": "bar"
}
* foo
{
    "speed": "high"
}
$ python3 config02.py bar
* main
{
    "choice": "green",
    "flag": true,
    "integer": 23,
    "str": "bar"
}
* bar
{
    "speed": "low"
}
$ python3 config02.py bar baz
* main
{
    "choice": "green",
    "flag": true,
    "integer": 23,
    "str": "bar"
}
* bar
{
    "speed": "low"
}
* baz
{
    "time": "late"
}
$ python3 config02.py --choice red foo --speed medium
* main
{
    "choice": "red",
    "flag": true,
    "integer": 23,
    "str": "bar"
}
* foo
{
    "speed": "medium"
}

Knowledge Bits - Python

Python Asynchronous Programming Fundamentals

Creating Multi-Value Enums in Python

Skipping Pytest Tests Unless an Option is Given

Option 1: Use a Hook to Attach a skip Marker to Marked Tests

Option 2: Use @pytest.mark.skipif

Option 3: Use a Pre-Existing Plugin

Notable Features Introduced in Each Python Version

Using Package Data in Python Projects with Setuptools

Common Python Packaging Mistakes

Setting Default Option Values from Config Files with Click

default_map

Implementing a --config option

Configuring command groups

Option 1: Use a Hook to Attach a `skip` Marker to Marked Tests

Option 2: Use `@pytest.mark.skipif`

`default_map`

Implementing a `--config` option