Knowledge Bits

Retrieving Logs for a Single Service Run from systemd

2026-05-10T00:00:00-04:00

systemd is the init system and service manager (and a bunch of other things) used by many major Linux distributions. As a service manager, it provides the ability to run programs (called services) in the background with features like restarting failed programs or running programs on a schedule. By default (i.e., unless a service’s unit file says to send it somewhere else), the output from these programs is stored in a journal that can be viewed with the journalctl(1) command. However, basic journalctl(1) invocations have the potential to swamp you with logs; sometimes you just want to see the output from a single run of a service or see a history of when a service was started & stopped. Here’s how you do that.

This article is based on systemd 255 on Ubuntu Noble 24.04. There may be some corner cases I have missed that make what I say here not 100% accurate.

Basic `journalctl(1)` Usage

By default, a given user only has permission to read journal entries for their own per-user systemd service manager. In order to read the system journal or other users’ journals, you must be root (possibly via sudo) or belong to the systemd-journal, adm, or wheel group.

Important

Journal entries do not last forever; you only have so much disk space, after all. Depending on configuration, the journal service will delete old entries if they reach a certain age or if the total size of the journal reaches a certain limit. You therefore might not be able to retrieve information from the journal about things that happened too long ago.

You can get a basic view of all the journal entries for a service by running journalctl -u $service, where $service is the name of the service with or without the “.service” suffix. Add the --user option if the service belongs to your per-user manager. The entries shown can be limited with the -n/--lines, --since, and --until options, among others; see the journalctl(1) manpage for more information.

By default, the output from journalctl is a chronological sequence of both systemd-generated messages (things like “Starting myservice.service”) and service-generated messages (such as standard output from the service commands). Each line consists of a timestamp, a hostname, a command name, a PID, and the actual message. The -o/--output option can be used to adjust the output format, though most formats only differ in how the timestamp is displayed.

Internally, each entry in systemd’s journal consists of a number of key-value pairs called fields that include data on the logged message, when it was emitted, and where it came from. Most of the fields are documented in systemd.journal-fields(7), and you can output all fields of journal entries by passing -o json (or -o json-pretty or another supported JSON format) or -o verbose to journalctl.

journalctl entries can be filtered based on field values by passing the command one or more match arguments of the form FIELDNAME=VALUE. Note that only exact string matches on values are supported; you cannot ask journalctl to filter, say, by a regex or by whether a field is set/unset; for that, postprocessing the output with jq(1) or another program is necessary. Multiple match arguments for the same field name are ORed together, while match arguments for different fields are ANDed. Using + as an argument causes the preceding match arguments to be ORed as a group against the following match arguments.

Systemd vs. Service Entries

When working with a service’s journal logs, there are two main types of entries that we usually want to discern between: entries emitted by systemd itself about the service — usually messages of the form “Starting myservice.service”, “Stopped myservice.service”, etc. — and entries that contain output from the service’s actual command(s).

Entries from systemd itself about a service are distinguished by having the INVOCATION_ID field (or USER_INVOCATION_ID for user services) set but not _SYSTEMD_INVOCATION_ID; the SYSLOG_IDENTIFIER field will also have a value of “systemd”. The UNIT field (or USER_UNIT for user services) will also be set to the name of the service in question (including template arguments and the “.service” suffix).

Caution!

Field names without leading underscores are not “trusted” journal fields, and thus any program that writes directly to the journal can create an entry with INVOCATION_ID, SYSLOG_IDENTIFIER, or UNIT set to whatever it wants. In order to exclude any potential deceptive entries when using journalctl to filter by one or more of these fields, include _PID=1 as a match argument so that you only see entries from systemd itself. Note that this only works when querying the system manager; for user managers, there does not seem to be a foolproof way to say “only show entries generated by the manager.”

Entries for actual output from a service’s commands (including child commands) are distinguished by having the _SYSTEMD_INVOCATION_ID field set, but not INVOCATION_ID or USER_INVOCATION_ID. The _SYSTEMD_UNIT field (or _SYSTEMD_USER_UNIT for user services) will also be set to the name of the service in question (including template arguments and the “.service” suffix).

Note that the latter kind of entries don’t just cover the service’s stdout & stderr; if the service command logged anything via syslog or wrote to the journal directly, that’s in here, too. You can separate out these sources by filtering on the _TRANSPORT field, which has a value of “stdout” for the command’s standard output & standard error (and for input to systemd-cat(1)), “syslog” for syslog messages, and “journal” for messages written directly to the journal with the sd_journal_print(3) APIs.

You may be wondering about the value of the SYSLOG_IDENTIFIER mentioned above when it comes to command output entries. As far as I can determine, when _TRANSPORT is “stdout” or “journal”, SYSLOG_IDENTIFIER is usually the filename of the executable that produced the message, except for messages produced with systemd-cat, which use the identifier specified on the command line with --identifier, leaving the field unset if the option is not given. For _TRANSPORT=syslog, SYSLOG_IDENTIFIER is the program name by default, but programs that write to syslog are free to use any string they want as their identifier.

Invocation IDs

As you may have noticed above, journal entries related to a service run all have an invocation ID associated with them. Each invocation ID is a random 32-character hexadecimal string that identifies a specific runtime cycle of a service (the period between when the service changes from “inactive” to “active”/”activating” and when it becomes inactive again). An invocation ID is the key information needed to get the logs for a single service run from journalctl.

If you want to view the logs for the most-recently started run of a service, begin by getting that run’s invocation ID via:

systemctl show --no-pager -P InvocationID $service

Replace $service with the name of the service in question (with or without the “.service” suffix). If querying a user session unit, add --user to the command. If the service has stopped and not been restarted, this command will nevertheless output the invocation ID for the most recent (stopped) run. If the given service does not exist or was never started, the command will output a blank line.

If the run you want logs for is not the most recent, you’ll need to determine the run’s invocation ID by looking back through the journal for the service. If you just know the approximate time at which the desired run occurred, you can browse the “Starting $service.service” and “Stopped $service.service” systemd messages with their timestamps & corresponding invocation IDs by using jq(1) and the following shell command:

journalctl -o json -t systemd -u $service | jq -r '"[\(.__REALTIME_TIMESTAMP|tonumber / 1000000 | todate)] [\(.INVOCATION_ID)] \(.MESSAGE)"'

Replace $service with the name of the service in question (with or without the “.service” suffix). If querying a user session unit, add --user to the journalctl command and change .INVOCATION_ID in the jq command to .USER_INVOCATION_ID.

This will give you output like the following, letting you map timestamp ranges to invocation IDs:

[2026-04-22T13:41:35Z] [439e459b1db44a97974eae5c133f4543] Starting apache2.service - The Apache HTTP Server...
[2026-04-22T13:41:36Z] [439e459b1db44a97974eae5c133f4543] Started apache2.service - The Apache HTTP Server.
[2026-04-23T00:00:00Z] [439e459b1db44a97974eae5c133f4543] Reloading apache2.service - The Apache HTTP Server...
[2026-04-23T00:00:00Z] [439e459b1db44a97974eae5c133f4543] Reloaded apache2.service - The Apache HTTP Server.
[2026-04-24T00:00:00Z] [439e459b1db44a97974eae5c133f4543] Reloading apache2.service - The Apache HTTP Server...
[2026-04-24T00:00:00Z] [439e459b1db44a97974eae5c133f4543] Reloaded apache2.service - The Apache HTTP Server.
[2026-04-26T00:00:00Z] [439e459b1db44a97974eae5c133f4543] Reloading apache2.service - The Apache HTTP Server...
[2026-04-26T00:00:01Z] [439e459b1db44a97974eae5c133f4543] Reloaded apache2.service - The Apache HTTP Server.
[2026-04-27T00:00:01Z] [439e459b1db44a97974eae5c133f4543] Reloading apache2.service - The Apache HTTP Server...
[2026-04-27T00:00:01Z] [439e459b1db44a97974eae5c133f4543] Reloaded apache2.service - The Apache HTTP Server.
[2026-05-07T16:19:39Z] [439e459b1db44a97974eae5c133f4543] Stopping apache2.service - The Apache HTTP Server...
[2026-05-07T16:19:40Z] [439e459b1db44a97974eae5c133f4543] apache2.service: Deactivated successfully.
[2026-05-07T16:19:40Z] [439e459b1db44a97974eae5c133f4543] Stopped apache2.service - The Apache HTTP Server.
[2026-05-07T16:19:40Z] [439e459b1db44a97974eae5c133f4543] apache2.service: Consumed 8min 45.667s CPU time, 111.8M memory peak, 22.5M memory swap peak.
[2026-05-07T16:19:40Z] [c80c7d5e0453447ea57bf833b3ead2cf] Starting apache2.service - The Apache HTTP Server...
[2026-05-07T16:19:40Z] [c80c7d5e0453447ea57bf833b3ead2cf] Started apache2.service - The Apache HTTP Server.
[2026-05-08T00:00:00Z] [c80c7d5e0453447ea57bf833b3ead2cf] Reloading apache2.service - The Apache HTTP Server...
[2026-05-08T00:00:01Z] [c80c7d5e0453447ea57bf833b3ead2cf] Reloaded apache2.service - The Apache HTTP Server.
[2026-05-09T00:00:01Z] [c80c7d5e0453447ea57bf833b3ead2cf] Reloading apache2.service - The Apache HTTP Server...
[2026-05-09T00:00:01Z] [c80c7d5e0453447ea57bf833b3ead2cf] Reloaded apache2.service - The Apache HTTP Server.
[2026-05-10T00:00:00Z] [c80c7d5e0453447ea57bf833b3ead2cf] Reloading apache2.service - The Apache HTTP Server...
[2026-05-10T00:00:00Z] [c80c7d5e0453447ea57bf833b3ead2cf] Reloaded apache2.service - The Apache HTTP Server.

If you don’t know the run’s timestamp but you can recognize the target run from the command output, you can view all log messages prefixed with their invocation IDs like so:

journalctl -o json -u $service | jq -r '"[\(.INVOCATION_ID // ._SYSTEMD_INVOCATION_ID)] \(.MESSAGE)"'

Getting Logs for an Invocation ID

Once you have the invocation ID for the service run you want to view logs for, you can get the service’s output with:

journalctl _SYSTEMD_INVOCATION_ID=$id

where $id is replaced by the invocation ID. Note that there is no need to specify the service name, nor even to specify --user for user session units.

Note that the output from this command will include fields like timestamp, hostname, command name, and PID in each line. If you just want only the actual output from the service with no “decorations,” add -o cat to the command. If you want custom formatting, your best bet is to output all the journal fields with -o json and process the entries with jq or another program.

If you also want to include systemd’s “Starting”, “Stopped”, etc. messages in the output (say, in order to see the command’s exit status), change the match arguments like so:

journalctl INVOCATION_ID=$id + _SYSTEMD_INVOCATION_ID=$id

If querying a user session unit, change INVOCATION_ID to USER_INVOCATION_ID.

Process Groups, Sessions, and Controlling Terminals

2025-06-30T00:00:00-04:00

In UNIX, job control is implemented via process groups and sessions, which allow a shell to place processes in the foreground (directly communicating with a terminal) or in the background and to send signals to multiple processes at once. This article provides an overview of the concepts and shows the basics of how to work with them.

Unless specified otherwise, all information in this article is as per POSIX.1-2024.

Concepts

Each UNIX process belongs to a process group (a.k.a. job).

Each process group is identified by an integer process group ID, and the process (if any) whose process ID equals this process group ID is called the process group leader.
It is possible to send a signal to all processes in a process group at once using kill() [POSIX] [man7] or killpg() [POSIX] [man7].

Each process group belongs to a session.

The process that created a session is the session leader. This process is also the process group leader of its process group.
POSIX does not specify IDs for sessions, though Linux uses the process (group) ID of the session leader as the session ID.

A session may be associated with at most one terminal called its controlling terminal, and each controlling terminal is associated with exactly one session.

The controlling terminal is established by the session leader, which then becomes known as the controlling process for as long as the terminal remains the controlling terminal. When the controlling process terminates, the session loses the controlling terminal, and any attempts by the remaining processes in the session to access the terminal may result in a SIGHUP signal.
It is possible for an individual process in a session to dissociate from the controlling terminal without affecting the rest of the session.
When a modem disconnect is detected for a controlling terminal, unless CLOCAL is set in the terminal’s c_cflag field, SIGHUP is sent to the terminal’s controlling process, which by default terminates it. Any further attempts to read from the terminal will return EOF.

Given a session associated with a controlling terminal, at most one process group in the session is the foreground process group, and all others are background process groups.

Processes in a foreground process group may read from & write to the controlling terminal. If a process in a background process group tries to read from the controlling terminal, the entire process group will normally [1] be sent a SITTIN signal, which by default stops & suspends the group. If a process in a background process group tries to write to a controlling terminal, the entire process group will normally [1] be sent a SIGTTOU signal, which by default stops & suspends the group; if TOSTOP is not set in the controlling terminal’s c_lflag field, the process will instead be allowed to write to the terminal, and no signal will be sent.

[1] (1, 2)
See the special cases listed under the “Terminal Access Control” section of the POSIX standard for when this is not the case.
Certain input key sequences like Ctrl-C, when entered at a controlling terminal, will cause a signal to be sent to all processes in the associated foreground process group.
Processes in the foreground process group are sent a SIGWINCH signal whenever the size of the controlling terminal changes.

Whenever a new process is created via fork() or similar, it starts out with the same session, process group, and controlling terminal as its parent. A process’s session, process group, and controlling terminal remain the same across a call to execve().

Process Groups at the Shell

In a POSIX-compatible shell, running a line composed of one or more AND-OR lists (commands containing zero or more of the operators !, |, &&, and/or ||) separated by semicolons creates a single foreground process group. If an AND-OR list in a line is terminated by & (making it an asynchronous AND-OR list), then everything before it in the line (up to the previous &, if any) is run in a single foreground process group, and the asynchronous AND-OR list itself is run as a single background process group; processing of the line then continues afterwards.

Examples:

# These commands are all run in a single foreground process group:
head bigfile.txt | grep foo && echo 'Those were the foos.'; rm bigfile.txt

# These commands are all run in a single background process group:
curl -fsSL -o download.html https://www.example.com || touch download-failed.txt &

# The `echo` is run in a foreground process group, while the `wget` is
# run in a background process group:
echo 'Going to download now'; wget --quiet https://www.example.com &

# The first `rm` and the `mkdir` are run in a foreground process group,
# then the `wget` is started in a background process group, then the second
# `rm` is started in a second background process group, and finally the
# `echo` is run in another foreground process group.
rm -rf download; mkdir downloads; \
    wget -qO downloads/example.html https://www.example.com & \
    rm -rf bigdir & \
    echo 'Am I done?'

A background process group created by a shell can be brought to the foreground with the fg command, and a foreground process group can be placed in the background by first stopping/suspending it with Ctrl-Z and then running bg.

Querying & Manipulating Process Groups & Sessions via System Calls

The process group ID of the current process can be retrieved with the getpgrp() [POSIX] [man7] function; the process group ID of an arbitrary process can be retrieved with the getpgid() [POSIX] [man7] function.

A process can change its process group or the process group of a child process via the setpgid() [POSIX] [man7] function; the target process group can be either a pre-existing group in the same session or a new process group that will be created in the same session.

The process group ID of a session leader cannot be changed. Thus, programs intending to create a new process group typically call fork() first and then call setpgid() from the child process in order to ensure that it’s not being called by a session leader.

The getsid() [POSIX] [man7] function can be used to retrieve the process group ID of the session leader (equal to Linux’s session ID) of a given process.

A new session can be created via the setsid() [POSIX] [man7] function, which makes the calling process into the new session’s session leader and into the process group leader of a new process group in the session; the calling process will have no controlling terminal afterwards.

setsid() cannot be called by a process group leader. Thus, programs intending to create a new session typically call fork() first and then call setsid() from the child process in order to ensure that it’s not being called by a process group leader.

The ctermid() [POSIX] [man7] function can be used to obtain the path to the controlling terminal for the current process; the GNU C Library implementation always returns "/dev/tty", which is a synonym for the controlling terminal on Linux (and macOS?).

Tip

If you really want the actual path to a process’s controlling terminal, and you don’t want to invoke ps(1) to get it, you can get partway there using Linux’s /proc filesystem: the seventh field of /proc/$PID/stat contains the device number for the controlling terminal of process $PID, or 0 if the process doesn’t have a controlling terminal. Unfortunately, there is no convenient way to map the device number to a path; cf. how ps does it.

Alternatively, you can approximate the controlling terminal for the current process with ttyname(STDIN_FILENO) or similar, but this won’t be accurate in the rare cases where stdin has been replaced with something other than the controlling terminal, possibly even a different, unrelated terminal.

POSIX does not specify a mechanism for setting the controlling terminal. On Linux and macOS, the controlling terminal for a session is established when a session leader first opens a terminal, unless the O_NOCTTY flag was passed to the open() call. Linux and macOS also support setting the controlling terminal via a session leader calling ioctl() with op set to TIOCSCTTY [man7], and any process may dissociate from its controlling terminal by calling ioctl() with op set to TIOCNOTTY [man7].

When a session gains a controlling terminal, the process group of the session leader becomes the foreground process group.
Note that a session gaining a controlling terminal will not cause any pre-existing processes in the session (other than the session leader) to gain a controlling terminal, but any processes spawned from the session leader afterwards will have a controlling terminal.

A process with a controlling terminal can acquire the process group ID of its session’s foreground process group by calling tcgetpgrp() [POSIX] [man7], and it can set the foreground process group by calling tcsetpgrp() [POSIX] [man7].

There does not appear to be any way to get a list of processes in a process group, a list of process groups in a session, or a list of extant sessions other than by iterating over /proc/*/stat files or using a facility that does that for you, like ps(1).

Creating a Background Process Without a Controlling Terminal (Daemonization)

In order to run a process as a daemon, running truly in the background, without a controlling terminal that could send SIGHUP on session exit, you could use a super-server like systemd or supervisord, but if you’re reading this, you probably want to know how they do it.

A program seeking to run itself or another executable as a daemon should take the following steps:

Call fork(). The rest of the steps are carried out in the resulting child process, which is guaranteed not to be a session leader or process group leader. The parent process can either exit immediately or else track the child process in order to detect & report any immediate unsuccessful terminations.
Call setsid() to create a new session, one not associated with any controlling terminal.
Close or redirect stdin, stdout, & stderr so that they no longer refer to the original terminal. It’s also recommended to set the current working directory to the root directory, as using a different working directory could prevent unmounting.
Call fork() again to create a child process that is not a session leader and thus cannot establish a controlling terminal. This child process is then used for the actual program proper (possibly via execve()), and the parent process exits.

Getting a Terminal’s Default Foreground & Background Colors

2025-06-10T00:00:00-04:00

When using ANSI escape sequences to style text on a terminal, you may need to know exactly what the default foreground & background colors — the ones set with \e[39m and \e[49m — are, as knowing whether the user’s terminal is light-on-dark or dark-on-light can help you pick an appropriate color theme. Fortunately, the escape sequences supported by xterm and compatible terminals include sequences for doing just that.

A program whose standard input & standard output are both connected to an xterm-compatible terminal can query the default foreground color by writing the characters \e]10;?\e\\\\ to standard output, where \e is the Escape character (0x1B) and \\\\ represents a single backslash; to query the default background color instead, change the 10 to an 11. The terminal will then respond by writing back the query string with the ? replaced by a string describing the default foreground or background color.

In my limited experience, all color strings I’ve seen in responses have been RGB values of the form rgb:XXXX/XXXX/XXXX, where the X’s are lowercase hexadecimal digits (so white would be rgb:ffff/ffff/ffff and black would be rgb:0000/0000/0000), but xterm’s documentation seems to suggest that color names (presumably ones from the X11 color list) and any RGB specifications accepted by XParseColor(3) are also possible return values.

Note

The \e\\\\ portion of the query & response (called the string terminator or ST) also has a legacy variant, a single BEL character (0x07), which you may see sometimes. xterm always responds using the same string terminator as used in the request.

Tip

By replacing the ? with an RGB specification, you can change the default foreground & background colors instead!

As with getting the current cursor position, the terminal will need to be set in cbreak and noecho modes when reading the response.

If you just want to see some code for doing all this on a Unix-like system, here it is as a Python script:

from __future__ import annotations
from collections.abc import Iterator
from contextlib import contextmanager
from copy import deepcopy
import re
import sys
import termios


def get_default_fg() -> str:
    """
    Query the attached terminal for the default foreground color and return the
    color string from the response

    :raises IOError: if stdin or stdout is not a terminal
    :raises ValueError: if the reply from the terminal is malformed
    """
    return osc_query(10)


def get_default_bg() -> str:
    """
    Query the attached terminal for the default background color and return the
    color string from the response

    :raises IOError: if stdin or stdout is not a terminal
    :raises ValueError: if the reply from the terminal is malformed
    """
    return osc_query(11)


def osc_query(ps: int) -> str:
    if sys.stdin.isatty() and sys.stdout.isatty():
        with cbreak_noecho():
            print(f"\x1b]{ps};?\x1b\\", end="", flush=True)
            resp = b""
            while not resp.endswith((b"\x1b\\", b"\x07")):
                resp += sys.stdin.buffer.read(1)
        s = resp.decode("utf-8", "surrogateescape")
        if m := re.fullmatch(rf"\x1B\]{ps};(.+)(?:\x1B\\|\x07)", s):
            return m[1]
        else:
            raise ValueError(s)
    else:
        raise IOError("not connected to a terminal")


# File descriptor for standard input:
STDIN = 0

# Indices into the tuple returned by `tcgetattr()`:
LFLAG = 3
CC = 6


@contextmanager
def cbreak_noecho() -> Iterator[None]:
    """
    A context manager that configures the terminal on standard input to use
    cbreak mode and to disable input echoing.  The original terminal
    configuration is restored on exit.
    """
    orig = termios.tcgetattr(STDIN)
    term = deepcopy(orig)
    term[LFLAG] &= ~(termios.ICANON | termios.ECHO)
    term[CC][termios.VMIN] = 1
    term[CC][termios.VTIME] = 0
    termios.tcsetattr(STDIN, termios.TCSANOW, term)
    try:
        yield
    finally:
        termios.tcsetattr(STDIN, termios.TCSANOW, orig)


if __name__ == "__main__":
    print("Foreground color:", get_default_fg())
    print("Background color:", get_default_bg())

Getting the Current Cursor Position from a Terminal

2025-05-29T00:00:00-04:00

When using ANSI escape sequences to manipulate a compatible terminal, you may need to know where the text cursor is currently located on the screen, especially if its current location was set by the user or a previous program rather than your code. Fortunately, the ANSI escape sequence standard ECMA-48 provides a sequence for doing just that.

A program whose standard input & standard output are both connected to an ANSI-compatible terminal can query the current cursor location by writing the characters \e[6n to standard output, where \e is the Escape character (0x1B). The terminal will then respond by writing back a string of the form \e[l;cR to standard input, where \e is the Escape character, l is a decimal integer giving the line number of the cursor’s current position, and c is a decimal integer giving the column number of the cursor’s current position. The line & column numbers are both 1-based: the upper-left corner of the screen is represented by (1, 1) rather than (0, 0).

Note that reading the terminal’s response isn’t as simple as it may seem at first. On Unix-like systems, standard input from the terminal is buffered by default and only made available to the running program when a newline is entered, but the terminal’s response doesn’t end in a newline, so typical line-reading functions won’t return until the user manually presses “Enter.” This can be solved by putting the terminal in cbreak mode, in which this buffering is disabled and characters sent to standard input can be read immediately. (Alternatively, the terminal may be put in raw mode instead, which is unbuffered like cbreak mode but also disables the special meanings of certain terminal-affecting key sequences like Ctrl-Q and Ctrl-S.)

In addition to disabling buffering, you’ll also want to disable echoing of input. By default, when the terminal sends its response to standard input, the characters will appear on the screen as if the user typed them, which is likely not what you want; turning off input echoing fixes this.

The full details of working with cbreak & noecho mode are a bit beyond the scope of this article. On Unix-like systems, you’ll need to use the tcgetattr() and tcsetattr() functions from <termios.h> or whatever wrapper around them your programming language of choice provides.

Tip

To avoid a race condition, set cbreak and noecho mode before printing the query.

Important

Be sure to set the terminal’s cbreak and echo settings back to what they were originally when you’re done reading the response!

Finally, if you just want to see some code for doing all this on a Unix-like system, here it is as a Python script:

from __future__ import annotations
from collections.abc import Iterator
from contextlib import contextmanager
from copy import deepcopy
import re
import sys
import termios


def cursor_pos() -> tuple[int, int]:
    """
    Query the attached terminal for the current cursor position and return
    the result as a ``(line, column)`` pair.

    :raises IOError: if stdin or stdout is not a terminal
    :raises ValueError: if the reply from the terminal is malformed
    """
    if sys.stdin.isatty() and sys.stdout.isatty():
        with cbreak_noecho():
            print("\x1b[6n", end="", flush=True)
            resp = b""
            while not resp.endswith(b"R"):
                resp += sys.stdin.buffer.read(1)
        s = resp.decode("utf-8", "surrogateescape")
        if m := re.fullmatch(r"\x1B\[(?P<line>[0-9]+);(?P<col>[0-9]+)R", s):
            return (int(m["line"]), int(m["col"]))
        else:
            raise ValueError(s)
    else:
        raise IOError("not connected to a terminal")


# File descriptor for standard input:
STDIN = 0

# Indices into the tuple returned by `tcgetattr()`:
LFLAG = 3
CC = 6


@contextmanager
def cbreak_noecho() -> Iterator[None]:
    """
    A context manager that configures the terminal on standard input to use
    cbreak mode and to disable input echoing.  The original terminal
    configuration is restored on exit.
    """
    orig = termios.tcgetattr(STDIN)
    term = deepcopy(orig)
    term[LFLAG] &= ~(termios.ICANON | termios.ECHO)
    term[CC][termios.VMIN] = 1
    term[CC][termios.VTIME] = 0
    termios.tcsetattr(STDIN, termios.TCSANOW, term)
    try:
        yield
    finally:
        termios.tcsetattr(STDIN, termios.TCSANOW, orig)


if __name__ == "__main__":
    print(cursor_pos())

The BitTorrent Encryption Protocol

2023-10-19T00:00:00-04:00

The encryption protocol used by the BitTorrent peer-to-peer file-sharing protocol — known variously as Message Stream Encryption (MSE), Protocol Encryption (PE), or MSE/PE, among other names — is what keeps you secure while you download massive amounts of … Linux ISOs. While some (including the creator of BitTorrent) have been critical of the protocol, comparing it to mere “obfuscation,” it remains highly popular among users distributing perfectly legitimate files over the internet that ISPs shouldn’t concern themselves over.

The de facto specification for the protocol appears to be <https://wiki.vuze.com/w/Message_Stream_Encryption>, but the page has been down whenever I’ve checked recently, and the actual content is not optimally presented. Thus, I’ve written up everything I’ve been able to determine about MSE/PE here.

Contents

Overview
Definitions
Client’s Handshake
Server’s Handshake
After the Handshake
BitTorrent-Specific Aspects
- Detecting Encryption
- Broadcasting Encryption Support
References

Overview

An MSE/PE connection begins with the client and server performing a Diffie–Hellman key exchange handshake, in which each party generates a private key with corresponding public key, the public keys are exchanged, and then — by the power of mathematics — both parties independently calculate the same number (the shared secret), which remains unknown to eavesdroppers. Each party then uses the shared secret to initialize two keystreams: infinite generators of pseudo-random bytes calculated from an initial value; the bytes from one keystream are then used to encrypt & decrypt data sent from the client to the server, and the other keystream is used for data sent from the server to the client.

After the handshake, the parties transmit their data, which is encrypted using a method negotiated during the handshake; the only defined options are to keep using the keystreams or to stop using encryption entirely.

Definitions

MSE/PE encryption uses the following constants and functions:

P is the following 96-byte (768-bit) prime, here rendered in hexadecimal:

ffffffffffffffff c90fdaa22168c234  c4c6628b80dc1cd1 29024e088a67cc74
020bbea63b139b22 514a08798e3404dd  ef9519b3cd3a431b 302b0a6df25f1437
4fe1356d6d51c245 e485b576625e7ec6  f44c42e9a63a3621 0000000000090563

G is 2 (a primitive root modulo P).
SKEY is a shared key — a byte string with a value known or recognizable by both parties. For BitTorrent, this is the 20-byte info hash of the torrent the connection is dedicated to.
len(X) is the length of the byte string X as a two-byte big-endian integer. The length of X cannot exceed 65535.
HASH(X) is the 20-byte SHA1 hash of the byte string X.

Client’s Handshake

The peer initiating an MSE/PE connection performs its side of the handshake as follows as soon as it opens the connection:

The client generates a random integer Xa to use as its private key for this connection only. Xa must be at least 128 bits long; using more than 180 bits is not believed to add further security. 160 bits is recommended.
The client calculates its public key: Ya = pow(G, Xa) mod P.
The client sends packet 1: Ya encoded as a 96-byte (768-bit) big-endian integer, followed by a string of random bytes with a randomly-chosen length from 0 to 512.
The client receives packet 2. Because the length of this packet is not known by the client and there is no means for indicating the end of the packet, the client must identify the end of the packet by waiting for sufficient time (the original spec says 30 seconds) for the entire packet to arrive. If the server sends fewer than 96 bytes in that time (For comparison, the unencrypted BitTorrent handshake message is 68 bytes), or the server sends more than 608 bytes, then the server is not performing a valid MSE/PE handshake.
The client extracts Yb from the first 96 bytes of packet 2 and computes the Diffie-Hellman shared secret: S = pow(Yb, Xa) mod P. Xa is then securely discarded.
- When used as a byte string in later steps, S is encoded as a 96-byte (768-bit) big-endian integer.
The client initializes an outgoing RC4 keystream from the key HASH("keyA" + S + SKEY) and an incoming RC4 keystream from the key HASH("keyB" + S + SKEY). The first 1024 bytes output by each keystream are discarded.
The client sends packet 3, constructed as follows:
```
HASH("req1" + S)
+ (HASH("req2" + SKEY) XOR HASH("req3" + S))
+ RC4(VC + crypto_provide + len(PadC) + PadC + len(IA) + IA)
```
where:
- RC4(X) encrypts the byte string X by XORing it with bytes produced by the outgoing RC4 keystream.
- VC is a verification constant of eight zero-valued bytes that is used to verify whether the other party knows S and SKEY and thus defeat replay attacks of the SKEY hash.
- crypto_provide is a 32-bit big-endian integer in which individual bits are set to indicate the encryption methods supported by the client. The defined methods are plain text (bit 0x01) and RC4 (bit 0x02).
- PadC is arbitrary data with a length of 0 to 512 bytes, reserved for extending the crypto handshake in future versions. Current implementations may choose to set it to the empty string. For padding-only usage in the current version, the bytes should be zeroed.
- IA is the initial (post-handshake) data that the client wishes to send. It may be 0 bytes long, and it cannot be more than 65535 bytes long.
The client receives & verifies packet 4:
- The first eight bytes are received and decrypted by XORing them with bytes from the incoming RC4 keystream. If the resulting bytes are not all zero, the handshake is invalid.
- The next four bytes are received and decrypted in the same manner to obtain crypto_select, a big-endian integer in which a bit corresponding to one of the methods given in crypto_provide has been set in order to indicate which encryption method will be used after the handshake. If crypto_select does not have exactly one bit set, or if the set bit does not correspond to one of the methods in crypto_provide, the handshake is invalid.
- The next two bytes are received and decrypted to obtain len(PadD).
- The next len(PadD) bytes are received and decrypted to obtain PadD. Note that, although the result of the decryption is unused, the decryption must still be performed in order to keep the incoming keystream in sync with the server’s outgoing keystream.

Server’s Handshake

The peer receiving an MSE/PE connection performs its side of the handshake as follows as soon as it accepts the connection:

The server generates a private key Xb following the same rules as for the client’s private key.
The server calculates its public key: Yb = pow(G, Xb) mod P.
The server sends packet 2: Yb encoded as a 96-byte (768-bit) big-endian integer, followed by a string of random bytes with a randomly-chosen length from 0 to 512.
The server receives packet 1. As with the client’s receipt of packet 2, the server must determine the end of packet 1 by waiting for sufficient time (the original spec says 30 seconds) for the entire packet to arrive.
The server extracts Ya from the first 96 bytes of packet 1 and computes the Diffie-Hellman shared secret: S = pow(Ya, Xb) mod P (This is equal to the S computed by the client). Xb is then securely discarded.
- When used as a byte string in later steps, S is encoded as a 96-byte (768-bit) big-endian integer.
The server receives & verifies packet 3:
- The first 20 bytes must equal HASH("req1" + S).
- The next 20 bytes are received and XORed with HASH("req3" + S) to obtain HASH("req2" + SKEY). The server then identifies SKEY by comparing this hash against HASH("req2" + K) for all known/accepted shared keys K. (For BitTorrent, this means comparing against HASH("req2" + info_hash) for all info hashes of torrents managed by the server.)
- The server initializes an outgoing RC4 keystream from the key HASH("keyB" + S + SKEY) and an incoming RC4 keystream from the key HASH("keyA" + S + SKEY). (Note that this is the reverse of the client.) The first 1024 bytes output by each keystream are discarded.
- The next eight bytes are received and decrypted by XORing them with bytes from the incoming RC4 keystream. If the resulting bytes are not all zero, the handshake is invalid.
- The next four bytes are received and decrypted in the same manner to obtain crypto_provide.
- The next two bytes are received and decrypted to obtain len(PadC).
- The next len(PadC) bytes are received and decrypted to obtain PadC. Note that, although the result of the decryption is unused, the decryption must still be performed in order to keep the incoming keystream in sync with the client’s outgoing keystream.
- The next two bytes are received and decrypted to obtain len(IA).
- The next len(IA) bytes are received and decrypted to obtain IA, the beginning of the actual data being transferred.
The server chooses one of the encryption methods given by crypto_provide to use for the rest of the connection. Bits with unknown meanings are ignored. If crypto_provide does not contain any encryption methods that the server supports, the handshake fails.
The server sends packet 4: RC4(VC + crypto_select + len(PadD) + PadD), where:
- RC4(X) encrypts the byte string X by XORing it with bytes produced by the server’s outgoing RC4 keystream.
- VC is eight zero-valued bytes, the same as in packet 3.
- crypto_select is a 32-bit big-endian integer with one bit set to indicate the encryption method chosen by the server. The bits have the same meanings as for crypto_provide.
- PadD is arbitrary data with a length of 0 to 512 bytes, reserved for extending the crypto handshake in future versions. Current implementations may choose to set them to 0-length. For padding-only usage in the current version, they should be zeroed.

After the Handshake

Once all of the above is carried out, the MSE/PE handshake is complete, and the client & server transmit the data they came here to transmit, encrypted using the selected encryption method. If plain text was selected, further data is sent & received as-is without encryption. If RC4 was selected, sent & received data will be encrypted using the sender’s outgoing RC4 keystream and decrypted using the receiver’s incoming RC4 keystream; these are the same keystreams as used during the handshake, i.e., they are not reinitialized.

BitTorrent-Specific Aspects

Detecting Encryption

MSE/PE was introduced into a world where BitTorrent connections were already being made without encryption, and many connections still aren’t encrypted, so peers need a way to determine whether a fresh connection is encrypted or not.

When a BitTorrent peer that supports MSE/PE receives an incoming connection, it can determine whether an MSE/PE handshake is being performed by checking whether the first 20 bytes received equal the BitTorrent handshake header "\x13BitTorrent protocol"; if the bytes match, the connection is (almost certainly) not using MSE/PE, and the peer can choose to either continue the connection unencrypted or else sever the connection.

When a BitTorrent peer that supports MSE/PE makes an outgoing connection, it has the following options, which it chooses among based on its configuration and any broadcasts of encryption support it’s received (see below):

The peer can attempt an MSE/PE handshake; if that fails, it abandons the remote peer.
The peer can attempt an MSE/PE handshake; if that fails, it severs the connection and tries to reconnect without using encryption.
The peer connects without using encryption. If the remote peer sends a BEP 10 extended handshake containing an e value of 1, the local peer severs the connection and reconnects using MSE/PE.
The peer connects without using encryption and does not use MSE/PE with the remote peer.

Broadcasting Encryption Support

BitTorrent peers can broadcast their support of MSE/PE to other peers via HTTP trackers and/or Peer Exchange. (The UDP tracker protocol and DHT do not appear to have any capabilities for broadcasting encryption support.)

When a BitTorrent peer that supports MSE/PE makes an announcement to an HTTP tracker, it can include one or more of the following URL query parameters:

supportcrypto=1 — Indicates that the peer can create & receive MSE/PE connections
requirecrypto=1 — Indicates that the peer only creates & accepts MSE/PE connections. If the tracker supports this parameter, then this peer will not be returned in responses to peers that do not set supportcrypto=1 or requirecrypto=1.
cryptoport=X (used in combination with port=0 and requirecrypto=1) — If the tracker supports the cryptoport parameter, it will provide this peer’s port as X in responses to other peers that also support MSE/PE and will not provide this peer at all to peers that do not support MSE/PE. If the tracker does not support the cryptoport parameter, then this peer’s actual port will not be given out to any peers.

When supportcrypto=1 or requirecrypto=1 is set in an announcement to a supporting HTTP tracker, the response will include a crypto_flags field, the value of which is a sequence of bytes, one for each peer in peers in order; a given byte will be 1 if the peer requires MSE/PE and 0 otherwise.

Preliminary searching on GitHub indicates that, when an HTTP tracker sends a response with “peers”, “peers6”, and “crypto_flags” fields, the “crypto_flags” only applies to the “peers” field and not “peers6”, though I have yet to encounter a tracker that actually sends “crypto_flags” in the wild.

If a peer prefers MSE/PE connections to unencrypted, it can indicate this to connecting peers by including an e field with a value of 1 in the BEP 10 extended handshakes it sends. This e value will then be broadcast to other peers using Peer Exchange (BEP 11).

References

Converting Lists to Simple Tables and Back with rs

2023-05-03T00:00:00-04:00

If you have a file that’s just a list of items, one item per line, and you want to turn it into a simple table — one delimited just by whitespace without any border characters — how would you do that? Follow up question: If you have a simple table, and you want to convert it into a list of items, one per line, how would you do that? The answer to both questions is: with the rs command, and this article will show you how.

rs is a Unix command-line program for reshaping arrays of data. Most Linux users can get it by installing the rs package, and macOS users will find it preinstalled.

Converting a List of Lines to a Simple Table

When rs is invoked without any options, it constructs a table from standard input, treating each line as its own table entry, and fills in columns before rows. Thus, if we have the following file saved in lines.txt:

Dapper Drake
Edgy Eft
Feisty Fawn
Gutsy Gibbon
Hardy Heron
Intrepid Ibex
Jaunty Jackalope
Karmic Koala
Lucid Lynx
Maverick Meerkat
Natty Narwhal
Oneiric Ocelot
Precise Pangolin
Quantal Quetzal
Raring Ringtail
Saucy Salamander
Trusty Tahr
Utopic Unicorn
Vivid Vervet
Wily Werewolf
Xenial Xerus
Yakkety Yak
Zesty Zapus

Then running rs < lines.txt will produce the following output:

Dapper Drake      Jaunty Jackalope  Precise Pangolin  Vivid Vervet
Edgy Eft          Karmic Koala      Quantal Quetzal   Wily Werewolf
Feisty Fawn       Lucid Lynx        Raring Ringtail   Xenial Xerus
Gutsy Gibbon      Maverick Meerkat  Saucy Salamander  Yakkety Yak
Hardy Heron       Natty Narwhal     Trusty Tahr       Zesty Zapus
Intrepid Ibex     Oneiric Ocelot    Utopic Unicorn

Important

When rs is run without any arguments, it acts like it was given the -e (Read one table entry from each line of input) and -t (Fill in columns before rows) options. If you specify any arguments of your own, then -e and -t will not be in effect unless you specify them explicitly.

rs automatically determines the number of table columns based on what will fit in an 80-character line. To change the line length to, say, 60 characters, pass -w60 to the command (No space is allowed between the option and its argument). To ask for a specific number of table rows and let rs figure out the number of columns, supply the desired number of rows as an argument, e.g., rs -e -t 8 < lines.txt. To ask for a specific number of table columns instead, pass a 0 argument (to tell rs “Figure the number of rows out yourself”) followed by the desired number of columns, e.g., rs -e -t 0 3 < lines.txt.

By default, rs will output every column at the same width; in the example above, this results in the first column having more padding on the right than is strictly necessary. The -z option will get rid of that padding; thus, by running rs -e -t -z < lines.txt, we get the following output:

Dapper Drake   Jaunty Jackalope  Precise Pangolin  Vivid Vervet
Edgy Eft       Karmic Koala      Quantal Quetzal   Wily Werewolf
Feisty Fawn    Lucid Lynx        Raring Ringtail   Xenial Xerus
Gutsy Gibbon   Maverick Meerkat  Saucy Salamander  Yakkety Yak
Hardy Heron    Natty Narwhal     Trusty Tahr       Zesty Zapus
Intrepid Ibex  Oneiric Ocelot    Utopic Unicorn

One thing you might not have noticed about the output is that the last line, having fewer columns than the others, ends in several spaces to make the end line up with the start of the last column. rs doesn’t have an option to disable this extra whitespace, but if you really want to get rid of it, you can pipe the output through sed -e 's/ *$//' to strip all trailing spaces.

Lastly, if you want the rows of the table to be filled in before the columns, invoke rs with the -e option and no -t option; thus, rs -e < lines.txt gives us:

Dapper Drake      Edgy Eft          Feisty Fawn       Gutsy Gibbon
Hardy Heron       Intrepid Ibex     Jaunty Jackalope  Karmic Koala
Lucid Lynx        Maverick Meerkat  Natty Narwhal     Oneiric Ocelot
Precise Pangolin  Quantal Quetzal   Raring Ringtail   Saucy Salamander
Trusty Tahr       Utopic Unicorn    Vivid Vervet      Wily Werewolf
Xenial Xerus      Yakkety Yak       Zesty Zapus

Converting a Simple Table to a List of Lines

Say we have the following text saved in table.txt:

Dapper Drake      Jaunty Jackalope  Precise Pangolin  Vivid Vervet
Edgy Eft          Karmic Koala      Quantal Quetzal   Wily Werewolf
Feisty Fawn       Lucid Lynx        Raring Ringtail   Xenial Xerus
Gutsy Gibbon      Maverick Meerkat  Saucy Salamander  Yakkety Yak
Hardy Heron       Natty Narwhal     Trusty Tahr       Zesty Zapus
Intrepid Ibex     Oneiric Ocelot    Utopic Unicorn    Artful Aardvark

and we want to convert this into a list of table entries, one per line, column by column (i.e., we want Intrepid Ibex to come before Jaunty Jackalope in the output). Because the columns are separated by multiple spaces and the table entries also contain spaces, we can’t use rs by itself to accomplish this. Instead, we first use sed -e 's/ \{2,\}/\t/g' to convert runs of two or more spaces to tab characters, and then rs can process the table using tabs as the cell delimiter.

Next, we have to pipe the output of sed through two rs invocations: the first invocation transposes the table so that the second invocation will split its entries apart in the correct order (For whatever reason, rs can’t do all this in a single invocation). The full command line will look like this:

sed -e 's/ \{2,\}/\t/g' table.txt | rs -c -C -T | rs -c 0 1

(The -c and -C options tell rs to use a tab as the input and output column delimiters, respectively.)

This works fine for the table above, but if the last column is short by two or more columns, like so:

Dapper Drake      Jaunty Jackalope  Precise Pangolin  Vivid Vervet
Edgy Eft          Karmic Koala      Quantal Quetzal   Wily Werewolf
Feisty Fawn       Lucid Lynx        Raring Ringtail   Xenial Xerus
Gutsy Gibbon      Maverick Meerkat  Saucy Salamander  Yakkety Yak
Hardy Heron       Natty Narwhal     Trusty Tahr
Intrepid Ibex     Oneiric Ocelot    Utopic Unicorn

then rs will not do what we want by default. When the rs -c -C -T step goes to transpose its input, it finds that a direct transposition would result in the last two columns of the output being short, which rs dislikes, and so it decides to fix things by moving “Intrepid Ibex” (the next cell after “Trusty Tahr”) to the cell below “Trusty Tahr” and right of “Yakkety Yak” in the output, producing this (with tabs converted to aligned spaces to make it easier to read):

Dapper Drake      Edgy Eft          Feisty Fawn       Gutsy Gibbon      Hardy Heron       Oneiric Ocelot
Jaunty Jackalope  Karmic Koala      Lucid Lynx        Maverick Meerkat  Natty Narwhal     Utopic Unicorn
Precise Pangolin  Quantal Quetzal   Raring Ringtail   Saucy Salamander  Trusty Tahr
Vivid Vervet      Wily Werewolf     Xenial Xerus      Yakkety Yak       Intrepid Ibex

and then when this table is flattened with rs -c 0 1, the output will be in the wrong order.

We can tell rs to not do this by passing the -n option to the rs -c -C -T command, which will make it output an all-blank entry below “Trusty” and right of “Yakkety” in the output instead of moving “Intrepid.” However, this blank cell will result in the final rs -c 0 1 command printing blank lines at the end of the output; we can avoid this by stripping trailing tabs before passing the table on. Hence, the final command line, that works regardless of how short the last column happens to be, is:

sed -e 's/ \{2,\}/\t/g' table.txt | rs -c -C -T -n | sed -e 's/\t*$//' | rs -c 0 1

On the other hand, if the table we’re converting to a list that happens to be read by going across the first row, then the second row, etc. — e.g., if it was produced with rs -e without the -t option — instead of being read by going down the first, second, etc. column, then the whole business with rs -c -C -T -n and stripping trailing tabs is unnecessary, and we can turn this table:

Dapper Drake      Edgy Eft          Feisty Fawn       Gutsy Gibbon
Hardy Heron       Intrepid Ibex     Jaunty Jackalope  Karmic Koala
Lucid Lynx        Maverick Meerkat  Natty Narwhal     Oneiric Ocelot
Precise Pangolin  Quantal Quetzal   Raring Ringtail   Saucy Salamander
Trusty Tahr       Utopic Unicorn    Vivid Vervet      Wily Werewolf
Xenial Xerus      Yakkety Yak

into a list by just running:

sed -e 's/ \{2,\}/\t/g' table.txt | rs -c 0 1

Basics of Writing Unix Man Pages

2022-11-24T00:00:00-05:00

On Unix-like systems, the documents that one views with the man command (known as “man pages”) are written in a markup language called “roff” (short for “run off”) that dates back to the pre-Unix days in the 1960’s. roff is technically a full-blown programming language with numerous features, but you only need to know a very small subset in order to produce most man pages. This article will describe that subset so that you can go on to write man pages for your own software.

Man pages, like most roff documents, use a roff macro package that defines a set of high-level commands. The macro package used by most man pages is, unsurprisingly, the man package, and that is what is covered here. Some man pages instead use the mdoc package, which originated in BSD. Modern versions of the man command use the mandoc package for processing man pages, which autodetects whether a file is written using man or mdoc and processes it accordingly.

Contents

roff Syntax
Document Structure Commands
Font Commands
Additional Escape Sequences
Man Page Conventions
Rendering a Man Page
A Sample Man Page
Further References

roff Syntax

A roff file is composed of a mixture of control lines — lines that start with a control character, usually a period or single-quote — and text lines — lines that do not.

Control lines are commands to roff (also known as requests). They consist of a control character & a command name followed by some number of space-separated (not tab-separated!) arguments. To include spaces in an argument, either escape them with a backslash or enclose the entire argument in double-quotes; to use double-quotes inside an argument, write them as \(dq.

For example, the input:

.B Bold
and brash?  More like
.IR Belongs " in the " Trash.

contains the command .B (for making text bold) with the argument “Bold” and the command .IR (for alternating between italic and roman text) with the arguments “Belongs” (which will be italic), “ in the ” (which will be roman), and “Trash” (which will be italic). This is all rendered as:

Bold and brash? More like Belongs in the Trash.

There may be any number of spaces & tabs (or none at all) between the control character and the command name, but the control character must be the first character in the line.
A line with just a period is ignored.

Text lines give the text that will be displayed. They can contain escape sequences, inline commands that start with backslashes.

Whitespace around escape sequences is significant and is not discarded. The syntax of an escape sequence allows roff to automatically determine where it ends; for example, escape sequences starting with \( are always followed by two more characters to complete the sequence, and so an escape sequence like \(em (em-dash) can be embedded in the middle of a word: foo\(embar becomes “foo—bar”.
To start a text line with a period, precede the period with either a backslash or the escape sequence \&.
To render a literal backslash in text, use the escape sequence \\\\, \e, or \(rs.

For example, the input from above can be rewritten using escape sequences like so:

\fBBold\fR and brash?  More like \fIBelongs\fR in the \fITrash.\fR

A comment consists of a backslash and double quote (\") and extends to the end of the line. A full-line comment can be formed by using the command .\".

A single logical line can be broken across multiple physical lines by placing a backslash at the end of each physical line.

Document Structure Commands

.TH title section [footer-middle] [footer-outside] [header-middle]

This must be the first non-comment command in a man page, and it must appear exactly once. It sets the page title (which is conventionally in all-caps) and section number for the man page (see “Manual Sections” below) to the given values, to be displayed in the form title(section) at the left & right sides of the header of the rendered manual page.

The remaining three arguments are optional. To omit an argument but still be able to specify an argument after it, write \& in place of the omitted argument.

If footer-middle is given, it will be rendered in the middle of the footer. This argument is usually set to the date the man page was written.
If footer-outside is given, it will be rendered on the left side of the footer. This is usually set to the version of the software being documented.
If header-middle is given, it will be rendered in the middle of the header. If it is omitted and section is a number from 1 to 9, certain versions of man will supply a default value.

.SH [name]

Start a section with the given name. If no argument is given, the next line will be used as the name.

.SS [name]

Start a subsection with the given name. If no argument is given, the next line will be used as the name.

.LP or .PP or .P

Paragraph break, resetting any indentation caused by the .TP, .IP, or .HP command

.TP [n]

Start a labelled, indented paragraph. The next text line after this command is the paragraph label, and any following text lines up to the next .PP will be indented. If a numeric argument is given, the paragraph will be indented by that many columns.

.IP [text] [n]

Start an indented paragraph. The text argument, if given, will be used as the paragraph’s bullet/“tag”; this is usually the bullet escape sequence (\(bu), the em-dash escape sequence (\(em), or a number followed by a period. If no text argument is supplied, no bullet will be present, but the following paragraph will still be indented; this can be used to start a new indented paragraph after an initial indented paragraph created by .TP, .IP, or .HP.

If a numeric second argument is given, the paragraph will be indented by that many columns.

.HP [n]

Start an indented paragraph in which the first line is not indented. If a numeric second argument is given, the paragraph will be indented by that many columns.

.RS [n]

Increase the amount of indentation until a corresponding .RE. If a numeric second argument is given, the indentation will be increased by that many columns.

.RE

Undo the increase in indentation caused by the last .RS

.br

Insert a line break

.sp

Skip a line

.sp N

Skip N lines

Font Commands

The following escape sequences can be used to change the font within a text line:

\fB: Change the font to bold
\fI: Change the font to italic (on a terminal, underlined)
\fR: Change the font to roman
\fP: Change back to the previous font

The following requests render an argument in a given font:

.B [text]: Renders the given text (or the text of the next line if no argument is given) in bold with a word break before & after.
.I [text]: Renders the given text (or the text of the next line if no argument is given) in italic (on a terminal, underlined) with a word break before & after.

The following commands take multiple arguments and render them in alternating fonts. A word break/whitespace will be inserted before & after the arguments, but there will be no words breaks/whitespace inserted between the arguments. For example, the following:

This is
.BI very styled
text.

will render as:

This is verystyled text.

.RB text ...: Renders the given arguments in alternating roman and bold, roman first.
.BR text ...: Renders the given arguments in alternating bold and roman, bold first.
.RI text ...: Renders the given arguments in alternating roman and italic/underlined, roman first.
.IR text ...: Renders the given arguments in alternating italic/underlined and roman, italic/underlined first.
.BI text ...: Renders the given arguments in alternating bold and italic/underlined, bold first.
.IB text ...: Renders the given arguments in alternating italic/underlined and bold, italic/underlined first.

Tip

While modern groff lets you use these commands with any number of arguments, traditional implementations limit usage to six arguments; keep this in mind if you want to make your man page portable.

Additional Escape Sequences

Select Non-ASCII Characters

See grof_char(7) for the complete set of available character escape sequences.

Escape	Character
`\(bu`	bullet (•)
`\*R`	registration symbol (®)
`\*(Tm`	trademark symbol (™)
`\(co`	copyright (©)
`\(em`	em-dash (—)
`\(en`	en-dash (–)
`\(rq`	right double-quote (“)
`\(lq`	left double-quote (”)
`\(oq`	left single-quote (‘)
`\(cq`	right single-quote (’)

Note

When viewing a man page in the terminal, not all installations will display Unicode characters. On systems that display man pages in ASCII (which include macOS as of Big Sur), non-ASCII characters will be rendered as the visually closest ASCII character where possible.

Special ASCII Characters

Some output devices transform certain ASCII input characters to similar Unicode characters, so the following escape sequences can be used to ensure that the desired ASCII character appears in the rendered man page:

Escape	Character
`\(aq`	Apostrophe (‘)
`\(ga`	Grave accent (`)
`\(ha`	Caret/circumflex accent (^)
`\(ti`	Tilde (~)
`\-`	Hyphen/minus (-)

Other

A backslash followed by a space produces a non-breaking space that remains at a fixed width when text is justified.
\~ produces a non-breaking space that nevertheless stretches like a normal inter-word space when justifying text.
\& produces a non-printable, zero-width character. It can be placed next to a token to deprive it of any special meaning; for example, it can be placed after a period at the end of an abbreviation to prevent it from being treated as the end of a sentence.

Man Page Conventions

Manual Sections

Each man page is traditionally placed into one of nine manual sections (not to be confused with the sections within a man page created by the .SH command; for that, see below). The pages for a given section are stored together, and the section number is also used as the file extension. The sections are:

Commands for use by general users
C system calls
C library functions, libraries, & headers
Devices, special files, and sockets
File formats
Games
Miscellaneous
Commands for use by system administrators (including servers/daemons)
C kernel functions

Sections of a Man Page

The contents of a man page are divided into sections by the .SH command. Different sources give slightly different lists of the “standard” sections, but the most common, in roughly the order they should appear in a man page, are:

NAME

Gives the name of the man page and a short description of what it documents. This is usually considered the only mandatory section.

In order for a man page named “foobar” to be properly indexed by whatis and apropos, NAME must be the first section in the man page, and the .SH NAME line must be followed immediately by a line of the form foobar \- short description of foobar.

SYNOPSIS

Shows the syntax for invoking a command or calling a C function.

The synopsis for a command usually follows the following conventions:

Text that should be entered as-is by the user (e.g., options and the name of the command) should be in bold, while placeholder text that should be replaced with some value by the user (e.g., command arguments) should be in italics/underlined.
Optional syntax elements (e.g., most options) should be enclosed in square brackets.
Alternative forms of a syntax element (e.g., the short and long form of an option) should be separated by whitespace and a vertical bar.
Syntax elements that can be repeated (e.g., arguments that can be given multiple times) are indicated by appending three periods (...).

See A Sample Man Page below for an example of these rules in action.

DESCRIPTION

An explanation of what the command, function, etc. does

OPTIONS

A list of any command-line options a program takes and their meanings. This list is usually created using .TP or .IP to produce paragraphs labeled with the option they describe, with each option in bold and any argument it may take in italics/underlined.

EXIT STATUS

A list of the possible exit statuses for a command and their meanings

ENVIRONMENT

A description of any environment variables that affect the command or function

FILES

A description of any files the command uses (configuration files, startup files, etc.). It is recommended that file paths be styled in italics/underlined.

NOTES

Any miscellaneous notes

BUGS

A list of known shortcomings in the documented software or functionality

EXAMPLES

Examples of how to use the software or functionality

AUTHORS

A list of the authors of the software and/or man page

REPORTING BUGS

Information on how to report any bugs found in the software

COPYRIGHT

SEE ALSO

A comma-separated list of related man pages and/or other documents. References to other man pages should be formatted with the name in bold and the section number in roman enclosed in parentheses.

Other Conventions

Use \- instead of plain - for ASCII hyphens, e.g., in command-line options. Use plain - in hyphenated words.
Blank lines are discouraged, as they may not render correctly in all output formats. Use the .sp command instead to produce a blank line, or use .PP to start a new paragraph.
Each sentence should typically start on a new line.
Use only a single space between words in text; multiple spaces will not be collapsed into one when rendering.

Rendering a Man Page

To render a man page in the terminal the way that man would, run man -l path/to/man/page. (On macOS, you have to instead run mandoc -a path/to/man/page). You may also want to pass the --warnings option (-Wall for mandoc) in order to catch any problematic syntax.

If your version of man does not support the -l option, you can instead run groff (GNU roff, the typical roff implementation on nearly all Unix machines nowadays) directly with groff -man -Tutf8 path/to/man/page | less -is.

Of course, groff can render to more than just terminals by changing the value passed to the -T option in the groff command. Values of interest include pdf, ps (for PostScript), and html; these require the gropdf, grops, and grohtml commands, respectively, to be installed in order to work.

A Sample Man Page

.TH FOOBAR 1 2022-11-24 v1.0
.SH NAME
foobar \- foo all your bars
.SH SYNOPSIS
.B foobar
[\fB\-a\fR \fIarg\fP]
.RB [ \-f | \-\-fu ]
.RB [ \-xyz ]
.IR "bar " ...
.SH DESCRIPTION
.B foobar
takes a list of one or more
.IR bar s
and foos each of them.
.PP
In detail, lorem ipsum dolor sit amet, consectetur adipisicing elit,
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi
ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore
eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident,
sunt in culpa qui officia deserunt mollit anim id est laborum.
.SH OPTIONS
.TP
.BI "\-a " arg
Put the apple at
.IR arg .
.TP
.BR \-f ", " \-\-fu
Do a FUBAR instead.
.IP
Note that this can permanently destroy your computer.
.IP \fB\-x\fR
Be exact about it.
.TP
.B -y
Be yucky about it.
.IP \fB-z\fR
Be zucchini about it.
.SH FILES
.TP
.I ~/.cache/foo
This is where the bars are stored.
.SH "SEE ALSO"
.BR man (1),
.BR groff (7),
.BR groff_man (7)
.PP
.I A Brief History of Foo
by Fabian Oscar Oh
.PP
.RI < https://www.example.com/foo/bar.html >

[Download this file]

Rendered:

[View animation on asciinema.org]

Further References

groff(7) man page: <https://man7.org/linux/man-pages/man7/groff.7.html>
groff_char(7) man page: <https://man7.org/linux/man-pages/man7/groff_char.7.html>
groff_man(7) man page: <https://man7.org/linux/man-pages/man7/groff_man.7.html>
man-pages(7) man page: <https://man7.org/linux/man-pages/man7/man-pages.7.html>

Implementing Boolean Negation Flags with Clap

2022-10-23T00:00:00-04:00

Rust’s clap library is the language’s most popular crate for parsing command-line arguments. Though it has many useful features, programmers used to libraries like Python’s Click may find themselves struggling to implement one certain CLI convention: boolean flags --foo and --no-foo that undo each other. This article will show you how to implement such flags using clap’s Parser derivation mode — no need to dig into the far more verbose “builder” mode!

Note

The code in this article was tested with clap 4.0.18 on Rust 1.64.0.

Tip

There is currently an open issue on clap’s repository asking for an easier way to implement negation flags, but it doesn’t seem that it’ll be resolved any time soon.

To implement two boolean flags --foo and --no-foo such that the field foo is true when --foo is given last and false when either no arguments are given or when --no-foo is given last, simply declare a --foo boolean option as usual, then add a separate --no-foo option, and declare that they override each other using overrides_with. The --foo option field will then end up holding the final boolean value, and the --no-foo field will be unused, so put an underscore at the start of its name so that clippy doesn’t complain. (If you don’t want any unused fields in your arguments struct, I’m afraid your only option is to turn to the more verbose builder mode.)

A minimal example of such code would look like this:

use clap::Parser;

#[derive(Debug, Parser)]
struct Arguments {
    /// Foo all the bars
    #[clap(long, overrides_with = "_no_foo")]
    foo: bool,

    /// Don't foo any bars [default]
    #[clap(long = "no-foo")]
    _no_foo: bool,
}

fn main() {
    for opts in [
        vec!["arg0"],
        vec!["arg0", "--foo"],
        vec!["arg0", "--no-foo"],
        vec!["arg0", "--no-foo", "--foo"],
        vec!["arg0", "--foo", "--no-foo"],
        vec!["arg0", "--no-foo", "--foo", "--no-foo"],
        vec!["arg0", "--foo", "--no-foo", "--foo"],
    ] {
        let args = Arguments::parse_from(&opts);
        println!("{opts:?} -> {args:?}");
    }
}

[Link to the code on the Rust Playground]

The output from the above code is:

["arg0"] -> Arguments { foo: false, _no_foo: false }
["arg0", "--foo"] -> Arguments { foo: true, _no_foo: false }
["arg0", "--no-foo"] -> Arguments { foo: false, _no_foo: true }
["arg0", "--no-foo", "--foo"] -> Arguments { foo: true, _no_foo: false }
["arg0", "--foo", "--no-foo"] -> Arguments { foo: false, _no_foo: true }
["arg0", "--no-foo", "--foo", "--no-foo"] -> Arguments { foo: false, _no_foo: true }
["arg0", "--foo", "--no-foo", "--foo"] -> Arguments { foo: true, _no_foo: false }

As you can see, the value of the foo field is true when --foo is the last --foo/--no-foo option given and false otherwise.

What if you want a pair of boolean flags where the default is --foo/true rather than --no-foo/false? This is doable, but the code can look quite a bit confusing. Starting with the setup above, add action = clap::builder::ArgAction::SetFalse to the --no-foo option; this will invert the values of the --no-foo field, causing it to be true whenever --foo is the option in effect and false otherwise. But wait — that’s what we want for the --foo field, isn’t it? So we then swap the names of the fields (along with any documentation comments) while preserving their long and/or short attributes (which must now be given explicit values), so that the field named foo, now corresponding to the --no-foo option, will be true when --foo is given last and also when no options are given.

Sample code:

use clap::builder::ArgAction;
use clap::Parser;

#[derive(Debug, Parser)]
struct Arguments {
    /// Don't foo any bars
    #[clap(long = "no-foo", action = ArgAction::SetFalse)]
    foo: bool,

    /// Foo all the bars [default]
    #[clap(long = "foo", overrides_with = "foo")]
    _no_foo: bool,
}

fn main() {
    for opts in [
        vec!["arg0"],
        vec!["arg0", "--foo"],
        vec!["arg0", "--no-foo"],
        vec!["arg0", "--no-foo", "--foo"],
        vec!["arg0", "--foo", "--no-foo"],
        vec!["arg0", "--no-foo", "--foo", "--no-foo"],
        vec!["arg0", "--foo", "--no-foo", "--foo"],
    ] {
        let args = Arguments::parse_from(&opts);
        println!("{opts:?} -> {args:?}");
    }
}

[Link to the code on the Rust Playground]

Output:

["arg0"] -> Arguments { foo: true, _no_foo: false }
["arg0", "--foo"] -> Arguments { foo: true, _no_foo: true }
["arg0", "--no-foo"] -> Arguments { foo: false, _no_foo: false }
["arg0", "--no-foo", "--foo"] -> Arguments { foo: true, _no_foo: true }
["arg0", "--foo", "--no-foo"] -> Arguments { foo: false, _no_foo: false }
["arg0", "--no-foo", "--foo", "--no-foo"] -> Arguments { foo: false, _no_foo: false }
["arg0", "--foo", "--no-foo", "--foo"] -> Arguments { foo: true, _no_foo: true }

Python Asynchronous Programming Fundamentals

2022-05-29T00:00:00-04:00

Python introduced asynchronous programming capabilities in version 3.4 in 2014, with further notable improvements in almost every minor version since. However, to many Python programmers, this area of the language remains esoteric, misunderstood, and underutilized. This article aims to elucidate the fundamental concepts of asynchronous programming as part of the first step towards mastery.

There won’t be many code samples in this article, but reading it should make it easier to grok the asyncio documentation and figure out how to piece things together.

Contents

High-Level Overview
Definitions
Syntax
Running More than One Coroutine at Once
Exception Handling
Example Code
Asynchronous Programming vs. Threads
Async Version History
Alternative Async Implementations

High-Level Overview

Asynchronous programming provides a way to interleave execution of multiple functions (coroutines) at once; at any given point, only one of the functions is actually doing operations, while the others are waiting for things like blocking I/O to complete (or, if the I/O has already completed, they’re waiting for the current coroutine to be suspended so that they get a chance to resume).

Note

In the Python standard library and all third-party async libraries that I am aware of, the kinds of low-level operations that one can ultimately suspend execution waiting for are almost exclusively I/O-related. If you’re seeking to add concurrency to CPU-bound code (e.g., anything involving number crunching), you are likely to be better off using multiprocessing instead.

Specifically, whenever execution of a currently-running coroutine reaches an await expression, the coroutine may be suspended, and another previously-suspended coroutine may resume execution if what it was suspended on has since returned a value. Suspension can also happen when an async for block requests the next value from an asynchronous iterator or when an async with block is entered or exited, as these operations use await under the hood.

Note that, although multiple coroutines can be processed at once, effectively finishing in any order, operations within a single coroutine continue to be executed in the order they’re written. For example, given the following code:

async def f():
    ...

async def g():
    ...

async def amain():
    await f()
    await g()

asyncio.run(amain())

When await f() is reached, amain() will be suspended until f() is finished, and only after that will execution proceed to the next line, starting coroutine g(). If you want to execute f() and g() concurrently, you need to use asyncio.create_task(), asyncio.gather(), or similar. See this article for more details.

In general, coroutines can only be called or scheduled by other coroutines. To run a “top-level” coroutine from inside synchronous code (i.e., either inside a non-coroutine function or at module level), the simplest & preferred way is to use the asyncio.run() function introduced in Python 3.7. Code meant to run on older Pythons must use the lower-level loop.run_until_complete() instead, along with other loop methods for cleanup.

Definitions

Asynchronous programming features the execution of multiple tasks concurrently, with one task being run while waiting for others to complete. Asynchronous programming in Python is an example of cooperative multitasking [wiki], as it requires the running coroutines to cooperate and yield control on their own; if the current coroutine goes for a long time without calling await, execution will remain on that coroutine all the while, and all other coroutines in the thread will remain in a suspended state. This is contrast to the preemptive multitasking [wiki] of multithreaded and multiprocess programs, where the Python interpreter or OS scheduler decides on its own when to switch between running contexts, with the points at which switches can occur being difficult or impossible to predict in general.

A coroutine function is a function defined with async def instead of just def. (In the context of asynchronous programming, a function defined with just def is called a synchronous function.) Only coroutine functions can contain await, async for, and async with constructs, and, as of Python 3.10, they cannot contain yield from constructs.

Note that just calling a coroutine function does not cause it to start running; you need to either schedule it for concurrent execution with asyncio.create_task() or similar or else await on it directly.

A coroutine object is the result of calling a coroutine function. This is not the value returned (or raised) by the function; rather, it is a pending computation that can be suspended & resumed at any point that the coroutine function uses await, async for, or async with. The actual return value or exception is obtained either by awaiting on the coroutine object (possibly wrapped in a task and/or something like asyncio.gather()) from within another coroutine function or by running it as a “top-level” entry point using asyncio.run() within synchronous code.

Asynchronous generators — coroutine functions that use yield — are a bit of an exception. You do not await the result of the function; instead, you iterate through it using either async for ... in ...: or await anext(...).
Confusingly, both coroutine functions and coroutine objects can be referred to as just “coroutines.”

The actual scheduling of coroutine execution is managed by an event loop. An event loop is created by asyncio.run() or asyncio.new_event_loop(), handed one or more coroutines and/or synchronous callbacks, and then set off to run either forever or until completion of a “top-level” coroutine. It’s the event loop’s job to execute the current coroutine until it suspends on an await, after which it looks to see if any suspended coroutines are now done with their suspension and either picks one to resume or, if none are ready, waits until one is.

An awaitable is any value that the await keyword can be applied to; this includes coroutine objects, futures, future-likes, and tasks (see below). Awaiting on an awaitable causes the current coroutine to be suspended until the awaitable is ready to provide a value or raise an exception.

A future (class asyncio.Future) is a low-level container for the result of a computation (a value or exception) that starts out empty and is assigned a value or exception later. Awaiting on a future will suspend the current coroutine until something else either stores a result in the future or cancels it.

You may already be familiar with futures in the form of the Future class from the concurrent.futures module, which provides access to the results of operations evaluated in other threads or processes. The asyncio.Future class is similar in spirit, but has a different API.

A future-like is an object with an __await__() method, which must return an iterator. Awaiting on a future-like causes the current coroutine to be suspended until the iterator is exhausted, at which point the value attribute of the terminating StopIteration exception is returned as the result of the await expression.

A task (class asyncio.Task) represents a running coroutine. Creating a task from a coroutine object with asyncio.create_task() will cause the coroutine to be scheduled for execution concurrently with other running coroutines. The task instance can later be awaited on to suspend the current coroutine until the wrapped coroutine finishes executing, returning its result.

A running task can be cancelled by calling its Task.cancel() method. This will cause the underlying coroutine to receive an asyncio.CancelledError exception the next time it awaits, likely putting an end to the task’s execution.

Syntax

Asynchronous programming in Python takes place inside coroutines, functions defined using async def instead of just def. Within a coroutine, the await keyword can be applied to any awaitable expression (such as a call to another coroutine) to suspend execution of the coroutine until the awaitable has a value or exception ready, at which point the coroutine is resumed and the await expression returns that value or raises that exception.

Here’s a basic example you’ve seen in all the tutorials:

import asyncio

async def waiter():
    print("Before sleep")
    await asyncio.sleep(5)
    print("After sleep")

asyncio.run(waiter())

This code prints a message, waits for a five-second sleep to elapse, and then prints another message. As written, it’s rather pointless; we’re only running one coroutine at once, so there’s no advantage over using a synchronous function with time.sleep(). Here’s something slightly more involved:

import asyncio

async def operate(time, result):
    print(f"Spending {time} seconds doing operations ...")
    await asyncio.sleep(time)
    print(f"Operations done after {time} seconds!")
    return result

async def amain():
    x, y = await asyncio.gather(operate(5, 42), operate(2, 23))
    print(f"Got {x=}, {y=}")
    assert x == 42
    assert y == 23

asyncio.run(amain())

This code mocks spending time on two blocking operations in parallel. If you run the script (Python 3.8+ required) and time it, you’ll see that it only takes about 5 seconds in total, and the 2-second task completes three seconds before the 5-second one. After both tasks are done, the final “Got x=42, y=23” message is printed.

Besides await, there are two other syntactical constructs specific to coroutines: async for ... in ...: (for iterating over asynchronous iterables) and async with ...: (for entering & exiting asynchronous context managers). These work the same way as their non-async counterparts, except that the iterables and context managers in question need to support asynchronous usage; for example, an async for cannot iterate over a list, and an async with cannot operate on an ordinary filehandle. Similarly, a regular for cannot be applied to an asynchronous iterator, and a regular with cannot be applied to, say, asyncio.Lock.

Speaking of asynchronous iteration, this works pretty much how you’d expect: by using yield inside a coroutine, it becomes an asynchronous generator that can be iterated over with async for or await anext(...). Note that, in contrast to non-generator coroutines, you do not apply await to an asynchronous generator. For example, given this function:

async def aiterator():
    for i in range(5):
        await asyncio.sleep(i)
        yield i

you use it like this:

async for x in aiterator():
    print(x)

No await anywhere in the async for loop.

Note that there is no way to get a value out of a coroutine without awaiting on it (either directly or via something like asyncio.gather()); if a coroutine is never awaited and never converted into a task, asyncio will complain when it is garbage-collected. Moreover, await (and async for and async with) cannot be used outside of a coroutine; in order to start the awaiting on a “top-level” coroutine, you need to use asyncio.run().

Note that the body of a coroutine isn’t required to contain any awaits or similar, though if it doesn’t, there often isn’t much point in making it a coroutine in the first place. An exception is the __aenter__() special method of asynchronous context managers; usually, the body will just be return self, but it’s still required to define the method with async def.

Type Annotations

Adding type annotations to asynchronous code works the same way as for synchronous code. If an asynchronous func() take an integer x and returns a string, you write its annotated signature as async def func(x: int) -> str. However, if you pass around an unawaited coroutine object (not always the best idea), you annotate it as Awaitable[T], where T is the return type of the coroutine.

Async callables do not have their own type; they are instead annotated as Callable[..., Awaitable[T]], where T is the return type of the coroutine function.

Asynchronous iterators, iterables, and context managers, though, do get their own types: AsyncIterator, AsyncIterable, and typing.AsyncContextManager/contextlib.AbstractAsyncContextManager.

Special Methods

__aiter__() and __anext__()

These methods are used to implement asynchronous iterator & iterable classes as an alternative to writing asynchronous generator functions, analogously to how a class can be defined with __iter__() and __next__() methods to implement a synchronous iterator as an alternative to writing a generator function.

__aiter__() must be a synchronous function that returns an object with __aiter__() and __anext__() methods. __anext__() must be a coroutine that either returns a value or raises a StopAsyncIteration exception.

__aenter__() and __aexit__()

These methods are used to implement asynchronous context managers. They are defined the same way as the __enter__() and __exit__() methods of synchronous context managers, except that the asynchronous versions must be coroutines.

__await__()

This method is used to create a future-like class that can be awaited directly. There is generally very little need to implement this, but we include it here for completeness.

__await__() must be a synchronous function that returns an iterator. This iterator will be advanced each time the event loop checks to see if the future-like is ready to return a value. If the future-like is not ready, the iterator must yield a special value (see below). If the future-like is ready, it must either return a result or raise an exception; this result or exception will then be the result of the await expression acting on the future-like.

The values that the iterator yields depend on what async implementation the code is running under (See “Alternative Async Implementations” below). When using the Python standard library’s asyncio, you generally want to yield None. When using trio, you need to yield an instance of one of several private classes internal to trio. When using curio, you need to yield a ("trap_name", *trap_args) tuple instructing the kernel to invoke a special “trap” function; yielding ("trap_sleep", 0) instructs the kernel to do nothing special.

For example, if you want to implement your own Future class for use with asyncio, you might start out writing it like so:

class MyFuture:
    def __init__(self):
        self.value = None
        self.is_set = False

    def set_value(self, value):
        self.value = value
        self.is_set = True

    def __await__(self):
        while not self.is_set:
            yield
        return self.value

Historical Note: Generator-Based Coroutines

When asyncio was first introduced in Python 3.4, the async and await keywords were not yet present. Instead, coroutine functions were created by applying the @asyncio.coroutine decorator to a normal generator function, and awaiting was done using yield from. There were no asynchronous iterators or asynchronous context managers in 3.4, either. Even after async and await were introduced in Python 3.5, the older generator-based coroutines could not use them.

This style of writing coroutines was deprecated in Python 3.8 and removed entirely in Python 3.11.

Running More than One Coroutine at Once

Now for the part you’ve been waiting for, and the part that makes asynchronous programming worth it: actually running multiple functions concurrently.

The simplest way to start running another coroutine concurrently is to pass a coroutine object to asyncio.create_task(); this schedules the coroutine for execution, but it won’t actually start running until the next time the current coroutine calls await (and maybe not even then). asyncio.create_task() returns an asyncio.Task object that can be used to query the state of the coroutine or cancel it; awaiting on the task object will cause the current coroutine to be suspended until the task is complete, at which point the return value of the task’s underlying coroutine is returned.

If you create multiple tasks and then await on them one by one, a given await will not return a result until the task in question is done; if task B finishes running while you’re still awaiting on task A, the coroutine doing the awaiting will continue to be suspended until A is done, and when it then later awaits on B, it will get back B’s return value immediately, because B is already done. For example, the following code:

import asyncio
from time import strftime

def hms():
    return strftime("%H:%M:%S")

async def operate(time, result):
    print(f"{hms()}: Spending {time} seconds doing operations ...")
    await asyncio.sleep(time)
    print(f"{hms()}: Operations done after {time} seconds!")
    return result

async def amain():
    task1 = asyncio.create_task(operate(5, 42))
    task2 = asyncio.create_task(operate(2, 23))
    r1 = await task1
    print(f"{hms()}: task1 returned {r1}")
    r2 = await task2
    print(f"{hms()}: task2 returned {r2}")

asyncio.run(amain())

outputs something like the following:

17:12:56: Spending 5 seconds doing operations ...
17:12:56: Spending 2 seconds doing operations ...
17:12:58: Operations done after 2 seconds!
17:13:01: Operations done after 5 seconds!
17:13:01: task1 returned 42
17:13:01: task2 returned 23

If you need to await on multiple coroutines but don’t care about the exact order in which they finish, you can use asyncio.gather(), asyncio.as_completed(), or asyncio.wait() to await on them together; this article gives a good overview and explanation of the differences between the functions.

You don’t have to await on a task if you don’t need its return value or don’t need to be assured that it ever finishes, but if such a “background task” raises an uncaught exception, asyncio will complain. One way to address this is to attach a synchronous callback function to the task with Task.add_done_callback() that retrieves any uncaught exceptions with Task.exception(), like so:

import asyncio
from time import strftime

def hms():
    return strftime("%H:%M:%S")

async def bg_task():
    print(f"{hms()}: In the background")
    raise RuntimeError("Ouch")

def done_callback(task):
    try:
        if e := task.exception():
            print(f"{hms()}: Task <{task.get_name()}> raised an error: {e}")
        else:
            print(f"{hms()}: Task <{task.get_name()}> finished successfully")
    except asyncio.CancelledError:
        print(f"{hms()}: Task <{task.get_name()}> was cancelled!")

async def fg_task():
    task = asyncio.create_task(bg_task(), name="bg_task")
    task.add_done_callback(done_callback)
    print(f"{hms()}: Now we sleep and let bg_task do its thing")
    await asyncio.sleep(2)
    print(f"{hms()}: I'm awake!")

asyncio.run(fg_task())

The output from the above will look like:

17:16:33: Now we sleep and let bg_task do its thing
17:16:33: In the background
17:16:33: Task <bg_task> raised an error: Ouch
17:16:35: I'm awake!

Exception Handling

Whenever an exception occurs inside a coroutine, it propagates upwards to whatever’s awaiting on it; if unhandled, it will propagate all the way out through the asyncio.run() call, at which point all still-running tasks are cancelled. If there is no chain of awaits leading to the “top-level” coroutine (say, because you did asyncio.create_task() and then didn’t await on the result, letting it run in the background), asyncio will end up complaining when the coroutine is eventually garbage-collected. See the passage above on using Task.add_done_callback() to handle such errors.

For the specific case of KeyboardInterrupt, the exception is raised in whatever coroutine the main thread is currently running at the time.

Example Code

This gist provides an example of using asynchronous programming in Python to download assets for one or more releases of a GitHub repository in parallel. Try it out with an invocation like:

python download-assets.py --download-dir jq stedolan/jq jq-1.5 jq-1.6

The script requires Python 3.8 or higher and the ghrepo and httpx packages on PyPI to run.

Asynchronous Programming vs. Threads

Asynchronous programming does not use threads; by default, all coroutines and their operations are run in whichever thread called asyncio.run(). The exception is when asyncio.to_thread() or loop.run_in_executor() is used to run a synchronous function in a separate thread (or, with the latter function, even a separate process), returning an object that can be awaited on to receive the function’s result.

If multiple threads call asyncio.run() separately, each thread will get its own event loop and collection of coroutines.

Note that each thread in a Python process has at most one event loop at a time, and an event loop can only belong to one thread. An important consequence of this is that, if you have a synchronous function foo() that calls asyncio.run() on some coroutine, then foo() cannot be called by another coroutine, because that would lead to two event loops in the same thread, which doesn’t work.

Compared to threads, asynchronous programming provides the following advantages:

In asynchronous programming, the coroutine being executed can only change when the current coroutine uses await or similar. This allows the programmer to be assured that, between awaits in the same coroutine, operations will not be interfered with and data will not be modified by other coroutines.

When using threads, on the other hand, the running thread [1] can change at almost any point as chosen by the interpreter, which necessitates careful programming and copious use of locks in order to ensure that variables are not modified by one thread “behind the back of” another thread that’s also using them.
If you’ve done serious work with threads, you’ve likely encountered the fact that you cannot “kill” a thread in the middle of its execution unless the “killable” thread is deliberately programmed to allow for this by, say, regularly checking some flag and exiting if it’s true. Asynchronous programming, on the other hand, makes it possible to cancel a running coroutine via the asyncio.Task.cancel() method; once a coroutine is cancelled, the next time the event loop checks on it while it’s suspended on an await or similar, the coroutine will be resumed, but instead of receiving the value it was awaiting for, an asyncio.CancelledError will be raised at the await expression, likely putting an end to the coroutine’s execution.

[1]	Recall that, due to Python’s global interpreter lock (GIL), regardless of how many threads a Python program uses or how many cores your machine has, only one thread will be executing Python bytecode at any moment.

Async Version History

Here follows a list of the notable developments & changes in asynchronous programming across Python versions.

Python 3.4

The asyncio module is added, implementing PEP 3156. This enables the creation of coroutines via the @asyncio.coroutine decorator; within a coroutine, awaiting is performed with yield from. (Support for creating coroutines in this way would later be deprecated in Python 3.8 and removed in Python 3.11.)

Most of the functionality added in this version is now categorized as the “low level” part of asyncio.

Python 3.5

PEP 492 implemented:

It is now possible to define coroutines via async def and await with await.
- In Python 3.5, yield cannot be used inside the body of an async def coroutine function.
Asynchronous iteration with async for is now possible.
Asynchronous context managers are now supported via async with.
__await__(), __aiter__(), __anext__(), __aenter__(), and __aexit__() special methods added
Originally, __aiter__() methods were expected to be coroutines (or anything else returning an awaitable) resolving to asynchronous iterators. This was changed in 3.5.2 to have __aiter__() instead return an asynchronous iterator directly. Returning an awaitable from __aiter__() produces a PendingDeprecationWarning starting in 3.5.2, a DeprecationWarning starting in 3.6, and a RuntimeError starting in 3.7.

Python 3.6

yield can now be used in the body of an async def coroutine function, thereby enabling asynchronous generators (PEP 525). (yield from remains prohibited, though.)
async for can now be used in list, set, & dict comprehensions and in generator expressions
await expressions can now be used in any comprehension
Using async or await as an identifier now generates a DeprecationWarning

Python 3.7

async and await are now reserved keywords
asyncio.run() added
asyncio.create_task() added

Python 3.8

Running python -m asyncio now starts an async REPL
@asyncio.coroutine() is now deprecated
Passing a loop parameter is now deprecated for most of asyncio’s high-level API
asyncio.CancelledError now inherits directly from BaseException instead of Exception

Python 3.9

asyncio.to_thread() added

Python 3.10

aiter() and anext() functions added
The loop parameter (deprecated in Python 3.8) is now removed from most of asyncio’s high-level API

Python 3.11

asyncio.TaskGroup added
@asyncio.coroutine (deprecated in Python 3.8) is now removed

Alternative Async Implementations

While all the code we’ve shown so far uses the Python standard library’s asyncio module, it’s not required to use this to work with coroutines. Alternative async library implementations exist that define their own event loops and primitive operations. The more notable implementations include:

trio seeks to enable structured concurrency [wiki] in asynchronous code. In trio, a collection of tasks are run concurrently by grouping them together under a nursery (also known as a task group); if one of the tasks in a nursery raises an error, all the other tasks in the same nursery are automatically cancelled.
curio is an asyncio alternative featuring a more streamlined API and intended to be easier to reason about

In general, different async implementations are incompatible, and features from different implementations cannot be used in the same code unless you make careful use of whatever compatibility facilities they may provide.

In fact, just being able to use the same code unmodified regardless of whether using implementation A or implementation B is tricky, as all implementations use different primitives. Libraries to help you with that include:

sniffio can be used to detect which async library is in use
anyio provides a common API (based on trio) that can be used to run the same code under both asyncio and trio (and previously curio, until it & anyio parted ways in anyio 3.0)

Creating Multi-Value Enums in Python

2022-01-14T00:00:00-05:00

Say you’re creating an enum type in Python and you want each enum member to have one or more additional constant attributes alongside value, but these attributes can’t be easily derived from value (and thus don’t make for good @propertys). The Python docs provide an example involving planets similar to what you want, but on closer inspection, you see it won’t do — the example code produces an enum whose members’ values are tuples of the other attributes, whereas you want or need value to be something else. Is there a way to define enum members with a combination of a “main” value and a set of “extra” values?

Yes. Yes, there is.

For a specific example, let’s say you want to define an enum for cardinal and ordinal directions where the members’ values are human-readable names used in some data source you have to parse and each member should additionally have x and y attributes giving the deltas to add to a coordinate’s components in order to move a step in the respective direction. That is, you want the following to evaluate as shown:

>>> Direction("north-west")
<Direction.NORTH_WEST: 'north-west'>
>>> Direction.NORTH_WEST.value
'north-west'
>>> Direction.NORTH_WEST.x
-1
>>> Direction.NORTH_WEST.y
1

The solution is to define each enum member with a tuple of the value and extra attributes, and then write a __new__ method that assigns the elements of the tuple (which are passed as arguments to __new__) appropriately, like so:

from enum import Enum

class Direction(Enum):
    NORTH      = ("north", 0, 1)
    NORTH_EAST = ("north-east", 1, 1)
    EAST       = ("east", 1, 0)
    SOUTH_EAST = ("south-east", 1, -1)
    SOUTH      = ("south", 0, -1)
    SOUTH_WEST = ("south-west", -1, -1)
    WEST       = ("west", -1, 0)
    NORTH_WEST = ("north-west", -1, 1)

    def __new__(cls, value, x, y):
        obj = object.__new__(cls)
        obj._value_ = value
        obj.x = x
        obj.y = y
        return obj

# Check that `value` and other attributes are what they're supposed to be:
for d in Direction:
    print(f"{d!r} {d.name=} {d.value=} {d.x=} {d.y=}")

# Member access by value uses only the "main" value:
print(f"{Direction('north')=}")

Output from the above script:

<Direction.NORTH: 'north'> d.name='NORTH' d.value='north' d.x=0 d.y=1
<Direction.NORTH_EAST: 'north-east'> d.name='NORTH_EAST' d.value='north-east' d.x=1 d.y=1
<Direction.EAST: 'east'> d.name='EAST' d.value='east' d.x=1 d.y=0
<Direction.SOUTH_EAST: 'south-east'> d.name='SOUTH_EAST' d.value='south-east' d.x=1 d.y=-1
<Direction.SOUTH: 'south'> d.name='SOUTH' d.value='south' d.x=0 d.y=-1
<Direction.SOUTH_WEST: 'south-west'> d.name='SOUTH_WEST' d.value='south-west' d.x=-1 d.y=-1
<Direction.WEST: 'west'> d.name='WEST' d.value='west' d.x=-1 d.y=0
<Direction.NORTH_WEST: 'north-west'> d.name='NORTH_WEST' d.value='north-west' d.x=-1 d.y=1
Direction('north')=<Direction.NORTH: 'north'>

Note

If you’re inheriting a concrete non-Enum type in addition to Enum (whether explicitly or via IntEnum or StrEnum), you may need to replace the obj = object.__new__(cls) line in the __new__ method with an invocation of the concrete type’s __new__. For example, if inheriting from (str, Enum) or Python 3.11’s StrEnum, this line should become obj = str.__new__(cls, value).

Now, if you’re type-checking with mypy (and you really should be), you’ll likely hit some errors on the above code, even after adding annotations to __new__. Specifically, as of mypy 0.931, there are two outstanding mypy issues that affect the above snippet:

Issue #1021 means that mypy ignores attribute assignments in __new__, so it won’t recognize x and y as attributes of Direction, leading to print(f"… {d.x=} {d.y=}") being seen as erroneous. There are two ways to make mypy aware of the extra-value attributes: either add annotations for them in the main body of the class, like so:

class Direction(Enum):
    NORTH      = ("north", 0, 1)
    # [snip]
    NORTH_WEST = ("north-west", -1, 1)

    # Add these two lines:
    x: int
    y: int

    def __new__(cls, value: str, x: int, y: int) -> Direction:
        ...

… or else move the extra-value assignments to an __init__ method, like so:

class Direction(Enum):
    NORTH      = ("north", 0, 1)
    # [snip]
    NORTH_WEST = ("north-west", -1, 1)

    def __new__(cls, value: str, x: int, y: int) -> Direction:
        obj = object.__new__(cls)
        obj._value_ = value
        return obj

    def __init__(self, value: str, x: int, y: int) -> None:
        self.x = x
        self.y = y

The second mypy problem is issue #10573: mypy is not aware of the Enum class’s trickery with its constructor, so when it sees Direction('north') on the last line, having already seen a Direction.__new__ that takes three arguments, it thinks there are two arguments missing from the Direction() call. Unfortunately, at time of writing, there does not seem to be a way around this other than just slapping a “# type: ignore[call-arg]” comment on each call to a multi-value enum type.

Skipping Pytest Tests Unless an Option is Given

2021-12-05T00:00:00-05:00

When testing Python code with pytest, you may occasionally write tests that you only want to run under special circumstances, such as long-running tests that should only be run under continuous integration and not when invoking pytest locally. The naïve way to accomplish this is to decorate the tests in question with a pytest mark like @pytest.mark.slow and then specify -m "not slow" when running pytest locally, but then you have to remember to pass this option every time, and if you hardcode it into your tox.ini or pytest configuration, you’ll need something else to remove it when testing under CI. Fortunately, there are better ways to make pytest skip tests by default.

In this article, we will add a “--run-slow” option to pytest and configure selected “slow” tests to be skipped unless this option is given on the command line.

Option 1: Use a Hook to Attach a `skip` Marker to Marked Tests

One way to disable selected tests by default is to give them all some mark and then use the pytest_collection_modifyitems hook to add an additional pytest.mark.skip mark if a certain command-line option was not given. We do this by adding the following to our conftest.py file:

import pytest

def pytest_addoption(parser):
    parser.addoption(
        "--run-slow",
        action="store_true",
        default=False,
        help="Run slow tests",
    )

def pytest_collection_modifyitems(config, items):
    if not config.getoption("--run-slow"):
        skipper = pytest.mark.skip(reason="Only run when --run-slow is given")
        for item in items:
            if "slow" in item.keywords:
                item.add_marker(skipper)

and then decorate all of our slow tests with @pytest.mark.slow, like so:

import pytest

@pytest.mark.slow
def test_something_slow():
    ...

Now, when we run pytest, the marked tests will be skipped unless the --run-slow option is passed on the command line.

Option 2: Use `@pytest.mark.skipif`

Did you know? The condition passed to @pytest.mark.skipif doesn’t have to be a Python boolean expression; it can instead be a condition string containing a Python expression, and condition strings are evaluated in a namespace that includes the pytest config object. This lets us write skipif decorators that skip tests based on whether or not certain command-line options were given.

We start out by defining a command-line option in conftest.py that will be used to tell pytest to run otherwise-skipped tests:

def pytest_addoption(parser):
    parser.addoption(
        "--run-slow",
        action="store_true",
        default=False,
        help="Run slow tests",
    )

We then decorate the tests we want to skip by default like so:

import pytest

@pytest.mark.skipif(
    "not config.getoption('--run-slow')",
    reason="Only run when --run-slow is given",
)
def test_something_slow():
    ...

If there are multiple tests to apply this condition to, we can assign the skipif decorator to a variable that is then used to decorate each one, like so:

import pytest

slow_test = pytest.mark.skipif(
    "not config.getoption('--run-slow')",
    reason="Only run when --run-slow is given",
)

@slow_test
def test_something_slow():
    ...

@slow_test
def test_something_very_slow():
    ...

Warning

You may see some old guides on the internet that instead write the skipif decorator with an actual boolean condition defined in terms of pytest.config, e.g.:

import pytest

# Outdated!
@pytest.mark.skipif(
    not pytest.config.getoption("--run-slow"),
    reason="Only run when --run-slow is given",
)
def test_something_slow():
    ...

However, this no longer works; the pytest.config global variable was removed from pytest in version 5.1.0.

Option 3: Use a Pre-Existing Plugin

As is the case for many, many things in Python, other people have already done the job of generalizing the above and publishing it on PyPI. I am aware of two pytest plugin projects for skipping tests unless a command-line option is given: pytest-explicit and pytest-optional-tests. I have not used either, but if they work as advertised, they should prove helpful.

Notable Features Introduced in Each Python Version

2021-07-11T00:00:00-04:00

The following is a list of all of the notable or interesting (in my biased opinion) features introduced in each major release of Python, starting with Python 3.0.

Contents

Python 3.0 (2008-12-03), Relative to Python 2.6
Python 3.1 (2009-06-26)
Python 3.2 (2011-02-20)
Python 3.3 (2012-09-29)
Python 3.4 (2014-03-17)
Python 3.5 (2015-09-13)
Python 3.6 (2016-12-23)
Python 3.7 (2018-06-27)
Python 3.8 (2019-10-14)
Python 3.9 (2020-10-05)
Python 3.10 (2021-10-04)
Python 3.11 (2022-10-03)
Python 3.12 (2023-10-02)

Python 3.0 (2008-12-03), Relative to Python 2.6

Release notes: [link]

print statement replaced with a print() function
exec keyword replaced with an exec() function, which does not take a stream argument
dict.keys(), dict.items() and dict.values() now return “views” instead of lists
dict.iterkeys(), dict.iteritems(), and dict.itervalues() removed
map() now returns an iterator, stops when the shortest iterable is exhausted, and cannot take None as a function
filter() now returns an iterator
Python 2’s xrange() is Python 3’s range(); Python 2’s range() is no more
Python 2’s raw_input() is Python 3’s input(); Python 2’s input() is no more
Python 2’s long is Python 3’s int
- The repr() of a (long) integer no longer includes a trailing L
Python 2’s unichr is Python 3’s chr
zip() now returns an iterator
Order comparison operators now raise TypeError when given operands of different types
sorted() and list.sort() no longer accept the cmp parameter
cmp() and __cmp__() removed
Dividing two integers with / now returns a float
sys module:
- stdin, stdout, and stderr are now text streams
- maxint, exitfunc(), exc_clear(), exc_type, exc_value, and exc_traceback removed
itertools module:
- imap(), ifilter(), and izip() removed
- ifilterfalse() renamed to filterfalse()
- izip_longest() renamed to zip_longest()
math module: ceil() and floor() now return integers
array module:
- array.read() and array.write() removed
- c typecode removed
operator module: sequenceIncludes() and isCallable() removed
random module: jumpahead() API removed
os module: tmpnam(), tempnam(), and tmpfile() removed
string module: letters, lowercase, and uppercase removed
__builtin__ module renamed to builtins
Octal literals now use the prefix 0o instead of just 0
Python 2’s str is Python 3’s bytes; Python 2’s unicode is Python 3’s str
All string literals are now text/Unicode strs
The u prefix for text string literals is no longer supported (later reintroduced in Python 3.3)
Mixing byte strings and text strings now produces a TypeError
\uXXXX and \UXXXXXXXX escapes in raw string literals are now interpreted literally instead of being converted to Unicode characters
- As a consequence of this, the ur string prefix is no longer supported
basestring removed
The repr() of a string now longer escapes non-ASCII characters
The default Python source encoding is now UTF-8
Non-ASCII letters are now allowed in identifiers
StringIO and cStringIO removed; use io.StringIO and io.BytesIO instead
Function arguments and return values can now be annotated
Functions now have an __annotations__ attribute
Named parameters after *args or a bare * in a function signature are now keyword-only
Keyword arguments are allowed after the list of base classes in a class definition
New keyword: nonlocal
Extended iterable unpacking: Statements like a, b, *rest = some_sequence are now supported
Dictionary comprehensions added: {k: v for k, v in stuff}
Set literals and set comprehensions added: {1, 2} and {x for x in stuff}
New builtin functions: ascii() and next()
intern() moved to sys.intern()
Builtins apply(), buffer(), callable() (later reintroduced in Python 3.2), coerce(), execfile(), and file removed
reduce() moved to functools.reduce()
reload() moved to imp.reload()
dict.has_key() removed
round()’s rounding strategy and return type changed
True, False, and None are now reserved words

The syntax for using a metaclass has changed from:

class C:
    __metaclass__ = M
    ...

to:

class C(metaclass=M):
    ...

Special method __prepare__ on metaclasses added
Module-global __metaclass__ variable no longer supported
List comprehensions of the form [... for var in item1, item2, ...] must now be written [... for var in (item1, item2, ...)] instead.
The loop control variables of list comprehensions are no longer leaked into the surrounding scope
... (the ellipsis) can now be used as an expression anywhere, and it can no longer be spelled . . .
Tuple unpacking in function parameters no longer supported
Calling repr() via backticks no longer supported
<> removed
Trailing l or L on integer literals no longer supported
from module import * is no longer allowed inside functions
All module imports where the module name does not start with a period are now treated as absolute imports
Classic classes removed; all classes are now new-style
The following modules are removed: audiodev, Bastion/rexec, bsddb185, bsddb3, Canvas, cfmfile, cl, commands, compiler, dircache, dl, fpformat, gopherlib, htmllib, ihooks, imageop, imputil, linuxaudiodev, md5, mhlib, mimetools, MimeWriter, mimify, multifile, mutex, new, popen2, posixfile, pure, rfc822, sets, sgmllib, sha, sre, statvfs, stringold, sunaudio, sv, test.testall, thread, timing, toaiff, user, UserDict (moved to collections), UserList (moved to collections), and UserString (moved to collections)
All IRIX-specific, Mac-specific, and Solaris-specific modules removed
_winreg module renamed to winreg
ConfigParser module renamed to configparser
copy_reg module renamed to copyreg
Queue module renamed to queue
SocketServer module renamed to socketserver
cPickle module renamed to _pickle
cProfile module renamed to _profile
repr module renamed to reprlib
test.test_support module renamed to test.support
The modules anydbm, dbhash, dbm, dumbdm, gdbm, and whichdb have been combined into a new dbm module
HTMLParser module renamed to html.parser
htmlentitydefs module renamed to html.entities
httplib module renamed to http.client
The modules BaseHTTPServer, CGIHTTPServer, and SimpleHTTPServer have been combined into a new http.server module
Cookie module renamed to `http.cookies
cookielib module renamed to http.cookiejar

tkinter modules renamed as follows:

Old Name	New Name
Dialog	tkinter.dialog
FileDialog	tkinter.filedialog
FixTk	tkinter._fix
ScrolledText	tkinter.scrolledtext
SimpleDialog	tkinter.simpledialog
Tix	tkinter.tix
Tkconstants	tkinter.constants
Tkdnd	tkinter.dnd
Tkinter	tkinter.__init__
tkColorChooser	tkinter.colorchooser
tkCommonDialog	tkinter.commondialog
tkFileDialog	tkinter.filedialog
tkFont	tkinter.font
tkMessageBox	tkinter.messagebox
tkSimpleDialog	tkinter.simpledialog
turtle	tkinter.turtle

urllib2 module split into urllib.request and urllib.error modules
urlparse module renamed to urllib.parse
urllib module split into urllib.parse, urllib.request, and urllib.error modules
robotparser module renamed to urllib.robotparser
xmlrpclib module renamed to xmlrpc.client
The modules DocXMLRPCServer and SimpleXMLRPCServer have been combined into a new xmlrpc.server module
Exceptions must now inherit from BaseException
StandardError removed
Exceptions no longer behave like sequences; use the args attribute instead
except exc, var is now written except exc as var
The variable used to catch an exception is now deleted when the except block is left
raise Exception, args is now written raise Exception(args)
Raising an exception inside an except or finally block now causes implicit exception chaining
Explicit exception chaining can be done with raise SecondaryException() from primary_exception
__getslice__(), __setslice__(), and __delslice__() removed
Special method next() renamed to __next__()
__oct__() and __hex__() removed
Removed support for __members__ and __methods__
Function attributes of the form func_X renamed to __X__
Special method __nonzero__() renamed to __bool__()
super() can now be invoked without arguments inside an instance method

Python 3.1 (2009-06-26)

Release notes: [link]

Multiple context managers can now be used in a single with statement
Directories & zip archives containing a __main__.py can now be executed by passing their path to the interpreter
Packages containing a __main__ submodule can now be executed with python -m and runpy
New bytes and bytearray method: maketrans()
The repr()s of floats are now shorter
New int method: bit_length()
The fields in strings formatted with str.format() can now omit numbering in order to be automatically numbered, as in 'Sir {} of {}'.format('Gallahad', 'Camelot')
The format specification mini-language now includes a thousands separator specifier
round(x, n) now returns an integer if x is an integer
New modules: importlib and tkinter.ttk
collections module:
- Counter and OrderedDict added
- namedtuple() now accepts a rename parameter
contextlib module: nested() is now deprecated
decimal module: Decimal.from_float() added
io module: SEEK_SET, SEEK_CUR, and SEEK_END added
itertools module:
- combinations_with_replacement() and compress() added
- count() now accepts a step parameter
json module: Decoders now accept an object_pairs_hook parameter
logging module: NullHandler added
re module: sub(), subn(), and split() now accept a flags parameter
string module: maketrans() is now deprecated

Python 3.2 (2011-02-20)

Release notes: [link]

New modules: argparse, concurrent.futures, html, and sysconfig
.pyc files are now stored in __pycache__/ directories
Modules now have a __cached__ attribute
New str method: format_map()
The str() of a float or complex is now the same as its repr()
New range methods: index() and count()
callable() function from Python 2 restored
ResourceWarning warning type added
abc module: abstractclassmethod() and abstractstaticmethod() added
ast module: literal_eval() now supports set & bytes literals
collections module:
- Counter.subtract() added
- OrderedDict.move_to_end() added
- deque.count() and deque.reverse() added
compileall command-line interface: -i and -b options added
configparser module:
- ConfigParser class replaced with SafeConfigParser, which is now customizable
- New API added based on the mapping protocol
contextlib module:
- ContextDecorator added
- nested() removed
csv module:
- unix_dialect dialect (dialect name "unix") added
- DictWriter.writeheader() added
datetime module:
- timezone added
- timedelta instances can now be multiplied by floats and divided by floats & ints
- date.strftime() now supports years from 1000 through 9999
decimal module:
- The Decimal constructor now accepts floats
- Decimal instances can now be compared with float and fractions.Fraction instances
- Context.clamp added
email.parser module: BytesFeedParser, BytesParser, message_from_bytes() and message_from_binary_file() added
email.generator module: BytesGenerator added
fractions module: The Fraction constructor now accepts floats and decimal.Decimals
functools module: cmp_to_key(), lru_cache(), and total_ordering() added
gzip module: compress() and decompress() added
hashlib module: algorithms_available and algorithms_guaranteed added
Various ABCs added to importlib.abc
inspect module: getgeneratorstate() and getattr_static() added
io module: BytesIO.getbuffer() added
itertools module: accumulate() added
json module: The indent parameter to dumps() etc. can now be a string
logging module:
- basicConfig() now accepts a style parameter
- If a logging event occurs before any explicit configuration is set up, a default configuration (available in lastResort) is now enabled
- Python callables returning booleans can now be used as filters
logging.config module: dictConfig() added
math module: isfinite(), expm1(), erf(), erfc(), gamma(), and lgamma() added
os module: fsencode(), fsdecode(), supports_bytes_environ, getenvb(), and environb added
shutil module:
- copytree() now accepts ignore_dangling_symlinks and copy_function parameters
- make_archive(), unpack_archive(), etc. added
site module: getsitepackage(), getuserbase(), and getusersitepackages() added
smtplib module: SMTP.send_message() added
ssl module:
- SSLContext and match_hostname() added
- Server Name Indication (SNI) is now supported when linked against recent versions of OpenSSL
string module: maketrans() removed
sys module: hash_info added
tarfile module: TarFile.add() now accepts a filter parameter, and the exclude parameter is now deprecated
tempfile module: TemporaryDirectory added
threading module: Barrier added
unicodedata updated to Unicode 6.0.0
urllib.parse module: urlparse() now supports IPv6 addresses

Python 3.3 (2012-09-29)

Release notes: [link]

New modules: faulthandler, ipaddress, lzma, unittest.mock, and venv
Support for implicit namespace packages (directories without an __init__.py) added
All Unicode codepoints, from U+0000 to U+10FFFF, are now always supported; there is no longer a distinction between “narrow” and “wide” builds
Multiple exception types have been merged into OSError, which now has various subclasses for common error conditions
Delegating to a subgenerator/subiterator with yield from is now possible
Chained exception context can be suppressed with raise e from None
The u"unicode" syntax for Unicode strings from Python 2 is now supported again
Functions & classes now have a __qualname__ attribute
inspect module: signature(), Signature, Parameter, and BoundArguments added
sys module: implementation added
types module: SimpleNamespace added
importlib module: Various classes & functions added
Modules now have a __loader__ attribute
"\N{...}" can now take name aliases
unicodedata updated to UCD 6.1.0
New list and bytearray methods: copy() and clear()
Raw bytes literals can now be written rb"..." in addition to br"..."
open() now accepts an opener parameter
print() now accepts a flush parameter
Hash randomization with hash() is now enabled by default
New str method: casefold()
abc:
- It is now possible to combine abstractmethod with property, classmethod, or staticmethod
- abstractproperty, abstractclassmethod, and abstractstaticmethod are now deprecated
- ABCMeta.register() can now be used as a class decorator
array module: long long type now supported
base64 module: Decoding functions now accept ASCII-only strs
binascii module: The a2b_* functions now accept ASCII-only strs
bz2 module:
- open() added
- BZ2File() now accepts arbitrary file-like objects and implements most of the io.BufferedIOBase API
collections module:
- ChainMap added
- ABCs moved to collections.abc; aliases are still present in collections itself, but importing them is deprecated
- Counter now supports +, -, +=, -=, |=, and &=
contextlib module: ExitStack added
datetime module: New datetime methods: timestamp(), strftime(), and astimezone()
email module: Policy framework added
email.parser module: BytesHeaderParser added
email.utils module: format_datetime(), parsedate_to_datetime(), and localtime() added
functools module: lru_cache() now accepts a typed parameter
hmac module: compare_digest() added
http.client module: HTTPResponse.readinto() added
html.parser module: HTMLParser can now parse broken markup without errors
html.entities: html5 added
inspect module: getclosurevars() and getgeneratorlocals() added
io module:
- x mode added to open() function
- The TextIOWrapper constructor now accepts a write_through parameter
itertools module: accumulate() now accepts a func parameter
logging module: basicConfig() now accepts a handlers parameter
math module: log2() added
multiprocessing module:
- The Process constructor now accepts a daemon parameter
- Process.sentinel added
multiprocessing.connection module: wait() added
multiprocessing.pool module: New Pool methods starmap() and starmap_async() added
os module:
- fwalk(), pipe2(), sendfile(), getpriority(), setpriority(), replace(), get_terminal_size(), getxattr(), listxattr(), removexattr(), setxatter(), sync(), and others added
- Various functions now accept dir_fd and/or follow_symlinks parameters
- Various functions can now take file descriptors as path arguments
- stat(), fstat(), lstat(), and utime() now support timestamps with nanosecond precision
re module: str regular expressions now support \uXXXX and \UXXXXXXXX escapes
pipes.quote() moved to shlex
shutil module:
- disk_usage(), chown(), and get_terminal_size() added
- Several functions now accept a symlink parameter
stat module: filemode() added
struct module: size_t and ssize_t now supported
subprocess:
- Command strings can now be bytes on POSIX
- DEVNULL added
sys module: thread_info added
textwrap module: indent() added
time module: get_clock_info(), monotonic(), perf_counter(), process_time(), clock_getres(), clock_gettime(), clock_settime(), and CLOCK_* constants added
types module: MappingProxyType, new_class(), and prepare_class() added
urllib.request module: The Request constructor now accepts a method parameter
array module: The u format is now deprecated

Python 3.4 (2014-03-17)

Release notes: [link]

New modules: asyncio, ensurepip, enum, pathlib, selectors, statistics, and tracemalloc
codecs module: encode() and decode() are now documented
unicodedata updated to UCD 6.3
min() and max() now accept a default parameter
New special method: __length_hint__()
abc module: ABC and get_cache_token() added
argparse module: FileType now accepts encoding and errors parameters
base64 module:
- Encoding & decoding functions now accept any bytes-like object
- a85encode(), a85decode(), b85encode(), and b85decode() added
contextlib module: suppress() and redirect_stdout() added
doctest command-line interface: -o and -f options added
email module:
- as_string() now accepts a policy argument
- as_bytes() method added
- EmailMessage and MIMEPart added as part of new API
- contextmanager submodule added
filecmp module: clear_cache() and DEFAULT_IGNORES added
functools module: partialmethod() and singledispatch() added
glob module: escape() added
hashlib module: pbkdf2_hmac() added
html module: unescape() added
html.parser module: HTMLParser constructor now accepts a convert_charrefs parameter, and the strict argument is now deprecated
http.server command-line interface: --bind option added
imp.reload() moved to importlib
importlib module: InspectLoader.source_to_code() added
importlib.util module: MAGIC_NUMBER, cache_from_source(), source_from_cache(), and decode_source() added
importlib.machinery module: ExtensionFileLoader.get_filename() added
runpy and python -m can now be used with namespace packages
inspect module:
- Command-line interface added
- unwrap() added
ipaddress module: is_global property added
json module: dumps() etc. will now automatically set separators to (",", ": ") when indent is non-None
multiprocessing module:
- Start methods spawn and forkserver added
- Contexts added
- get_all_start_methods(), get_start_method(), set_start_method(), and get_context() added
operator module: length_hint() added
os module:
- cpu_count() added
- open() now supports the O_PATH and O_TMPFILE flags
pdb module: print command removed
pickle module: Protocol 4 added
plistlib module: load(), dump(), loads(), and dumps() added
pprint module: PrettyPrinter, pformat(), and pprint() now accept a compact parameter
re module: fullmatch() and regex.fullmatch() added
resource module: prlimit() added
shutil module: copyfile() now raises a SameFileError when the source and destination are the same file
ssl module:
- create_default_context() and get_default_verify_paths() added
- New SSLContext methods: cert_store_stats(), get_ca_certs(), and load_default_certs()
stat module: S_IFDOOR, S_IFPORT, and S_IFWHT added
struct module: iter_unpack() and Struct.iter_unpack() added
subprocess module:
- check_output() now accepts an input parameter
sys module: getallocatedblocks() and __interactivehook__ added
tarfile module: Command-line interface added
textwrap module:
- The TextWrapper constructor now accepts max_lines and placeholder parameters
- shorten() added
threading module: main_thread() added
traceback module: clear_frames() added
types module: DynamicClassAttribute() added
urllib.request module:
- data: URLs now supported
- Request objects are now reusable
urllib.error module: HTTPError.headers added
venv module: The EnvBuilder constructor and create() now accept a with_pip parameter
importlib module: A number of methods & functions are deprecated
The imp module is now deprecated and will be removed in Python 3.12
The formatter module is now deprecated and will be removed in Python 3.10
The U mode of various “open” functions is now deprecated and will be removed in Python 3.11

Python 3.5 (2015-09-13)

Release notes: [link]

Coroutine functions (async def), awaitable objects (await and __await__()), asynchronous iteration (async for, __aiter__(), and __anext__()), and asynchronous context managers (async with, __aenter__(), and __aexit__()) added
@ operator (with __matmul__() etc. special methods) for matrix multiplication added
Multiple * and/or ** unpackings can now be used in a single function call
Tuple, list, set, & dictionary displays may now contain * or ** unpackings (as appropriate)
Percent-formatting of bytes and bytearray objects with % added
New modules: typing and zipapp
System calls are now retried when interrupted by a signal
from __future__ import generator_stop added to cause StopIteration exceptions raised inside generators to be transformed into RuntimeExceptions, which becomes the default in Python 3.7
- Without the __future__ import, such exceptions generate PendingDeprecationWarnings.
RecursionError exception type (a subclass of RuntimeError) added
.pyo files eliminated; optimized bytecode is now stored in .pyc files with opt- tags in their name
"namereplace" error handler added
Various additions & improvements to the asyncio module
cmath module: isclose() added
collections module:
- New deque methods: index(), insert(), and copy()
- deque now supports + and *
collections.abc module: Generator, Awaitable, Coroutine, AsyncIterator, and AsyncIterable added
configparser module: ConfigParser can now take a dictionary of converters, and subclasses can define additional converters as methods
contextlib module: redirect_stderr() added
enum module: The Enum callable now accepts a start parameter
glob module: glob() and iglob()` now support the ``** pattern
http module: HTTPStatus added
importlib.util module: module_from_spec() added
inspect module:
- BoundArguments.apply_defaults() added
- Signature.from_callable() added
- signature() now accepts a follow_wrapped parameter
- iscoroutine(), iscoroutinefunction(), isawaitable(), getcoroutinelocals(), and getcoroutinestate() added
io module: new BufferedIOBase method: readinto1()
ipaddress module:
- The IPv4Network and IPv6Network constructors now accept an (address, netmask) argument
- New IPv4Network and IPv6Network attribute: reverse_pointer
json module: JSON decoding errors now raise JSONDecodeError
json.tool command-line interface: The input order of keys is now preserved on output; the --sort-keys option will sort the keys instead
linecache module: lazycache() added
locale module: delocalize() added
logging module: Logging methods now accept exception instances as exc_info arguments
math module: isclose(), gcd(), inf, and nan added
fractions module: gcd() is now deprecated and will be removed in Python 3.9
operator module: matmul() and imatmul() added
os module: scandir() added
os.path module: commonpath() added
pathlib module:
- New Path methods: samefile(), expanduser(), write_text(), read_text(), write_bytes(), and read_bytes()
- Class method Path.home() added
- Path.mkdir() now accepts an exist_ok parameter
readline module: append_history_file() added
selectors module: DevpollSelector added
shutil module: move() now accepts a copy_function argument
signal module: SIG* constants have been converted to enums
socket module: socket.sendfile() added
ssl module: SSLObject added
subprocess: run() added
sys module: set_coroutine_wrapper(), get_coroutine_wrapper(), and is_finalizing() added
time module: monotonic() is now always available
timeit command-line interface: --unit option added
traceback module: TracebackException, StackSummary, FrameSummary, walk_stack(), and walk_tb() added
types module: CoroutineType and coroutine() added
unicodedata updated to Unicode 8.0.0
unittest command-line interface: --locals option added
unittest.mock module: Mock.assert_not_called() added
urllib.request module: HTTPPasswordMgrWithPriorAuth added
platform module: dist() and linux_distribution() are now deprecated
html.parser module: The convert_charrefs parameter to the HTMLParser constructor now defaults to True

Python 3.6 (2016-12-23)

Release notes: [link]

Formatted string literals (“f-strings”)
Variables can now be annotated by following the name of the variable with a colon and the annotation
Underscores can now be used in numeric literals
await and yield can now be used in the same function, thereby enabling asynchronous generators
async for can now be used in list, set, & dict comprehensions and in generator expressions
await expressions can now be used in any comprehension
Special methods __init_subclass__() and __set_name__() added
os.PathLike, the __fspath__() method, and os.fspath() added
- Relevant file functions now accept os.PathLike objects
ModuleNotFoundError exception type (a subclass of ImportError) added
datetime module:
- fold attribute added to datetime and time for denoting the second instance of a time duplicated due to DST
- The strftime() method of date and datetime now supports %G, %u, and %V
- datetime.astimezone() can now be called on naïve instances
The file system and console encodings on Windows are now both UTF-8
A class’s __dict__ now preserves the order in which the attributes were defined
**kwargs now preserves insertion order
dicts are now implemented in such a way that they preserve insertion order
New module: secrets
Various additions & improvements to the asyncio module
cmath module: tau, inf, nan, infj, and nanj added
collections.abc: Collection, Reversible, and AsyncGenerator added
enum module: auto, Flag, and IntFlag added
json module: load() and loads() now support binary input in UTF-8, UTF-16, and UTF-32
math module: tau added
random module: choices() added
re module:
- Modifier spans (e.g., as in '(?i)g(?-i:v)r') are now supported in regular expressions
- Match objects can now be indexed to access groups
statistics module: harmonic_mean() added
The smtpd module is now deprecated and will be removed in Python 3.12
subprocess module:
- encoding and errors arguments added to Popen and the wrappers around it
- The args parameter to Popen and the wrappers around it can now be a path-like object or sequence of path-like objects on POSIX systems
time module: The tm_gmtoff and tm_zone attributes of struct_time are now available on all platforms
typing module:
- Generic type aliases like Dict[str, Tuple[S, T]] are now supported
- TYPE_CHECKING added
- ClassVar added
- NewType() added
- NamedTuple now supports variable annotation syntax
unittest.mock module: New Mock methods assert_called() and assert_called_once() added
venv command-line interface: --prompt option added
Using async or await as a name will now generate a DeprecationWarning
StopIteration exceptions raised inside generators now generate DeprecationWarnings
Invalid escape sequences now generate a DeprecationWarning
The asynchat and asyncore modules are now deprecated and will be removed in Python 3.12
The pyvenv script for creating venvs is now deprecated and will be removed in Python 3.8
unicodedata updated to Unicode 9.0.0

Python 3.7 (2018-06-27)

Release notes: [link]

from __future__ import annotations added to enable postponed evaluation of annotations
dicts are now guaranteed to preserve insertion order
async and await are now reserved keywords
New modules: contextvars, dataclasses, importlib.resources
New builtin function: breakpoint()
The interpreter now coerces ASCII locales to UTF-8 under certain circumstances on non-Windows OSes
__aiter__() methods can no longer be asynchronous
__getattr__() and __dir__() can now be defined on modules
time module: variants of the timer functions added that return a number of nanoseconds as an integer
Special methods __class_getitem__() and __mro_entries__() added
Python Development Mode added
New str, bytes, and bytearray method: isascii()
argparse module: ArgumentParser.parser_intermixed_args() added
Various additions & improvements to the asyncio module
- asyncio.run() added
collections module: defaults argument added to namedtuple()
contextlib module: nullcontext(), asynccontextmanager(), and AsyncExitStack added
datetime module:
- datetime.fromisoformat() added
- The "%z" format of the strptime() methods now accepts timezone offsets containing colons as well as a timezone specifier of “Z”.
enum module: Support for the _ignore_ class property added to Enum
functools module: singledispatch() now recognizes type annotations
http.server command-line interface: --directory option added
ipaddress module: subnet_of() and supernet_of() methods added to IPv4Network and IPv6Network
locale module: format() is now deprecated and will be removed in Python 3.12
math module: remainder() added
pathlib module: Path.is_mount() added
queue module: SimpleQueue added
subprocess module:
- capture_output argument added to run()
- text argument added to run() and the Popen constructor
Removed the fpectl module
StopIteration exceptions raised inside coroutines and generators are now transformed into RuntimeExceptions
unicodedata updated to Unicode 11

Python 3.8 (2019-10-14)

Release notes: [link]

Assignment expressions: := (the “walrus operator”) can now be used to assign a value to a variable in the middle of an expression, e.g.:
```
if m := re.search(r'\d+', s):
    x = int(m.group())
```
Function parameters can now be made positional-only by placing a / after them in the argument list
One can now write f"{var=}" to get f"{var}={repr(var)}"
continue is now allowed in finally: clauses
New int, bool, and fractions.Fraction method: as_integer_ratio()
\N{name} escapes are now allowed in regular expressions
dicts and dictviews can now be passed to reversed()
Generalized iterable unpacking in yield and return statements no longer requires parentheses; e.g., one can now write return foo, *bar instead of having to do return (foo, *bar)
Missing commas between tuples in a list now generate a SyntaxWarning with a suggestion as to what went wrong
For integer arguments, the three-argument form of pow() can now take a negative exponent when the base is coprime to the modulus, in which case the modular multiplicative inverse (or a power thereof) is calculated
New modules: importlib.metadata and multiprocessing.shared_memory
Running python -m asyncio now starts an async REPL
asyncio module:
- CancelledError now inherits directly from BaseException instead of Exception
- coroutine() is now deprecated and will be removed in Python 3.11
- Passing a loop parameter is now deprecated for most of asyncio’s high-level API; it will be removed entirely in Python 3.10
- Explicitly passing coroutines to wait() is now deprecated
datetime module: New date and datetime method: fromisocalendar()
functools module: cached_property() and singledispatchmethod() added
io module: open_code() (the verified open hook) added
itertools module: accumulate() now accepts an initial parameter
json.tool command-line interface: --json-lines option added
math module:
- comb() added
- dist() added
- hypot() can now take multiple arguments
- isqrt() added
- perm() added
- prod() added
pathlib module:
- Path.link_to() added
- Path.rename() and Path.replace() now return the new path
- Path.unlink() now accepts a missing_ok argument
pickle module: Protocol 5 (with support for out-of-band buffers) added
platform module: dist() and linux_distribution() removed
shlex module: join() added
statistics module: NormalDist, fmean(), geometric_mean(), multimode(), and quantiles() added
subprocess module: The args parameter to Popen and the wrappers around it can now be a path-like object or sequence of path-like objects on Windows systems in addition to POSIX
sys module:
- audit() and addaudithook() (for audit hooks) added
- pycache_prefix added
- unraisablehook() and __unraisablehook__ added
typing module: TypedDict, Literal, Final, final(), Protocol, SupportsIndex, get_origin(), and get_args() added
unicodedata module:
- Updated to Unicode 12.1.0
- is_normalized() added
unittest.mock module:
- AsyncMock added
- call objects now have args and kwargs properties
zipfile module: Path added
Removed:
- macpath module (deprecated since Python 3.7)
- time.clock() (deprecated since Python 3.3)
- the pyvenv script for creating venvs (Use pythonX.Y -m venv instead)
- sys.set_coroutine_wrapper() and sys.get_coroutine_wrapper() (deprecated since Python 3.7)
Using is or is not with strings, numbers, and certain other literals now produces a SyntaxWarning

Python 3.9 (2020-10-05)

Release notes: [link]

dicts can now be merged & updated using the | and |= operators
Any valid expression can now be used as a decorator
New str, bytes, and bytearray methods: removeprefix() and removesuffix()
Built-in collection types like list and dict can now be used as generic types; e.g., List[str] can now be written list[str]
- This also applies to collections ABCs; e.g., typing.Sequence[str] can now be written collections.abc.Sequence[str].
- Importing the old List, Sequence, etc. types from typing is now deprecated but does not generate DeprecationWarningss at this time. The deprecated names will be removed from typing in the first Python release five years after 3.9.
New modules: graphlib and zoneinfo
asyncio module: to_thread() added
functools module: cache() added
importlib.resources module: files() added (introduced in importlib-resources v1.1.0)
Aligned importlib.metadata with importlib-metadata v1.6.1
ipaddress module now supports IPv6 scoped addresses
math module:
- gcd() can now take multiple arguments
- lcm(), nextafter(), and ulp() added
pathlib module:
- Path.readlink() added
- PurePath.is_relative_to() added
- PurePath.with_stem() added
random module: randbytes() added
typing module: Annotated added
The binhex, parser, and symbol modules are now deprecated
Using NotImplemented in a boolean context is now deprecated and will produce a TypeError in a future version of Python
Removed:
- fractions.gcd() (deprecated since Python 3.5)
- encoding parameter of json.loads() (deprecated & ignored since Python 3.1)
- asyncio.Task.current_task() and asyncio.Task.all_tasks() (deprecated since Python 3.7); use asyncio.current_task() and asyncio.all_tasks() instead
with (await asyncio.lock): and with (yield from asyncio.lock): statements are no longer supported; use async with lock instead. Likewise for asyncio.Condition and asyncio.Semaphore.
lib2to3 now emits a PendingDeprecationWarning and will be removed in Python 3.13
unicodedata updated to Unicode 13.0.0

Python 3.10 (2021-10-04)

Release notes: [link]

Pattern matching!

match status:
    case 400:
        return "Bad request"
    case 401 | 403 | 404:
        return "Not allowed"
    case _:
        return "Something's wrong with the Internet"

Context managers in with statements can now be enclosed in parentheses, e.g.:
```
with (CtxManager1() as example,
      CtxManager2()):
```
Assignment expressions can now be used unparenthesized within set literals, set comprehensions, and sequence indexes (but not slices)
Numeric literals immediately followed by keywords (e.g., 0in x) now generate a deprecation warning. Future Python versions will change this to a syntax warning and then a syntax error.
Common syntax errors now have better error messages
AttributeError and NameError error messages now include suggestions as to what you might have meant
Union[X, Y] can now be written X | Y
Dictionary views returned by dict.keys(), dict.values(), and dict.items() now have mapping attributes wrapping the original dict
The second argument of isinstance() and issubclass() can now be a Union
New int method: bit_count()
open() and friends can now be passed encoding="locale" in order to explicitly use the current locale’s encoding
zip() now has a strict parameter for requiring that all input iterables have the same length
New builtin functions: aiter() and anext()
EncodingWarning warning type added
The loop parameter (deprecated in Python 3.8) is now removed from most of asyncio’s high-level API
base64 module: b32hexencode() and b32hexdecode() added
bisect module: The functions now accept a key argument
codecs module: unregister() added
Collections ABCs can no longer be imported from collections; import them from collections.abc instead
collections module: Counter.total() added
contextlib: aclosing() added
dataclasses:
- The dataclass decorator now accepts a slots argument
- Fields can now be made keyword-only
distutils is now deprecated and will be removed in Python 3.12
glob module: glob() and iglob() now accept root_dir and dir_fd arguments
Aligned importlib.metadata with importlib-metadata v4.6
- entry_points() and package_distributions() added
inspect module: get_annotations() added
io module: text_encoding() added
itertools module: pairwise() added
os.path.realpath() now has a strict parameter for erroring when a path doesn’t exist or a symlink loop is encountered
pathlib module:
- Slice and negative indexing support added to Path.parents
- Path.hardlink_to() added, superseding Path.link_to(), which is now deprecated and will be removed in Python 3.12
- Path.stat() and Path.chmod() now have a follow_symlinks argument
platform module: freedesktop_os_release() added
statistics module: covariance(), correlation(), and linear_regression() added
sys module: orig_argv and stdlib_module_names added
traceback module: format_exception(), format_exception_only(), and print_exception() can now take just an exception argument as a positional-only parameter
types module: EllipsisType, NoneType, and NotImplementedType added
typing module: Concatenate, ParamSpec, TypeAlias, TypeGuard, and is_typeddict() added
Removed the formatter and parser modules (deprecated in Python 3.4 and 3.9, respectively)

Python 3.11 (2022-10-03)

Release notes: [link]

Exception groups
- ExceptionGroup and BaseExceptionGroup exception types added
- try: ... except* ...: ... syntax (note asterisk) for matching exception groups
Traceback messages now highlight the exact expression that caused the error
Starred expressions can now be used in for statements (e.g., for x in *lst1, *lst2:)
- This was enabled by the grammar rewrite in Python 3.9 but was not noticed or documented until 3.11.
Octal escapes in string & bytes literals with values greater than 0o377 (e.g., \477) now generate a DeprecationWarning
The various “open” functions no longer accept the U mode
New object method: __getstate__()
New BaseException method: add_note()
Chaining @classmethod decorators and using them to wrap other decorators (such as @property) is now deprecated and will be removed in Python 3.13
Format specification mini-language: A z option can now be used with floating-point types in order to coerce negative zero to “regular” zero after rounding.
New modules: tomllib and wsgiref.types
asyncio module:
- @coroutine (deprecated since 3.8) and asyncio.coroutines.CoroWrapper removed
- Barrier, Runner, TaskGroup, and timeout() added
binascii module: a2b_hqx(), b2a_hqx(), rlecode_hqx(), and rledecode_hqx() removed
binhex module removed
contextlib module: chdir() added
datetime module:
- UTC added
- The fromisoformat() methods now accept most ISO 8601 strings
enum module:
- EnumCheck, FlagBoundary, ReprEnum, StrEnum, @global_enum(), @member(), @nonmember(), @property, show_flag_values(), and @verify added
- EnumMeta renamed to EnumType; the old name is currently as an alias
functools module: singledispatch now supports union types
hashlib module: file_digest() added
inspect module:
- getmembers_static() and ismethodwrapper() added
- formatargspec() and getargspec() removed
- Frame-related functions now return objects instead of tuples (though the new objects are backwards-compatible with the tuple interface)
locale module:
- getdefaultlocale() is now deprecated and will be removed in Python 3.13
- getencoding() added
logging module:
- getLevelNamesMapping() added
- createSocket() method added to logging.handlers.SysLogHandler
math module: cbrt() and exp2() added
operator module: call() added
pathlib module: The glob() and rglob() methods now return only directories if given a pattern ending with the pathname separator
re module:
- (?>...), *+, ++, ?+, and {m,n}+ can now be used in regular expressions
- NOFLAG added
string module: get_identifiers() and is_valid() methods added to string.Template
sys module: exception() added
typing module:
- LiteralString, Never, Self, TypeVarTuple, Unpack, assert_never(), assert_type(), clear_overloads(), get_overloads(), @dataclass_transform, and reveal_type() added
- Text is now deprecated
- Individual fields in a TypedDict may now be annotated as Required or NotRequired
- The keyword-argument syntax for constructing TypedDict types is now deprecated and will be removed in Python 3.13
unicodedata updated to Unicode 14.0.0
zipfile module:
- metadata_encoding argument added to ZipFile constructor
- ZipFile.mkdir() added
- stem, suffix, and suffixes attributes added to zipfile.Path
The following modules are now deprecated and are planned to be removed in Python 3.13: aifc, audioop, cgi, cgitb, chunk, crypt, imghdr, mailcap, msilib, nis, nntplib, ossaudiodev, pipes, sndhdr, spwd, sunau, telnetlib, uu, and xdrlib

Python 3.12 (2023-10-02)

Release notes: [link]

New syntax for type-annotating generic classes, generic functions, and type aliases:

Generic class:

Old syntax:

from typing import Generic, TypeVar

T_str = TypeVar("T_str", covariant=True, bound=str)

class ClassA(Generic[T_str]):
    def method1(self) -> T_str:
        ...

New syntax:

# No Generic or TypeVar

class ClassA[T: str]:
    def method1(self) -> T:
        ...

Generic function:

Old syntax:

from typing import TypeVar

T = TypeVar("T")

def func(a: T, b: T) -> T:
    ...

New syntax:

# No TypeVar

def func[T](a: T, b: T) -> T:
    ...

Type alias:

Old syntax:

from typing import TypeAlias

IntOrStr: TypeAlias = int | str

New syntax:
```
type IntOrStr = int | str
```

Generic type alias:

Old syntax:

from typing import TypeAlias

T = TypeVar("T")

ListOrSet: TypeAlias = list[T] | set[T]

New syntax:

# No TypeVar

type ListOrSet[T] = list[T] | set[T]

Variadic type alias:

Old syntax:

from typing import TypeAlias, TypeVarTuple

Ts = TypeVarTuple("Ts")

IntEtc: TypeAlias = tuple[int, *Ts]

New syntax:
```
type IntEtc[*Ts] = tuple[int, *Ts]
```

Type alias with a parameter spec:

Old syntax:

from collections.abc import Callable
from typing import ParamSpec, TypeAlias

P = ParamSpec("P")

IntMaker: TypeAlias = Callable[P, int]

New syntax:

from collections.abc import Callable

type IntMaker[**P] = Callable[P, int]

Types of individual fields in a **kwargs function parameter can now be annotated with typing.Unpack applied to a TypedDict, like so:

from typing import TypedDict, Unpack

class Movie(TypedDict):
    name: str
    year: int

def foo(**kwargs: Unpack[Movie]):
    # kwargs["name"] must be passed a `str`, and kwargs["year"] must be
    # passed an `int`.
    ...

{..} placeholders inside f-strings may now contain any valid Python expression
- Quotes from the enclosing string may now be reused:
```
>>> songs = ['Take me back to Eden', 'Alkaline', 'Ascensionism']
>>> f"This is the playlist: {", ".join(songs)}"
'This is the playlist: Take me back to Eden, Alkaline, Ascensionism'
```
- Expressions inside f-strings may now span multiple lines and contain inline comments.
- Expressions inside f-strings may now contain backslashes.
The buffer protocol is now accessible in Python code:
- Special methods __buffer__() and __release_buffer__() added
- collections.abc module: Buffer added
- inspect module: BufferFlags added
Source code containing NUL bytes now results in a SyntaxError
Invalid escape sequences now generate a SyntaxWarning
Octal escapes in string & bytes literals with values greater than 0o377 (e.g., \477) now generate a SyntaxWarning
Applying ~ (bitwise inversion) to a bool is now deprecated and will become an error in Python 3.14.
Removed modules:
- asynchat (deprecated in Python 3.6)
- asyncore (deprecated in Python 3.6)
- distutils (deprecated in Python 3.10)
- imp (deprecated in Python 3.4)
- smtpd (deprecated in Python 3.6)
array module: The array class is now subscriptable and thus properly generic
ast module: Accessing or using any of Bytes, Ellipsis, NameConstant, Num, and Str now generates a DeprecationWarning. They are planned to be removed in Python 3.14.
asyncio module:
- as_completed() and wait() now accept generators yielding tasks
- create_eager_task_factory() and eager_task_factory() added
- run() now takes an optional loop_factory parameter
calendar module: Day and Month added
collections.abc module: ByteString is now deprecated and will be removed in Python 3.14
configparser module: ConfigParser.readfp(), ParsingError.filename, and SafeConfigParser removed (deprecated in Python 3.2)
datetime module: datetime.utcnow() and datetime.utctimestamp() are now deprecated
fraction module: Fraction instances now support float-style formatting
importlib.abc module: ResourceReader, Traversable, and TraversableResources are now deprecated and will be removed in Python 3.14
importlib.resources: as_file() now supports resource directories
inspect module: getasyncgenlocals(), getasyncgenstate(), and markcoroutinefunction() added
itertools: batched() added
locale module: format() removed (deprecated in Python 3.7)
math module:
- nextafter() now takes an optional steps parameter
- sumprod() added
os module:
- DirEntry.is_junction() added
- listdrives(), listmounts(), listvolumes(), pidfd_open(), setns(), and unshare() added
- Calling fork() or forkpty() in a multithreaded program now generates a DeprecationWarning
os.path module: isjunction() and splitroot() added
pathlib module:
- Path classes can now be subclassed
- PurePath.with_segments() added
- Path.is_junction() and Path.walk() added
- PurePath.match(), Path.glob(), and Path.rglob() now take an optional case_sensitive parameter
- PurePath.relative_to() now takes an optional walk_up parameter
pkgutil module: find_loader() and get_loader() are now deprecated and will be removed in Python 3.14
random module:
- binomialvariate() added
- The lambd argument to expovariate() now defaults to 1.0
shutil module:
- rmtree() now takes an optional onexc parameter. The onerror parameter is deprecated and will be removed in Python 3.14.
- unpack_archive() now takes an optional filter parameter for limiting what gets extracted from tar archives
sqlite3 module:
- Command-line interface added
- connect() now takes an optional autocommit parameter
- Connection.autocommit, Connection.getconfig(), and Connection.setconfig() added
- The default adapters and converters are now deprecated
ssl module: match_hostname() and wrap_socket() removed (deprecated in Python 3.7)
statistics module: correlation() now takes an optional method parameter that can be either "linear" (the default) or "ranked"
sys module:
- activate_stack_trampoline(), deactivate_stack_trampoline(), is_stack_trampoline_active(), and last_exc added
- last_type, last_value, and last_traceback are now deprecated
sys.monitoring added
tarfile module: The TarFile.extract() and TarFile.extractall() methods now take an optional filter parameter for limiting what gets extracted in order to avoid security issues. Calling one of these methods without setting a filter will generate a DeprecationWarning until Python 3.14, in which the default filter will become "data".
tempfile module:
- NamedTemporaryFile now takes an optional delete_on_close parameter
- mkdtemp() now always returns an absolute path
types module: get_original_bases() added
typing module:
- override added
- dataclass_transform() now takes an optional frozen_default parameter
- Hashable and Sized are now deprecated
unicodedata updated to Unicode 15.0.0
unittest command-line interface: --durations option added
uuid module: Command-line interface added
venv module: setuptools is no longer automatically installed in virtual environments

Special Version Control Files

2021-06-22T00:00:00-04:00

The following is a list of VCS-specific files that one may find in the working directories of common version control systems. This list is useful for, say, knowing what files to ignore when traversing a project directory.

Did I leave anything out? Feel free to send a pull request.

VCS	File/Directory	Purpose	Documentation
Git	`.git/`	Repository data	gitrepository-layout(5)
	`.gitattributes`	Defines per-path attributes	gitattributes(5)
	`.gitignore`	Lists files to exclude from version control	gitignore(5)
	`.gitmodules`	Defines submodules	gitmodules(5)
	`.mailmap`	Maps names & e-mails to canonical values	gitmailmap(5)
Mercurial	`.hg/`	Repository data	wiki:FileFormats
	`.hgignore`	Lists files to exclude from version control	hgignore(5)
	`.hgsigs`	Contains changeset signatures from the GPG extension	—
	`.hgtags`	Defines tags	—
Darcs	`_darcs/`	Repository data	—
	`.binaries`	Lists files to treat as binary [1]	Manual
	`.boring`	Lists files to exclude from version control [1]	Manual
Bazaar	`.bzr/`	Repository data	—
Bazaar	`.bzrignore`	Lists files to exclude from version control	bzr ignore
Subversion	`.svn/` or `_svn/` [2]	Local copy of repository data [3]
CVS	`CVS/`	Local copy of repository data (one per directory)	Manual
RCS	filename`,v`	Version history for filename	Manual
RCS	`RCS/`	Conventional directory for storing `*,v` files	—

[1]	(1, 2) Darcs normally stores its binaries file and boring file inside the `_darcs/` directory, but it is possible to use any file under version control in their place; the names listed here are the conventional names for such files.

[2]	For compatibility with ASP.NET, Subversion will use the name `_svn` instead of `.svn` if the `SVN_ASP_DOT_NET_HACK` environment variable is defined.

[3]	Prior to version 1.7, Subversion placed a `.svn`/`_svn` directory in every subdirectory of the working directory under version control.

Integrating auto with bump2version

2021-02-11T00:00:00-05:00

auto by Intuit lets you set up automatic creation of tags & releases and population of changelogs in a GitHub project. It takes care of determining the version number for new releases, but, by default, it does not set the new version number in your code. This isn’t a problem if your project uses something like setuptools_scm or versioneer to fetch the version number from Git, but if your project’s version number is hardcoded in your code, you’ll need another solution. bump2version is that solution, and it can be integrated into auto as follows.

Set up `bump2version`

To make your project’s repository usable with bump2version, create a .bumpversion.cfg (no “2”!) file as follows:

[bumpversion]
# Replace the value here with your project's current version number:
current_version = 0.1.0

commit = True

# If integrating with GitHub Actions, you'll need to include "[skip ci]" in
# the bump2version commit message in order to prevent auto from running
# unnecessarily:
message = [skip ci] Bump version: {current_version} → {new_version}

# auto will be taking care of the tagging for us, so set `tag` to False:
tag = False

# For each file in your code that contains your project's version number,
# add a `[bumpversion:file:PATH]` section header, like so:
[bumpversion:file:src/myproject/__init__.py]

With this file in place and bump2version installed, you can automatically update your project’s version number by running bump2version $PART, where $PART is the part of the version number to increase (major, minor, or patch).

See the bump2version documentation for more information.

Integrate with `auto`

I assume you’ve already set up auto for your repository. To get auto to run bump2version automatically at the right time when creating a new release, use the exec plugin to register an afterChangelog hook. Add the following to the "plugins" field in your repository’s .autorc file:

{
    ...
    "plugins": [
        ...
        [
            "exec",
            {
                "afterChangelog": "bump2version \"$(printf '%s\n' \"$ARG_0\" | jq -r .bump)\""
            }
        ]
    ]
}

Tip

Despite what the exec documentation may imply, the $ARG_0 environment variable is a JSON object containing the payload passed to the respective hook. This means we must extract the semantic version part to bump using jq.

Warning

Do not use echo in place of printf here! The $ARG_0 variable contains JSON containing strings with \n escape sequences, which echo would convert into newlines, resulting in invalid JSON.

With this setting in place, auto will run bump2version on each new release after populating the changelog, and the commit created by bump2version will be the commit to receive the new version tag.

Integrate with GitHub Actions

Assuming you’ve already set up a GitHub Actions workflow for running auto (see here for an example), the only addition needed to support bump2version is to install it before running auto:

- name: Set up Python
  uses: actions/setup-python@v2
  with:
    python-version: '^3.6'

- name: Install bump2version
  run: python -m pip install bump2version

Using Package Data in Python Projects with Setuptools

2020-12-29T00:00:00-05:00

When creating a Python project, you may want to include a number of non-Python files in the project that the code can then access at runtime, such as templates, images, and data. These files are called package data [1], and they are included in & accessed from your project as follows.

Contents

Including Package Data in Projects
Accessing Package Data at Runtime
- Installing & Importing importlib-resources
- The importlib-resources API
Footnotes

Including Package Data in Projects

Note

The following section only pertains to projects built with setuptools. Users of flit, Poetry, and other tools must consult their respective backend’s documentation to see how to include package data.

The first requirement for being able to include package data in your package [2] is for the files to be located somewhere inside the actual package directory. If your project is a single-file .py module, you must convert it to a directory/package layout before you can store package data in it.

Once your package data is inside the package, you must tell setuptools to include the files in both the project’s wheel (so that the files will be installed when your package is installed) and sdist (so that the files can be included in the wheel when building from an sdist). There are two different major ways to do this: a blanket “include all” method that uses a MANIFEST.in file and a fine-grained method that configures one package & subpackage at time via the setup.py/setup.cfg file.

Important

In addition to the configurations described here, you also need to ensure that all folders within your package that contain package data (even if they contain no Python source code) are listed in the packages option in setup.py/setup.cfg. This can be done automatically by using the setuptools.find_namespace_packages() function instead of find_packages(). If you don’t do this, setuptools will emit a warning message starting in version 62.3.0, and later versions will cease to include such packages in your distribution at all.

Including Package Data via `MANIFEST.in`

There are two steps to this method:

Create or edit your project’s MANIFEST.in file with the proper commands to tell setuptools to include all of your package data file in the project’s sdists. See the Python Packaging User Guide for information on how to do this.

In the simplest case — in which you want all non-Python files in your package to be treated as package data — you can simply include your entire package in your sdist with the MANIFEST.in command “graft packagename” (or “graft src” if you’re using a src/ layout). If you use this command, you should also follow it with “global-exclude *.py[cod]” so that compiled Python bytecode files are not included in the sdist.
In your setup.py, pass include_package_data=True to the setup() function. If you are placing your project configuration in setup.cfg instead, set include_package_data = True in the [options] section of setup.cfg.

Doing this tells setuptools that any non-Python files found inside your package that are included in the sdist should also be included in the wheel.

Including Package Data via `setup.py`/`setup.cfg`

This method allows you to specify package data file patterns on a per-package basis through the package_data argument to setup(). package_data must be a dict mapping package or subpackage names (or the empty string, to specify all packages & subpackages) to lists of glob patterns indicating which files within those packages to include in the sdist and wheel.

A sample package_data looks like this:

package_data={
    # If any package or subpackage contains *.txt or *.rst files, include
    # them:
    "": ["*.txt", "*.rst"],
    # Include any *.msg files found in the "hello" package (but not in its
    # subpackages):
    "hello": ["*.msg"],
    # Include any *.csv files found in the "hello.utils" package:
    "hello.utils": ["*.csv"],
    # Include any *.dat files found in the "data" subdirectory of the
    # "mypkg" package:
    "mypkg": ["data/*.dat"],
}

If you are placing your project configuration in setup.cfg, you must instead specify package_data via an [options.package_data] section in which the keys are the package & subpackage names — using * instead of the empty string to signify all packages — and the values are comma-separated glob patterns. The above setup.py sample translates to setup.cfg as follows:

[options.package_data]
# If any package or subpackage contains *.txt or *.rst files, include them:
* = *.txt, *.rst
# Include any *.msg files found in the "hello" package (but not in its
# subpackages):
hello = *.msg
# Include any *.csv files found in the "hello.utils" package:
hello.utils = *.csv
# Include any *.dat files found in the "data" subdirectory of the "mypkg"
# package:
mypkg = data/*.dat

Note that glob patterns only select files located directly within the given package (or in the given subdirectory of the package, if the pattern includes a directory path); e.g., "hello": ["*.msg"] selects *.msg files in the hello package but not in any of its subpackages. To select files in subpackages, you must either include an entry for each subpackage or else use the empty string key (or asterisk key in setup.cfg) to specify a pattern for all packages & subpackages.

If a pattern contains any directory components, the forward slash (/) must be used as the directory separator, even on Windows.

If a package data file is located in a directory that does not have an __init__.py file (say, a data/ directory inside package.subpackage), that directory does not count as a package, and the file must be listed in package_data in the form "package.subpackage": ["data/PATTERN"].

Warning

If you use both include_package_data and package_data, files specified with package_data will not be automatically included in sdists; you must instead list them in your MANIFEST.in.

Excluding Package Data via `setup.py`/`setup.cfg`

The exclude_package_data argument to setup() can be used in conjunction with either of the above methods to prevent one or more files from being treated as package data. exclude_package_data takes a dict with the same structure as package_data, and any matched files are excluded from wheels. Matched files are also excluded from sdists if they are not already matched by the project’s MANIFEST.in.

In a setup.cfg, exclude_package_data becomes an [options.exclude_package_data] section whose contents have the same structure as [options.package_data].

Including Package Data via Setuptools Plugins

As an alternative to the above methods, you can use a plugin for setuptools that automatically recognizes & includes package data in sdists & wheels, usually based on what files in the project directory are under version control. One example of such a plugin is setuptools_scm, which automatically finds all files under version control in a Git or Mercurial repository and augments the project’s MANIFEST.in (if any) with the found files. This eliminates the need to write a MANIFEST.in manually (unless there are files under version control that you want to exclude from sdists or wheels), though you still need to set include_package_data to True for files in your package directory to be included in wheels.

Accessing Package Data at Runtime

There have been multiple ways to access package data over the years, from pkg_resources’ ResourceManager API to pkgutil.get_data(), but the most recent and currently-recommended way is with the importlib-resources package.

Installing & Importing `importlib-resources`

There are two versions of importlib-resources available:

The one on PyPI that is installed with pip install importlib-resources and imported with import importlib_resources (note underscore)
The one in the Python standard library starting with Python 3.7 that is imported with import importlib.resources (note period)

Development of the PyPI version tends to be ahead of whatever’s in the latest Python version. In particular, the new files()-based API described here was only introduced in version 1.1.0 of the PyPI project and was only added to the Python standard library in Python 3.9. In order to be guaranteed a version of importlib-resources that supports this API, you should add the following to your project’s install_requires:

importlib-resources>=1.1.0; python_version < '3.9'

and import importlib-resources in your code as follows:

import sys

if sys.version_info < (3, 9):
    # importlib.resources either doesn't exist or lacks the files()
    # function, so use the PyPI version:
    import importlib_resources
else:
    # importlib.resources has files(), so use that:
    import importlib.resources as importlib_resources

The `importlib-resources` API

To access a package data file in your project, start by calling importlib_resources.files() on the name of your package:

pkg = importlib_resources.files("packagename")
# The argument can optionally refer to a subpackage in the form
# "packagename.subpackage".

This gives you a Traversable object that acts like a limited pathlib.Path object for traversing package data files. To refer to a data.csv file in a data/ directory in your package, write:

pkg_data_file = pkg / "data" / "data.csv"

So now that we’ve got a reference to the package data file, how do we get anything out of it?

To open the file for reading, call the open() method:

with pkg_data_file.open() as fp:
    # Do things with fp

To get the file’s contents as bytes, call the read_bytes() method:
```
b = pkg_data_file.read_bytes()
```
To get the file’s contents as a str, call the read_text() method, optionally with an encoding argument:
```
s = pkg_data_file.read_text(encoding="utf-8")
```
To get the path to the file, call importlib_resources.as_file() on it and use the return value as a context manager:
```
with importlib_resources.as_file(pkg_data_file) as path:
    # Do things with the pathlib.Path object that is `path`
```
The use of context managers allows importlib-resources to support packages stored in zipfiles; when a path is requested for a package data file in a zipfile, the library can extract the file to a temporary location at the start of the with block and remove it at the end of the block.
To iterate through a directory (either a package or a non-package directory), use the iterdir() method. You can test whether a resource is a directory or a file with the is_dir() and is_file() methods, and you can get a resource’s basename via the name property:
```
for entry in (pkg / "data").iterdir():
    if entry.is_dir():
        print(entry.name, "DIR")
    else:
        print(entry.name, "FILE")
```

Footnotes

[1]	Specifically, package data files are files that are stored in a Python project’s package directory next to the Python source files. An alternative to package data is data files, which are files installed elsewhere on the file system. This article only deals with the former.

[2]	Throughout this article, the term “package” is used in the sense of a directory of `.py` files and other packages (a.k.a. an “import package”), not in the sense of a “distribution package” (i.e., an sdist or wheel).

Running Extra Steps after Releasing with auto in GitHub Actions

2020-10-26T00:00:00-04:00

Let’s say you’ve set up auto for your project via a GitHub Actions workflow, and now you want that workflow to carry out additional steps — such as building & uploading assets — whenever auto creates a new release. Let’s also say that none of the available plugins for auto covers your use-case and you’re not a JavaScript programmer, so you won’t be writing a new plugin to do what you want. How do you adjust your GitHub Actions workflow to run these extra steps at the right time? Read on to find out.

So you’ve got a GitHub Actions workflow for auto already set up. Presumably, it looks something liks this:

name: Auto-release on PR merge

on:
  # This is the closest thing to triggering on a PR merge, as long as you
  # don't push directly to master.
  push:
    branches:
      - master

jobs:
  auto-release:
    runs-on: ubuntu-latest
    if: "!contains(github.event.head_commit.message, 'ci skip') && !contains(github.event.head_commit.message, 'skip ci')"
    steps:
      - name: Checkout source
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Download latest auto
        run: |
          auto_download_url="$(curl -fsSL https://api.github.com/repos/intuit/auto/releases/latest | jq -r '.assets[] | select(.name == "auto-linux.gz") | .browser_download_url')"
          wget -O- "$auto_download_url" | gunzip > ~/auto
          chmod a+x ~/auto

      - name: Create release
        run: ~/auto shipit
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Now you want to insert further steps that will be triggered whenever auto shipit successfully creates a new release.

For simple cases — where the commands to run can be expressed in a single shell command — we can use auto’s exec plugin to run the desired commands via the afterRelease hook. For example, building & uploading a Python package for PyPI can be integrated into auto by adding the following item to the "plugins" field in the repository’s .autorc file:

{
    ...
    "plugins": [
        ...
        [
            "exec",
            {
                "afterRelease": "python -m build && twine upload/*"
            }
        ]
    ]
}

(This particular example assumes that the appropriate Python dependencies are already set up earlier in the workflow and that the twine username & password are passed as environment variables to the ~/auto shipit step.)

For more complex post-release activity which can only be implemented as GitHub Actions steps, we need something else. Just adding the steps and nothing else directly to the workflow won’t work, as auto shipit doesn’t always create a new release, such as when a pull request with a skip-release label is merged, or when a pull request without a release label is merged while onlyPublishWithReleaseLabel is set to true. So we need some logic to test whether there’s a new release.

Theoretically, one option would be to create a separate workflow that runs whenever a new tag is pushed or a new GitHub release is created, but this won’t work with an out-of-the-box setup; auto uses GITHUB_TOKEN to create the tag & release, and GitHub Actions workflows specifically do not trigger on events performed with a GITHUB_TOKEN. You could get this approach to work by passing auto a personal access token instead of GITHUB_TOKEN, but there’s a more direct way to make this work instead that also lets you keep your release-related workflow steps in a single file.

Here’s the key trick: By running auto version before auto shipit, you find out whether auto is about to create a new release, and you can save the output from auto version to use to trigger extra steps later. If auto is about to create a new release, auto version will output the release level (”patch”, “minor”, or “major”); otherwise, if auto shipit would do nothing, auto version just outputs an empty line.

Hence, insert the following step before the auto shipit step:

- name: Check whether a release is due
  id: auto-version
  run: |
    version="$(~/auto version)"
    echo "version=$version" >> "$GITHUB_OUTPUT"
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Here, we use a workflow command to make the output from auto version available to subsequent steps. Later steps can then be configured to only run if a new release is being made by adding an if field to them, like so:

- name: Build asset for new release
  run: ...
  if: steps.auto-version.outputs.version != ''

If you have multiple steps that you want to run after a release, adding an if field to all of them can become excessive; isn’t there a way to apply an if to a whole block of steps? There is indeed; you can split off the extra steps into a separate job in the same workflow and have that entire job be guarded by a single if.

First, in order to make the output from the auto version step available to other jobs in the workflow, you need to add an outputs field to the original auto job (the one that in the example above is named “auto-release”), at the same level as the runs-on and steps keys. This outputs field should contain a YAML object mapping a name for the output value to a ${{ steps.….outputs.… }} expression that evaluates to the output from the auto version step. For the example workflow configurations shown so far, this would mean a configuration like the following:

jobs:
  auto-release:
    runs-on: ubuntu-latest
    if: "!contains(github.event.head_commit.message, 'ci skip') && !contains(github.event.head_commit.message, 'skip ci')"
    ### vv Add this bit vv ###
    outputs:
      auto-version: ${{ steps.auto-version.outputs.version }}
    ### ^^ Add this bit ^^ ###
    steps:
      # ...

With this in place, a new job can be added to the workflow containing all the steps you want to run after a new release is made. This new job needs two special fields (at the same level as runs-on and steps):

A needs field containing the job ID of the auto job (so needs: auto-release for the examples given here) to declare a dependency on it
An if field containing an expression of the form needs.AUTO_JOB_NAME.outputs.AUTO_VERSION_OUTPUT_NAME != '' (so if: needs.auto-release.outputs.auto-version != '' for the examples given here); this causes the job to be skipped if auto version outputs nothing, i.e., if no release is made

The configuration for this new job would then look like:

jobs:

  # `auto release` job from above omitted
  # ...

  build-and-publish:
    runs-on: ubuntu-latest
    needs: auto-release
    if: needs.auto-release.outputs.auto-version != ''
    steps:
      # ...

There’s one more thing to be aware of: If you check out your repository in this new job, by default the HEAD will be the commit that triggered the workflow originally and will not include the changelog commit or tag created by auto. If you need the commit or tag (say, because your project uses setuptools_scm or the like to derive its version number from Git tags at build time), you can tell the actions/checkout action to check out the latest commit from the repository by passing the default branch as the ref input like so:

- name: Checkout source
  uses: actions/checkout@v4
  with:
    ref: master  # or `main` or whatever your default branch is
    # This setting is needed to fetch tags:
    fetch-depth: 0

This does come with a caveat, though: in the event that multiple commits or merges to the default branch were made in quick succession, you may end up checking out a later commit than the tag that auto created. If this is a problem, one way to deal with it is to specifically check out the tag for the latest GitHub release, like so:

- name: Get tag of latest release
  id: latest-release
  run: |
    latest_tag="$(curl -fsSL https://api.github.com/repos/$GITHUB_REPOSITORY/releases/latest | jq -r .tag_name)"
    echo "tag=$latest_tag" >> "$GITHUB_OUTPUT"

- name: Checkout source
  uses: actions/checkout@v4
  with:
    ref: ${{ steps.latest-release.outputs.tag }}
    fetch-depth: 0

This, of course, fails if auto creates multiple tags in quick succession. I’m not aware of a decent way to deal with this eventuality; how about listening to the docs and just not running auto that often in the first place?

Using all of these tricks, your final workflow configuration should look something like this:

name: Auto-release on PR merge

on:
  # This is the closest thing to triggering on a PR merge, as long as you
  # don't push directly to master.
  push:
    branches:
      - master

jobs:
  auto-release:
    runs-on: ubuntu-latest
    if: "!contains(github.event.head_commit.message, 'ci skip') && !contains(github.event.head_commit.message, 'skip ci')"
    outputs:
      auto-version: ${{ steps.auto-version.outputs.version }}
    steps:
      - name: Checkout source
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Download latest auto
        run: |
          auto_download_url="$(curl -fsSL https://api.github.com/repos/intuit/auto/releases/latest | jq -r '.assets[] | select(.name == "auto-linux.gz") | .browser_download_url')"
          wget -O- "$auto_download_url" | gunzip > ~/auto
          chmod a+x ~/auto

      - name: Check whether a release is due
        id: auto-version
        run: |
          version="$(~/auto version)"
          echo "version=$version" >> "$GITHUB_OUTPUT"
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Create release
        run: ~/auto shipit
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  build-and-publish:
    runs-on: ubuntu-latest
    needs: auto-release
    if: needs.auto-release.outputs.auto-version != ''
    steps:
      - name: Get tag of latest release
        id: latest-release
        run: |
          latest_tag="$(curl -fsSL https://api.github.com/repos/$GITHUB_REPOSITORY/releases/latest | jq -r .tag_name)"
          echo "tag=$latest_tag" >> "$GITHUB_OUTPUT"

      - name: Checkout source
        uses: actions/checkout@v4
        with:
          ref: ${{ steps.latest-release.outputs.tag }}
          fetch-depth: 0

      # Remaining steps go here
      # ...

Enjoy!

Common Python Packaging Mistakes

2020-08-22T00:00:00-04:00

I think we can all agree that packaging a Python project is harder than it should be. With numerous guides & tutorials out there, people still make mistakes. Some of these mistakes break a project, some just make it less attractive, and some even cause a project to step on the toes of other projects.

As the admin of the wheel-analysis and -browsing site Wheelodex, I see a number of poorly-built wheels each morning as I peruse the day’s new entries. This eventually motivated me to create check-wheel-contents — a program for scanning a wheel for many of the below problems plus several others — in an attempt to get people to clean up their wheels, yet still the poorly-packaged projects persist.

In yet another attempt to get people to fix their broken packages, here now are some of the more frequent types of mistakes I see — along with advice on how to avoid & correct them — in no particular order.

Note

Unless otherwise specified, references to packaging configurations assume that the project is using setuptools and that configuration is being placed in setup.py instead of setup.cfg. Consult the appropriate documentation if your project is structured differently.

Contents

Avoiding General Mistakes
- Look at Your Built Distributions
- Installations are Not Namespaced
Top-Level tests/ Directory in Wheel
Top-Level README or LICENSE File in Wheel
Project Description Doesn’t Render
Project Description Not Included
Python Package Not Included in Wheel
Subpackages Not Included in Wheel
Package Data Not Included in Wheel
*.pyc Files Included in Wheel
Rebuilding Wheels without Deleting build/
Pinning Project Requirements to Exact Versions
Conclusion
Footnotes

Avoiding General Mistakes

First, some general advice that will help you avoid (or at least detect) the vast majority of packaging errors.

Look at Your Built Distributions

Before you upload your sdist and wheel (especially if it’s your first release for the project in question), actually take a look at what files are inside and make sure that everything you want to include is in there. An sdist is either a tarball (*.tar.gz) or a zipfile (*.zip), so its contents can be listed with one of these two commands:

# Tarball
$ tar ztf projectname-version.tar.gz

# Zipfile
$ zipinfo projectname-version.zip

At time of writing, the exact layout of an sdist has yet to be standardized, but if you’re building with a recent version of setuptools, the contents are structured as follows:

Everything in the sdist is inside a top-level directory named {projectname}-{version}/. This directory contains a copy of your package code, the project’s setup.py/pyproject.toml file, and various other files from your project directory; see here for more information on what gets included by default.
There exists a PKG-INFO file containing the project metadata. For historical reasons, this does not include project dependencies.
Next to your Python package, there is a {projectname}.egg-info/ directory containing more metadata, including a copy of PKG-INFO, a SOURCES.txt file listing the files in the sdist, and a requires.txt file listing your project’s dependencies. Other files may be present depending on what features your project uses.

If your sdist is missing some files from your project directory or contains some files that you don’t want in there, then (assuming you’re building your project with setuptools), you can adjust what gets included via a MANIFEST.in file and rebuild.

Note

Exactly what files should and should not be included in an sdist is, for the most part, largely a matter of opinion, but your sdist needs to include your Python code and anything from your project directory that’s needed to built a complete wheel. The files that setuptools includes by default should generally be left in there, and most people will recommend also including tests and documentation. Things that should generally be left out include *.pyc files, repository metadata like .gitignore and .hgtags, and (except in special circumstances) anything that you wouldn’t commit to version control.

Wheels (*.whl), meanwhile, are just zipfiles with a funny extension, so you can list their contents with zipinfo. The basic layout of a wheel is as follows:

Your Python package is located at the root of the wheel, rather than inside a directory.
There exists a {projectname}-{version}.dist-info/ directory containing metadata: a METADATA file describing the project (similar to an sdist’s PKG-INFO, but including dependencies), a WHEEL file describing the wheel version and tags, and a RECORD file listing the files in the wheel and their hashes. Licenses included in the wheel with the license_files setting also end up in this directory. Other files may also be present depending on what features your project uses and the versions of setuptools and wheel used to build the wheel.
If your project includes any files that are installed outside of site-packages — headers, scripts, data files (not to be confused with package data), etc. — they are stored in a {projectname}-{version}.data/ directory. Files in this directory are organized into subdirectories named after the distutils scheme keys (purelib, platlib, headers, scripts, or data) that map to the files’ install locations.

Note

Aside from the *.dist-info/ and *.data/ directories, a wheel should only contain Python packages & modules, consisting of code and data files. Files like your project’s setup.py, pyproject.toml, setup.cfg, etc. do not belong in a wheel.

Controlling what gets included in a wheel is more involved than for an sdist (when using setuptools, at least); consult documentation elsewhere on how to do this.

Warning

Do NOT try to fix your sdists or wheels by manually adding, removing, or editing the files inside, as this is likely to make the sdist/wheel invalid. Instead, change your project configuration and create new built distributions until you get what you want — and be sure to delete the build/ directory in between builds!

Installations are Not Namespaced

A key thing to understand about how Python packages are installed is that (almost) all of the files in a wheel are simply placed directly in site-packages/; the only subdirectories present will be the directories that are already in the wheel. This means that, if your wheel has a foo/ directory at the top level containing bar.py, then bar.py will be installed to site-packages/foo/bar.py; nothing is added to the path to separate it from other packages’ foo/bar.py files. Properly namespacing your files must be done by putting everything under a directory (normally your top-level Python package) with a name the same as or similar to the name of your project — which is the standard practice anyway. Where problems arise is when a top-level file or directory in a wheel has a name that other projects are also likely to use, in which case files end up overwritten with the wrong content and bugs result.

See pip issue #4625 for pip’s attempts at handling file collisions whenever they arise.

Top-Level `tests/` Directory in Wheel

The first (and probably most common) Python packaging mistake occurs when you put your tests in a tests/ directory at the root of your project (outside of your Python package) and then include this directory in your project’s wheels. The tests/ directory then ends up placed at the top-level of your wheel’s filesystem, and, as stated above, this means that it will be installed at site-packages/tests/. The problem comes from the fact that “tests/” is a name that everybody uses for their tests and too many other projects also include a top-level tests/ directory in their wheels. As a result, site-packages/tests/ becomes a mish-mash of code from different packages, some files even overwriting each other, and if someone tries to run those tests, chaos will ensue. (And if you don’t expect people to be running your tests from your installed project, why are you including tests in the wheel in the first place?)

The most common reason why tests/ ends up included in wheels is because the project’s author used find_packages() in their setup.py but didn’t use the function’s exclude argument. find_packages() works by searching for directories in the project root (or in the directory passed as the where argument) that contain an __init__.py file, and then it searches those directories for any subdirectories that also contain an __init__.py file, and so on until it runs out of directories with __init__.py files. Sometimes, people put an __init__.py file in tests/ (Whether this is necessary depends on the test framework being used), and so find_packages() with the default arguments picks it up and adds it to the project’s list of packages, resulting in it being included in the wheel.

To avoid this, you have five options:

Remove the __init__.py files from your tests/ directory and its subdirectories. Whether this is doable depends on your test framework.
Use find_packages()’s exclude argument to exclude tests/ and its subdirectories like so:
```
packages=find_packages(exclude=["tests", "tests.*"])
```
Note that we list both "tests" and "tests.*". Listing just "tests" would exclude tests/ but not its subdirectories, so we need to also list "tests.*" in order to exclude everything.
Use find_packages()’s include argument to include only your Python package and its subpackages like so:
```
packages=find_packages(include=["packagename", "packagename.*"])
```
As with exclude, we list both the package name and the package name followed by “.*” so that all subpackages of the package will be matched & included.
Move your tests/ directory inside your Python package directory so it’s no longer at the top level.
Switch your project to a src/ layout, where your Python package directory is located inside a directory named src/ and everything else — including tests/ — is outside of src/. With this layout, simply write your packages line as packages=find_packages("src"), and find_packages() will only look at what’s in src/.

Note that you will also need to add package_dir={"": "src"} to your setup() arguments in order for setuptools to grok your layout. More information about the src/ layout can be found here and here.

The second most common reason why tests/ ends up in wheels is that the project author used the exclude argument to find_packages() but listed only "tests" and not "tests.*", and so the subdirectories of tests/ (inside an otherwise-empty tests/ directory) ended up in the wheel. Both "tests" and "tests.*" need to be included in the exclude list in order to exclude the entire tests/ hierarchy.

Besides tests/, it is also a problem to include a top-level directory named test/ (singular), docs/, examples/, data/, or similar, as such directories are also often included in wheels despite the clashes that will result.

Do note that, when it comes to sdists, it’s perfectly fine to have a tests/ etc. directory at the base of your project, as sdists themselves are not installed, they’re just used to build wheels, which are what actually get installed.

Top-Level `README` or `LICENSE` File in Wheel

Similarly to the above mistake involving tests/, it is also a bad idea to include your project’s README.rst/README.md or LICENSE file (or CHANGELOG or really anything that’s not a Python module or *.pth file) at the root of your wheel, as it will collide with the READMEs and LICENSEs of other projects that do the same thing.

This mistake is particularly common among projects built using Poetry, where simple usage of the include option adds files directly into both the sdist and wheel. To include a file in only the sdist, one needs to change the include option from this form:

[tool.poetry]
include = ["CHANGELOG.md"]

to this form:

[tool.poetry]
include = [
    { path = "CHANGELOG.md", format = "sdist" }
]

If you do want to include your README or LICENSE in your wheel, the correct way is as follows:

For README, the file’s contents should already be used as the project’s (long) description, in which case the contents are already included in the project metadata, which is stored in PKG-INFO (for sdists) or *.dist-info/METADATA (for wheels), and thus there is no need to include the README as a separate file. If you need to be able to retrieve the README’s contents at runtime, this can be done by using importlib.metadata or similar to fetch the project’s description.
Licenses and related files belong inside a wheel’s *.dist-info directory. If using setuptools with wheel 0.32 or higher, licenses can be placed there by passing them to the [metadata]license_files option in setup.cfg; see the wheel documentation for more information.

At time of writing, Poetry does not support adding license files to a wheel’s *.dist-info directory, but PR #1367 would change that.

Project Description Doesn’t Render

The Python Package Index (PyPI) supports project (long) descriptions written in three possible formats: reStructuredText (the default if no format is specified), Markdown (either GitHub Flavored Markdown or CommonMark), and plain text. Markdown and plain text are lenient formats; anything you write in them is valid. However, documents written in reStructuredText can be malformed, producing errors & warning messages when rendered. When a project with a malformed reStructuredText description (either because it uses reStructuredText incorrectly or because it’s actually Markdown that wasn’t declared as Markdown) is uploaded to PyPI, PyPI does one of the following two things:

If the project does not declare a Content-Type for its malformed description, PyPI will fall back to displaying the source of the description as though it were plain text.
If the project explicitly declares the malformed description’s Content-Type as reStructuredText (i.e., as the MIME type text/x-rst), PyPI will reject the upload.

Neither situation is desirable, but at least the latter gives you the chance to correct your project description before it’s released on PyPI, while the former situation means your project’s PyPI page shows an ugly, unprofessional-looking description until you make a new release.

Note

When using setuptools, you may find that your project’s long description has been mangled somewhat, with a bunch of “Field: Value” entries added to the bottom and various information missing from the listing on the left side of the PyPI project page. This happens whenever you include a newline in your project’s summary/short description, thereby triggering setuptools bug #1390. Always make sure that no newlines end up passed to the description argument of setup()!

There are two things you can do to avoid uploading a project with a malformed description to PyPI:

Set your description’s Content-Type appropriately. If you’re using reStructuredText, this will cause PyPI to reject any uploads with malformed project descriptions. If you’re not using reStructuredText, setting the Content-Type is necessary in order for your description to be rendered properly.

The content types for the supported formats are as follows:

reStructuredText: text/x-rst

Markdown (GitHub Flavored Markdown): text/markdown or text/markdown; variant=GFM

Markdown (CommonMark): text/markdown; variant=CommonMark

Plain text: text/plain

If your project is built using setuptools, you set the description’s Content-Type by setting the long_description_content_type argument to setup() to the appropriate value from the above table. Note that this requires setuptools 36.4.0 or higher in order to work (or 38.3.0 or higher if you’re setting it in setup.cfg).
Run the twine check command from twine on your sdist and wheel before uploading them. This command checks whether your project description can be rendered on PyPI before you actually upload it.

reStructuredText:	`text/x-rst`
Markdown (GitHub Flavored Markdown):	`text/markdown` or `text/markdown; variant=GFM`
Markdown (CommonMark):	`text/markdown; variant=CommonMark`
Plain text:	`text/plain`

Project Description Not Included

It’s just embarrassing when this happens. A project without a long description just looks completely pointless; how am I supposed to know what it does or how to use it? Sadly, too many projects on PyPI lack long descriptions. Did the developer not care enough to write even a README? Did the developer forget to use the README as the long description or not know they had to?

If your project’s got a README — and really, a project that doesn’t have one isn’t ready to be released — and it’s written in reStructuredText, Markdown, or plain text (a safe bet), you can (and should) use it as your project’s long description by adding the following or similar to your setup.py:

with open("README.extension", encoding="utf-8") as fp:
    long_description = fp.read()

setup(
    ...
    long_description = long_description,
    ...
)

If your project isn’t in reStructuredText, you’ll also need to set long_description_content_type to the appropriate value in the table above so that the description renders properly on PyPI.

Python Package Not Included in Wheel

If not having a description is embarrassing, not having any code in your wheel is crippling. With a wheel like this, when people install your project, they get nothing! That’s certainly not what you want, is it?

Possible reasons why this can happen include:

You’re using find_packages() to autolocate your project’s packages, but you failed to add an __init__.py file to the top-level package (and possibly also some subpackages). Solution: Add that __init__.py.
- If your intention is to leave out the __init__.py file in order to create a namespace package, you’ll need to use find_namespace_packages() instead.
Your project’s code is a single Python module (as opposed to a directory of modules) and you’re using the packages argument to setup() and/or find_packages() in an attempt to declare the module to setuptools. This is wrong. When your project is a single Python module, instead of the packages argument, you need to use the py_modules argument. Set py_modules to a list of strings where each string is the name of a top-level Python module without the “.py” extension. (Usually, you’ll just have one module to list here.) You can’t use find_packages() for this.

If your project includes any tests (which it should), you can implicitly test that your wheel contains your project code by testing against the installed version of your project instead of the copy in your repository. To do this, pip-install your package (ideally in a virtualenv, and not in development/editable mode!) before running the tests and ensure that the directory containing the repository copy of your code is not in sys.path when the tests run. Tox can help with the first part. The second part depends in part on your test framework, but you can guarantee your tests aren’t picking up the local copy by switching to a src/ layout (see above). With these two things in place, your tests will be forced to import your package from site-packages, where it’s in a form determined by the contents of the project’s wheel. If your wheel is missing code and your tests try to import that code, you’ll get an error when the tests run, and you’ll know that you need to fix something.

Subpackages Not Included in Wheel

Sometimes, a project’s top-level package directory and the files within get included in a wheel, but the subdirectories and their contents get left out. Admittedly, I don’t know how common this is, as you can’t determine whether a wheel is missing subpackages just by looking at its contents unless you also know what’s in the project’s repository. However, it’s an easy thing to mess up, and various packaging articles I’ve read frequently make reference to this problem, so it can’t be that uncommon.

There are two major reasons why one or more of your Python package’s subpackages might be omitted from wheels:

You’re passing a list of packages to the packages argument to setup() and the list fails to include every package & subpackage in your project. If your project’s top-level package is named “foo” and it contains two subdirectories named “bar” and “baz” that contain (directly or indirectly) Python source files, then bar and baz are subpackages of foo, and they all need to be included in the packages list:
```
packages=["foo", "foo.bar", "foo.baz"]
```
If baz contains another directory named “glarch” that contains more Python source files, then "foo.baz.glarch" needs to be included in the list as well, and so on.

Note that directories that only contain data files and no Python source files do not count as packages and should not be passed to the packages argument. They are instead package data directories; see below for advice on dealing with them.

Of course, a simple alternative to listing every package explicitly is to just use the find_packages() function, which brings us to cause #2 …
You’re using find_packages() to autolocate your project’s packages, but you failed to add an __init__.py file to one or more subpackages. find_packages() only counts something as a package if it contains an __init__.py file, so you need to include that file in any subdirectory of your Python package that contains Python source files or contains a directory that contains Python source files.

As with omitting the package entirely from the wheel, proper testing practices can let you know when this happens in advance of a release.

Package Data Not Included in Wheel

Sometimes, you want to include non-Python data or resource files inside a Python package so that they can be used at runtime, but sometimes those files fail to end up in the final wheel. Like the omission of subpackages, it’s hard to know just how common this is, but even experienced Python programmers have made mistakes with package data configurations on occasion. This also happens to be yet another situation where testing the installed version of your code will help you out.

Setuptools provides two ways to specify package data. The first way is to configure MANIFEST.in so that the desired package data files are included in the sdist and then pass include_package_data=True to setup() so that all files inside the Python package that are included in the sdist are also included in the wheel. Pretty much the only way to make a mistake here is by not matching all of the files you want with MANIFEST.in commands; consult this reference if you run into problems.

The second way to specify package data is with the package_data argument to setup(). This argument takes a dict mapping package & subpackage names to lists of glob patterns defining what package data files to include in sdists & wheels. The biggest gotcha with this method is the fact that each glob pattern is only applied to the corresponding package and not any of its subpackages. This means that, with a package_data like this:

package_data={
    "package": ["*.txt"],
}

*.txt files in package will be recognized as package data and included in the sdist & wheel, but *.txt files in package.subpackage will not. To include *.txt files in package.subpackage, you’ll need to either add a "package.subpackage": ["*.txt"] entry to package_data or else include all *.txt files in all packages & subpackages by using the empty string as a key: "": ["*.txt"].

No matter which method you choose, be sure to exclude *.pyc files from consideration as package data; see the next section for why.

Note that if you combine the two ways to specify package data by setting include_package_data=True while also using package_data, then the files matched by package_data will not be included in the sdist unless they’re already included by MANIFEST.in. Getting this wrong can cause wheels built from an sdist to lack package data files.

See “Including Data Files” in the setuptools documentation for more information.

`*.pyc` Files Included in Wheel

When a Python source file is imported into a Python process, a *.pyc file containing compiled bytecode is created and (in Python 3) stored in a __pycache__/ directory so that future imports of the same file will be faster. These *.pyc files use a format that is specific to the OS, Python implementation, and Python version, and so it is pointless to share them. They do not belong in wheels (especially considering that pip already generates a host-appropriate set of *.pyc files when it installs a wheel), and yet too often people distribute wheels with *.pyc files in them.

Probably the most common reason why *.pyc files end up in wheels is that the project’s MANIFEST.in file contains “graft packagename”, “graft src”, or a similar line and include_package_data=True is passed to setup(). With this configuration, all files in the Python package directory when the wheel is built are added to the wheel. To prevent *.pyc files from being added, “global-exclude *.pyc” or similar needs to be added to the MANIFEST.in, ideally at the end of the file.

Alternatively, if the project specifies its package data with the package_data argument, including a "*" pattern in the package_data mapping is liable to cause *.pyc files to be included in the wheel. They should be excluded from package data by setting exclude_package_data to a dict that maps the appropriate keys to ["*.pyc"].

Rebuilding Wheels without Deleting `build/`

You should have noticed when building your project’s wheels that, in addition to creating a dist/ directory containing the output wheel, setuptools also creates a build/ directory containing a couple directories and a copy of your code. This build/ directory is an intermediate stage in the process of assembling a wheel; you should exclude it from version control and feel free to delete it at any time. In fact, it’s a good idea to delete it before running the command to create a wheel, especially if you’ve moved or renamed any files or directories in your code since the last time you built a wheel.

Consider the following scenario:

You build a wheel for your project, and you leave the build/ directory lying around afterwards.
You move, rename, and/or delete some files in your Python package, perhaps even renaming the package itself.
You build the wheel again — and when you do so, setuptools copies your new package tree into build/. Files that existed the last time the wheel was built overwrite their old copies in build/ successfully, but any old paths that have since been removed remain in build/.
As a result, your wheel ends up containing a mixture of your new and old code. In the case where you renamed your package, the wheel will contain both the pre-rename package and the post-rename package next to each other in their entirety, so you wheel has double the code with half of it under the wrong name.

This is clearly not desirable. The solution is to always delete the build/ directory before building a wheel, such as by cleaning your repository with git clean or similar, or by running python setup.py clean --all [1].

An even worse situation occurs if your setup.py uses find_namespace_packages() without any arguments. In this case, if you rebuild your package without first deleting the build/ directory, find_namespace_packages() will notice the .py files in build/ and assume that build/ is a namespace package, and so it’ll include build/ in your wheels — which means that build/ gets copied into build/, resulting in multiple package hierarchies in your wheels, with the problem compounding the more times you build your project without deleting the build/ directory. This particular problem can be mitigated by using the where, exclude, and/or include arguments to find_namespace_packages(), which have the same meaning as for find_packages().

Pinning Project Requirements to Exact Versions

There are a number of projects on PyPI where the dependencies are all of the form “foo == 1.2.3”, as opposed to “foo >= 1.2.3”, “foo >= 1.2, < 2”, or just “foo”. This is called pinning requirements. This makes sense when you’re developing a Python application that will be the primary project in its environment (in which case you often won’t be uploading it to PyPI), but it doesn’t make sense when you’re distributing a library for others to use alongside other arbitrary libraries. For one thing, your library is almost certainly going to work just as well with version 1.2.4 of foo [2], so why leave it out? For another thing, if someone wants to use your library with its pinned foo requirement alongside other libraries, sooner or later they’ll run into a situation where they’re installing both it and another project that requires a different version of foo (maybe even differing by one micro version!), and then problems ensue [3]. True, clashes between version dependencies in disparate projects can’t be avoided 100%, but they can be made to occur far less often if projects require generous version ranges instead of specific versions.

A general way to construct a decent version range for a requirement is to first determine the lowest version of the dependency that has all of the features you need and then use this version as the requirement’s lower bound. If the dependency follows or approximates semantic versioning, use the next major version (or the next minor version, if pre-v1) as the (exclusive) upper bound. If the dependency uses something like calendar versioning instead, things are less clear, but my preference is to leave out the upper bound and afterwards keep abreast of any future changes to the dependency. If any versions of the dependency inside the requirement’s bounds have known bugs that interfere with your project’s behavior, feel free to exclude them by adding specifiers of the form != X.Y.Z to the version range.

Conclusion

I’m very disappointed in all of you for making these mistakes so often, and I hope this article makes at least one Python package less broken. (I’d prefer it if all broken packages were less broken, but I know not to get my hopes up.)

Admittedly, most of these mistakes are due to users not using or understanding setuptools properly (aside from a Poetry anti-pattern that sneaked in at #2). Though flit and Poetry may promise to fix setuptools’ usability issues, people keep on using setuptools, and it keeps on outsmarting them. Hopefully sites like the Python Packaging User Guide eventually expand & become mature enough in the near future to cover — if not all the edge cases — at least the best practices that avoid them.

Footnotes

[1]	Setuptools is currently trying to get people to move away from `setup.py` commands, so `setup.py clean` will be discouraged — and probably deprecated — at some indeterminate point in the future. Until that happens, though, don’t feel bad about using it if you need to.

[2]	Unless `foo` is an unpredictable, compatibility-breaking mess, in which case you should probably reconsider depending on it.

[3]

Currently, pip handles conflicting version requirements with a warning and picking one requirement to follow, but pip’s new dependency resolver due out in October 2020 (already available if you pass the right flag to pip) will react to such situations by searching for older versions of the installation candidates with non-conflicting requirements, and if it can’t find any, it errors out without installing anything.

The Sum of Raising the First $n$ Integers to a Given Power

2020-08-13T00:00:00-04:00

The closed-form expression for $\sum_{i=1}^n i^m$ for a given $m\in\mathbb{Z}^+$ can be derived — once we know the formulas for all lesser $m$’s — as follows:

First, observe that:

\begin{align*} n^{m+1} & = n^{m+1} - (n-1)^{m+1} \\ & + (n-1)^{m+1} - (n-2)^{m+1} \\ & + (n-2)^{m+1} - (n-3)^{m+1} \\ & + \cdots \\ & + 2^{m+1} - 1^{m+1} \\ & + 1^{m+1} - 0^{m+1} \\ & = \sum_{i=1}^n (i^{m+1} - (i-1)^{m+1}) \end{align*}
Expand $i^{m+1} - (i-1)^{m+1}$ for the $m$ in question and use the linear transformation nature of summation to rewrite the right-hand side of $n^{m+1} = \sum_{i=1}^n (i^{m+1} - (i-1)^{m+1})$ as a sum of summations of $i^k$’s.
Isolate the $\sum_{i=1}^n i^m$ term of the equation.
Substitute the closed-form expressions for the remaining $\sum_{i=1}^n i^k$ terms.

For example, once we know that $\sum_{i=1}^n i = \frac{n(n+1)}{2}$ and $\sum_{i=1}^n i^2 = \frac{n(n+1)(2n+1)}{6}$, we can derive $\sum_{i=1}^n i^3$ as follows:

\begin{align*} n^4 & = \sum_{i=1}^n (i^4 - (i-1)^4) \\ & = \sum_{i=1}^n (4i^3 - 6i^2 + 4i - 1) \\ & = 4\sum_{i=1}^n i^3 - 6\sum_{i=1}^n i^2 + 4\sum_{i=1}^n i - n \\ \sum_{i=1}^n i^3 & = \frac{1}{4} (n^4 + 6\sum_{i=1}^n i^2 - 4\sum_{i=1}^n i + n) \\ & = \frac{1}{4} (n^4 + 6\times\frac{n(n+1)(2n+1)}{6} - 4\times\frac{n(n+1)}{2} + n)\\ & = \frac{1}{4} (n^4 + 2n^3 + n^2) \end{align*}

All About reStructuredText Hyperlinks

2020-07-28T00:00:00-04:00

reStructuredText offers a number of different ways to write hyperlinks, but keeping track of all of them and their gotchas isn’t easy, and the information is scattered around the spec. This document aims to summarize all of the hyperlink-related information from the reStructuredText Markup Specification in one (hopefully) well-organized place.

This document describes reStructuredText hyperlinks as of Docutils version 0.16, the latest version at time of writing.

Contents

Standalone Hyperlinks
Embedded URI Hyperlinks
Embedded Alias Hyperlinks and Hyperlink Targets
Named Hyperlink References
Anonymous Hyperlinks
Embedded URI Hyperlinks and Automatic Reference Names
- Gotcha: Duplicate Link Text
Embedded Alias Hyperlinks and Automatic Reference Names
Intra-Document Links
Chaining Hyperlink Targets
Styling Link Text
Link Text within a Word

Standalone Hyperlinks

Firstly, reStructuredText supports standalone hyperlinks — just a bare URI (including a scheme) or e-mail address without any link text:

Go to http://www.example.com to see something neat!

E-mail me at me@example.com.

The above renders as:

Go to http://www.example.com to see something neat!

E-mail me at me@example.com.

Note that, unlike Markdown, angle brackets are not required around standalone hyperlinks, and if you do include angle brackets, the brackets will be present in the rendered output.

Also note that the scheme is required in order for a URI to be recognized as a standalone hyperlink. The domains names in the following will not be converted to hyperlinks:

Go to www.example.com — or to <www.example.org>!

If you want a hyperlink that links to www.example.com without a scheme and uses the address as the link text, you can either write out the link the long way as an embedded URI hyperlink with link text, or you can use embedded URI syntax without any link text, in which case the URI becomes the link text. For example:

Go to `<www.example.com>`_.

renders as:

Go to www.example.com.

This isn’t very useful for linking to domains, but it can be useful when you want a link to a page in the same directory with the page filename used as the link text:

See `<other_page.html>`_ for other stuff.

Rendered:

See other_page.html for other stuff.

Embedded URI Hyperlinks

If you want to create a hyperlink with link text, the most straightforward way to do so is with the embedded URI syntax:

Here is `a link <http://www.example.com>`_ to somewhere.

This renders as:

Here is a link to somewhere.

The syntax consists of a backtick, the link text (with any embedded backticks escaped with backslashes), whitespace, the target URI or e-mail address inside angle brackets, another backtick, and a single underscore. If the URI ends with an underscore, the underscore must be preceded by a backslash in order to not be parsed as an embedded alias.

Note that link text is treated literally rather than processed for any inline markup. See “Styling Link Text” below for a way around this.

There is a gotcha you may run into when defining two embedded URI hyperlinks with the same link text but different URIs; see below for more information.

Embedded Alias Hyperlinks and Hyperlink Targets

If the URI is lengthy, or if you want to link to the same location more than once, you may want to use an embedded alias. In this form, instead of specifying the URI next to the link text, you specify a hyperlink reference containing a reference name, and then you specify elsewhere (in a hyperlink target) what URI the reference name points to. For example:

Here is `a link <link_target_>`_ to somewhere.

.. _link_target: http://www.example.com

Rendered:

Here is a link to somewhere.

This syntax consists of two parts:

At the location in the text where you want the link to appear, write a backtick, the link text (with embedded backticks escaped), whitespace, a reference name followed by an underscore and encased in angle brackets, another backtick, and another underscore.
- The reference name can be any sequence of characters, though if it contains any backticks or angle brackets, or begins or ends with space characters, the characters in question will need to be escaped with backslashes when using the name in an embedded alias link.
  
  When comparing two reference names for equality, runs of whitespace are normalized to a single space, and alphabetic characters are converted to lowercase.

Elsewhere in the reStructuredText document (either before or after the hyperlink reference), write a (external) hyperlink target on a line of its own: optional whitespace, two periods, a space, an underscore, the same reference name as before (without the trailing underscore from before), a colon, whitespace, and then the URI or e-mail address that the link should point to.
- If the reference name contains any colons or is just a single underscore, you must either escape the characters in question or else enclose the reference name in backticks (in which case any backticks in the reference name need to be escaped). Either way, any leading or trailing space characters or backticks in the reference name need to be escaped as well.
- If the URI contains any space characters or ends with an underscore, the characters in question will need to be escaped with backslashes.
- The reference name and the URI may be on the same line, or you can put them on separate lines, in which case the URI must be indented relative to the two periods and there must be no intervening blank lines. The URI may even span multiple lines, in which case the lines are concatenated and any whitespace in the URI that isn’t escaped is discarded. For example, the following hyperlink targets all map to the same URI:
```
.. _one-liner: http://docutils.sourceforge.net/rst.html

.. _starts-on-this-line: http://
   docutils.sourceforge.net/rst.html

.. _entirely-below:
   http://docutils.
   sourceforge.net/rst.html
```
- If the same reference name is used for two or more hyperlink targets with different URIs, a warning is produced, and the reference name will be unusable.
- Hyperlink reference names, footnote labels, and citation labels share the same namespace. This means you can link to a footnote or citation by its label, but it also means that you can’t define a reference name that is the same as a footnote or citation label.

Once a reference name is defined in a hyperlink target, the same reference name can be used in any number of embedded alias links to create multiple hyperlinks to the same destination.

As with embedded URI hyperlinks, link text is treated literally rather than processed for any inline markup. See “Styling Link Text” below for a way around this.

Named Hyperlink References

We can simplify embedded aliases one step further and use the link text as the reference name. Simply omit the part in angle brackets:

Here is `a link`_ to somewhere.

.. _a link: http://www.example.com

If the link text ends with text inside angle brackets, at least one of the angle brackets needs to be escaped — or an escaped space character should be added to the end of the link text — in order to prevent the link from being parsed as an embedded URI or alias hyperlink.

This gets even simpler if the link text is a simple reference name — a single word (no whitespace) consisting only of letters, numbers, hyphens, underscores, periods, colons, and/or plus signs, with no punctuation at the beginning or end, and with no occurrences of two or more punctuation characters in a row. A simple reference name can be written with just the trailing underscore, no backticks:

This following word_ is a hyperlink.

.. _word: https://www.example.com

The same reference name may be used both as a named hyperlink reference and in an embedded alias link:

`This site`_ links to the same location as `these words <This site_>`_.

.. _This site: https://www.example.com

Anonymous Hyperlinks

What if you want to use the hyperlink reference syntax, but it’s for a URI that will only be linked once, you don’t feel like giving it a reference name, and the link text is too long to use as an efficient reference name? The solution: anonymous hyperlinks.

`This link`__ goes to a dot-com.  `This other link`__ goes to a dot-net.

.. __: https://www.example.com
__ https://www.example.net

Rendered:

This link goes to a dot-com. This other link goes to a dot-net.

Write the link text as a named hyperlink reference, but instead of ending it with one underscore, use two. (As with named hyperlink references, the backticks can be omitted for a simple reference name.) Then, for the hyperlink target, use an underscore in place of the reference name (so that you have two underscores in a row); alternatively, the entire hyperlink target can be written as just two underscores, whitespace, and the URI or e-mail address.

The first anonymous hyperlink in a document will link to the URI given by the first anonymous hyperlink target, the second anonymous hyperlink will link to the second anonymous target, etc.

Embedded URI Hyperlinks and Automatic Reference Names

Here’s an interesting fact about embedded URI hyperlinks: they’re equivalent to a named hyperlink reference using the link text as the reference name. That means this:

`This website <https://www.example.com>`_ is awesome!

is exactly equivalent to:

`This website`_ is awesome!

.. _This website: https://www.example.com

As a consequence of this, you can link to the same location as an embedded URI link by using its link text as the reference name:

I changed my mind; `this website <https://www.example.com>`_ sucks.  Let me
reiterate: The website at `this link <this website_>`_ is nothing special.

Gotcha: Duplicate Link Text

As stated above, if a reference name is associated with two different URIs, rendering the document will produce a warning, and the reference name will be unusable in hyperlinks. So what happens if we define two embedded URI hyperlinks with the same link text but different URIs, like so?

See `here <https://www.example.com>`_ and `here <https://www.example.net>`_
for more information.

With the above input, a warning will be produced, but the hyperlinks will still point where you want them to, and the reference name here will refer to the first URI. This normally isn’t all that bad, but if you’re using a renderer that fails on warnings — e.g., if you’re uploading a project with a reStructuredText README to the Python Package Index — the rendering will fail, and your upload to PyPI will either be rejected or end up with an unrendered project description.

So how do we cleanly write embedded URI links with the same link text but different URIs? Answer: We add an extra underscore to the end of the link, turning it into an anonymous hyperlink.

See `here <https://www.example.com>`__ and `here
<https://www.example.net>`__ for more information.

With two underscores, no hyperlink target is created, and so there is no conflict.

Embedded Alias Hyperlinks and Automatic Reference Names

Similarly to embedded URI hyperlinks, using an embedded alias hyperlink turns the link text into a reference name pointing at the same location as the hyperlink reference. The following markup defines four hyperlinks that all point to <https://www.example.com>:

See `this site <site_>`_ for more information.

.. _site: https://www.example.com

Now that I've written that link, I can write these links: `Click me! <this
site_>`_  I link to `this site`_!  `Click me!`_

If you define multiple embedded alias hyperlinks with the same link text but different hyperlink references, the document will render without any warnings, and the link text will be usable as a reference name pointing to the same location as the first hyperlink reference.

Intra-Document Links

Linking to different parts of the same document is accomplished using embedded alias hyperlinks and named hyperlink references, just like external links; the difference is in how the hyperlink target is defined.

Internal Hyperlink Targets

A hyperlink target without a URI creates an internal hyperlink target that points to the next element in the document after the target.

Click `here <After Lorem_>`_ to skip the next paragraph.

Lorem ipsum dolor sit amet …

.. _After Lorem:

This paragraph can be linked to with the reference name "``After Lorem``."
Aren't you glad you didn't skip the previous paragraph now?

Rendered:

Click here to skip the next paragraph.

Lorem ipsum dolor sit amet …

This paragraph can be linked to with the reference name “After Lorem.” Aren’t you glad you didn’t skip the previous paragraph now?

The target points to the next element even if the target is indented so as to be “nested” at the end of an indented block. This allows us to attach targets to individual elements of a list:

1. Any following lines that line up with "Any" belong to this list item.

   .. _item2:

2. This list item can be linked to with the reference name ``item2``.

If the .. _item2: line above wasn’t indented, it would split the list into two lists, and the target would point to the second list. (A target before a list always points to the list as a whole; a target pointing to just the first item is not possible.)

Named Directives

Most reStructuredText directives support a :name: option that takes a string as an argument. Setting this option allows you to link to the directive using that name, equivalent to preceding the directive with an internal hyperlink target.

.. danger::
    :name: dont-or-whatever

    Don't stick your finger in the— You know what?  Forget it.  I'm not
    your mother.

… Text passes …

Hey, remember `that admonition from earlier <dont-or-whatever_>`_?  I was
serious.

Rendered:

!DANGER!

Don’t stick your finger in the— You know what? Forget it. I’m not your mother.

… Text passes …

Hey, remember that admonition from earlier? I was serious.

As this is the same as using an internal hyperlink target, a warning will be generated if two directives have the same name or if the name of a directive is the same as a reference name of another hyperlink target.

Inline Internal Targets

A phrase within a paragraph of text can be made into a target by preceding the phrase with an underscore and a backtick, escaping any backticks inside the phrase, and appending a backtick to the end of the phrase. (The backticks cannot be omitted, no matter how simple the phrase is.) This defines the phrase itself as a reference name that points to the phrase’s location in the document.

They're called "paragraphs," but I've never seen them _`para`!  Know what I
mean?

(I don't know what I was saying `here <para_>`_.)

Implicit Hyperlink Targets for Section Titles

A section title in a reStructuredText document implicitly defines a hyperlink target pointing to that section with the same reference name as the section title.

Go read "`All About Eels`_" to learn about our wriggly friends!

Didn't you hear me?  Who wouldn't want to click `this link <All About
Eels_>`_?

All About Eels
==============
Did you know?  When you're bit in the heel by a big giant eel, that's a
moray.

If a section has the same name as a hyperlink target or a directive, the hyperlink target or directive takes precedence, and the section cannot be linked to by name. If two or more sections have the same name, none of them can be linked to by name. In order to link to a section that cannot be linked by name, you must precede the section title with an internal hyperlink target and link to that.

Chaining Hyperlink Targets

It’s possible to define multiple hyperlink targets all pointing to the same location by “chaining” the targets together, one after the other:

.. _foo:
.. _bar:
.. _baz: https://www.example.com

Now the reference names foo_, bar_, and baz_ all link to the same place.

Chained hyperlink targets all point to the same location as the last target in the chain. If the last target is an internal hyperlink target, the chained targets will all point to the same document element as that last target:

.. _foo:
.. _bar:
.. _baz:

Now the reference names foo_, bar_, and baz_ all link to this paragraph.

Alternatively, a hyperlink target A can be defined to point to the same location as another target B by defining the hyperlink target with B (as a named hyperlink reference) in place of the URI:

.. _some link: https://www.example.com
.. _foo: `some link`_

Now `some link`_ and foo_ go to the same website.

The .. _foo: definition is called an indirect hyperlink target. As with named hyperlink references, the backticks can be dropped when the hyperlink reference is a simple reference name. As with external hyperlink targets, the hyperlink reference may begin on the same or next line as the target, and it may span multiple lines.

Styling Link Text

As you may have noticed, inline markup in link text is treated literally rather than being processed into emphasis etc. For example, this:

Try this recipe for `pie *à la mode* <https://www.example.com>`_.

renders as:

Try this recipe for pie *à la mode*.

The asterisks are rendered as-is instead of causing the “à la mode” to be emphasized.

Fortunately, there’s a trick to use inline markup in link text: Define the link using a substitution:

Try this recipe for |pie à la mode|_.

.. |pie à la mode| replace:: pie *à la mode*
.. _pie à la mode: https://www.example.com

Rendered:

Try this recipe for pie à la mode.

The steps to do this are as follows:

In our text, we insert a substitution reference where we want the link to be. A substitution reference consists of a vertical bar, some arbitrary substitution text, and another vertical bar. Because we also want this to be a hyperlink, an underscore is added after the closing vertical bar, causing the substitution reference to also be a named hyperlink reference with the substitution text as the reference name.
- The substitution text can be any text that does not begin or end with whitespace. Substitution text is matched case-sensitively, but if that fails, a case-insensitive match is tried.
Elsewhere in the document, a substitution definition is given for the substitution reference: two periods, a space, a vertical bar, the same substitution text as in the substitution reference, another vertical bar, whitespace, a replace:: directive (without leading ..), whitespace, and then finally the inline markup to display in place of the substitution reference in the rendered document.

Elsewhere in the document, a hyperlink target is created that maps the substitution text to the URI or e-mail address that the substituted text should link to. To link to a location in the document rather than to an external URI, either use an indirect hyperlink target:

Try this recipe for |pie à la mode|_.

.. |pie à la mode| replace:: pie *à la mode*
.. _pie à la mode: `pie recipe`_

Some intervening text

.. _pie recipe:

So here's how you make pie *à la mode*: …

or else make the substitution text the same as the reference name of the internal target:

Try this recipe for |pie recipe|_.

.. |pie recipe| replace:: pie *à la mode*

Some intervening text

.. _pie recipe:

So here's how you make pie *à la mode*: …

If desired, the substitution reference can be made into an anonymous hyperlink instead by placing two underscores instead of one after the closing vertical bar, in which case the hyperlink target must follow the anonymous hyperlink target syntax.

Link Text within a Word

Normally, a hyperlink spans one or more full words, but what if we want to only link part of a word? To do so, we must insert a backslash (optionally followed by a whitespace character) between the link and the rest of the word:

These `link <https://www.example.com>`_\s are getting out of control!  Now
they're in the in\ test_\ ines of our words.

.. _test: https://www.example.net

This renders as:

These links are getting out of control! Now they’re in the intestines of our words.

Unicode and LaTeX

2020-07-27T00:00:00-04:00

LaTeX is a powerful document typesetting system, but (especially if you’re using a version even a few years old), getting it to accept and display non-ASCII characters natively requires knowledge squirreled away in scattered documents. Here we cover how to get LaTeX to work with UTF-8 input characters in hopes of thwarting the squirrels of obscurity.

Or you could just skip all that and use XeLaTeX or LuaLaTeX instead.

pdfTeX Engine (pdfLaTeX)

When using “basic” LaTeX with the pdfTeX engine, enabling UTF-8 input requires simply placing the following commands at the top of your document preamble:

\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}

The exact effects of these commands are as follows:

\usepackage[T1]{fontenc} sets the output font encoding to T1. A font encoding is a mapping between character codes and glyphs in a font; the font encodings defined by LaTeX are described in [4].

By default, LaTeX uses the OT1 font encoding, a 7-bit encoding in which accented characters like “ö” are formed by adding an accent glyph to the base letter. T1, by contrast, is an 8-bit encoding supporting widespread European languages in which many letter+accent combinations exist as single glyphs. [3] [14]

Without this command, characters will be represented using what’s available in OT1, and, as a result, words containing accented characters won’t be correctly hyphenated, copying-and-pasting accented characters from a built PDF won’t work correctly, and inputting a |, <, or > will produce a completely different character in the resulting PDF. [14]

If one passes a comma-separated list of font encodings to the fontenc package, the last encoding in the list becomes the document’s default encoding, and switching to the other encodings (e.g., in order to enter characters only defined by those encodings) becomes possible using the command sequence \fontencoding{INSERT ENCODING NAME HERE} \selectfont. [6] [13] [9]
\usepackage[utf8]{inputenc} sets the input encoding for the document source to UTF-8, allowing UTF-8 characters to appear in the input document. Additionally, for each font encoding used in the document, inputenc loads a mapping of UTF-8 characters to commands usable in that font encoding. [1] When paired with the fontenc command above, on recent LaTeX versions, inputenc loads mappings from the files omsenc.dfu, ot1enc.dfu, t1enc.dfu, and ts1enc.dfu [13], located in $TEXDIR/texmf-dist/tex/latex/base in an installed TeX Live distribution; this in turn allows the user to input any of the characters mapped by those files into a document and have the characters be typeset appropriately.

Without this command, older versions of LaTeX will use a “raw” input encoding in which each input byte is typeset as the glyph in the same position in the current font. [8]

Beginning with the 2018-04-01 release, LaTeX uses UTF-8 as the default encoding for source files, making \usepackage[utf8]{inputenc} redundant. [8]

Once the above commands are added to your document preamble, you will be able to enter a number of UTF-8 characters directly into your document and have them show up in the built PDF without having to type out their commands. You’ll even be able to write smart quotes (“ and ”) directly instead of typing quotes out as `` and ''.

So instead of writing this:

``My na\"\i{}ve r\'esum\'e is attached.''  --- Se\~nor \TH{}or

you can write this:

“My naïve résumé is attached.”  — Señor Þor

and LaTeX will handle the input correctly.

Note that LaTeX does not support combining characters; input must be in a composed form.

If LaTeX encounters a Unicode character that it doesn’t have a definition for, typesetting will stop with an error message of the form:

! Package inputenc Error: Unicode char ☃ (U+2603)
(inputenc)                not set up for use with LaTeX.

If you want to use a certain character in your document that LaTeX doesn’t recognize, you can use the \DeclareUnicodeCharacter{hexcode}{cmd} command provided by inputenc. Its first argument is the hexadecimal code point of the Unicode character to define, and the second argument is the LaTeX command to execute when the character is encountered. [1] For example:

\usepackage{tikzsymbols}  % provides \Snowman
\DeclareUnicodeCharacter{2603}{\Snowman}
% Now you can put ☃ in your document!

If you don’t want to have to enter characters as codepoints, the \newunicodechar command provided by the newunicodechar package lets you use the character itself instead, [2] allowing us to rewrite the example above as:

\usepackage{newunicodechar}
\usepackage{tikzsymbols}  % provides \Snowman
\newunicodechar{☃}{\Snowman}
% Now you can put ☃ in your document!

As a special case, loading the textcomp package lets you input all of the Unicode characters that can be output with textcomp’s commands; for example, textcomp defines a \textmusicalnote command that produces ♪ (U+266A, EIGHTH NOTE), and so including textcomp in your preamble allows you to write “♪” in your document and have it be treated as the \textmusicalnote command, producing a “♪” in the output.

Non-Latin Alphabets

The commands described so far only provide meaningful support for text in Latin-derived alphabets. In order to enter text in other alphabets, more elaborate steps are required.

Cyrillic Alphabet

The most direct way to enable Cyrillic input is to specify a Cyrillic font encoding in the fontenc command. Due to the large number of Cyrillic characters in existence, the script is split up into three font encodings (T2A, T2B, and T2C) that each match up with the T1 encoding in the lower 7-bit range, plus a fourth encoding, X2, that contains all of the Cyrillic characters but is not compatible with T1. [4] [11]

A purely-Cyrillic document can be written with the X2 font encoding as follows:

\documentclass{article}
\usepackage[X2]{fontenc}
\usepackage[utf8]{inputenc}
\begin{document}
Пролетарии всех стран, соединяйтесь!
\end{document}

If you want to use both Cyrillic and Latin characters in your document, you need to pass both T1 and X2 to fontenc. Whichever one is listed last in the fontenc command becomes the default font encoding for the document; the other font encoding can be switched to by writing \fontencoding{INSERT ENCODING NAME HERE} \selectfont. [6] [13] [9] For example:

\documentclass{article}
\usepackage[X2,T1]{fontenc}
\usepackage[utf8]{inputenc}
\begin{document}
“{\fontencoding{X2}\selectfont Пролетарии всех стран, соединяйтесь!}” said
Señor Þor.
\end{document}

Managing encodings this way can get annoying; fortunately, the babel package provides a better way. Add a \usepackage[LANGUAGES]{babel} command to your preamble, where LANGUAGES is replaced by a comma-separated list of the languages that will be used in your document; the last language in the list will become the document’s default language. Within the document, the language can be changed with \selectlanguage{LANGUAGE} (though, for short passages, it’s better to use \foreignlanguage{LANGUAGE}{TEXT}), and when it’s set to a Cyrillic-using language, you can write in Cyrillic. [7] [11] For example:

\documentclass{article}
% If we don't explicitly load a Cyrillic font encoding, babel emits a
% warning and defaults to loading T2A.
\usepackage[T2A,T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[russian,english]{babel}
\begin{document}
“\foreignlanguage{russian}{Пролетарии всех стран, соединяйтесь!}” said
Señor Þor.
\end{document}

Greek Alphabet

As with Cyrillic, entering Greek in LaTeX requires setting the font encoding, in this case to LGR: [4]

\documentclass{article}
\usepackage[LGR,T1]{fontenc}
\usepackage[utf8]{inputenc}
\begin{document}
“{\fontencoding{LGR}\selectfont Ἄνδρα μοι ἔννεπε, Μοῦσα, πολύτροπον, ὃς
μάλα πολλὰ}” said Homer.

“Is he talking about me?” wondered Señor Þor.
\end{document}

As before, we can let also choose to let babel take care of the encodings for us:

\documentclass{article}
% No need to explicitly load LGR!
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[greek,english]{babel}
\begin{document}
“\foreignlanguage{greek}{Ἄνδρα μοι ἔννεπε, Μοῦσα, πολύτροπον, ὃς μάλα
πολλὰ}” said Homer.

“Is he talking about me?” wondered Señor Þor.
\end{document}

As another alternative, the greek-fontenc package provides a textalpha package that allows one to write Greek directly without the need for babel or language-switching: [5]

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{textalpha}
\begin{document}
“Ἄνδρα μοι ἔννεπε, Μοῦσα, πολύτροπον, ὃς μάλα πολλὰ” said Homer.

“Is he talking about me?” wondered Señor Þor.
\end{document}

greek-fontenc also provides an alphabeta package that lets one use Greek characters directly in math mode. [5]

Other Alphabets

LaTeX’s built-in font encodings only cover Latin, Cyrillic, and Greek. Enabling input in other alphabets is a separate topic for each alphabet with no easy one-size-fits-all answer.

XeTeX Engine (XeLaTeX) and LuaTeX Engine (LuaLaTeX)

Besides pdfTeX, LaTeX can also run on two major alternative engines:

The XeTeX engine, on which LaTeX runs as XeLaTeX
The LuaTeX engine, on which LaTeX runs as LuaLaTeX. This is a TeX engine with an embedded interpreter for the Lua programming language that allows developers to extend the engine by coding in Lua. [12] [15]

Both engines fully support Unicode input and support modern font technologies, including being able to use fonts from the operating system. [16] [12] When it comes to Unicode support, the major differences between pdfLaTeX and XeLaTeX/LuaLaTeX are:

XeLaTeX and LuaLaTeX documents must always be written in UTF-8, while pdfLaTeX accepts document in various input encodings. [10] [13]
The fontenc and inputenc commands used in pdfLaTeX should be omitted when working with XeLaTeX/LuaLaTeX; the Unicode engines ignore (and give a warning about) inputenc, while setting fontenc can actually cause some characters (like smart quotes) to not be recognized. Instead, you can just start entering Unicode characters directly into your document without having to include any packages.
The set of available Unicode characters in XeLaTeX/LuaLaTeX is determined by what characters are defined in the current font. [13] The default font in both XeLaTeX and LuaLaTeX is Latin Modern, a derivative of TeX’s Computer Modern default font that adds many more characters.
If XeLaTeX encounters a Unicode character that does not exist in the current font, the resulting PDF will show the font’s placeholder character if it has one; if the font has no placeholder character, nothing will be shown. Either way, the .log file will contain a line of the form:
```
Missing character: There is no ☃ in font [lmroman10-regular]:mapping=tex-text;!
```
If LuaLaTeX encounters a Unicode character that does not exist in the current font, the character will be omitted in the resulting PDF. No warning will be emitted or logged.
\DeclareUnicodeCharacter is not a valid command in XeLaTeX or LuaLaTeX; one must instead write something like:
```
\usepackage{tikzsymbols}  % provides \Snowman
\catcode`☃=\active
\protected\def ☃{\Snowman}
```
\newunicodechar can still be used in place of this method, though. [2]
Being able to write in another alphabet is largely a matter of switching to a font that supports that alphabet. See the fontspec package for how to change fonts in XeLaTeX and LuaLaTeX.
While neither XeLaTeX nor LuaLaTeX natively supports combining characters, the Lua scripting capabilities in the latter can be used to give combining characters in your source code the desired effect; see <https://tex.stackexchange.com/a/149197> for an example.

References

[1]	(1, 2) Alan Jeffrey and Frank Mittelbach, inputenc.sty. Version 1.3c. Last modified 2018 August 11, <http://mirrors.ctan.org/macros/latex/base/inputenc.pdf> (accessed 2020 July 27).

[2]	(1, 2) Enrico Gregorio, The newunicodechar package. Last modified 2018 April 8, <http://mirrors.ctan.org/macros/latex/contrib/newunicodechar/newunicodechar.pdf> (accessed 2020 July 27).

[3]	“fontenc vs inputenc”, TeX - LaTeX Stack Exchange. Last modified 2018 April 3, <https://tex.stackexchange.com/q/44694> (accessed 2020 July 27).

[4]	(1, 2, 3) Frank Mittelbach, Robin Fairbairns, Werner Lemberg, and LaTeX3 Project Team, LaTeX font encodings. Last modified 2016 February 18, <https://www.latex-project.org/help/documentation/encguide.pdf> (accessed 2020 July 27).

[5]	(1, 2) Günter Milde, Greek Unicode with 8-bit TeX and inputenc. Last modified 2019 July 11, <http://mirrors.ctan.org/language/greek/greek-inputenc/greek-utf8.pdf> (accessed 2020 July 27).

[6]	(1, 2) Johannes Braams, David Carlisle, Alan Jeffrey, Leslie Lamport, Frank Mittelbach, Chris Rowley, and Rainer Schöpf, The LaTeX2e Sources. Last modified 2020 February 2, <http://mirrors.ibiblio.org/CTAN/macros/latex/base/source2e.pdf> (accessed 2020 July 27).

[7]	Johannes L. Braams and Javier Bezos, Babel: Localization and internationalization. Version 3.47. Last modified 2020 July 13, <http://mirrors.ctan.org/macros/latex/required/babel/base/babel.pdf> (accessed 2020 July 27).

[8]	(1, 2) LaTeX News, issue 28, 2018 April. <https://www.latex-project.org/news/latex2e-news/ltnews28.pdf> (accessed 2020 July 27).

[9]	(1, 2) LaTeX2e unofficial reference manual. Last modified 2018 October, <http://tug.org/texinfohtml/latex2e.html> (accessed 2020 July 27).

[10]	Tobias Oetiker, Hubert Partl, Irene Hyna, and Elisabeth Schlegl, The Not So Short Introduction to LaTeX2ε. Version 6.2. Last modified 2018 February 28, <http://tug.ctan.org/info/lshort/english/lshort.pdf> (accessed 2020 July 27).

[11]	(1, 2) Vladimir Volovich, Werner Lemberg, and LaTeX3 Project Team, Cyrillic languages support in LaTeX. Last modified 1999 March 12, <https://www.latex-project.org/help/documentation/cyrguide.pdf> (accessed 2020 July 27).

[12]	(1, 2) “What are XeTeX and LuaTeX?”, The TeX Frequently Asked Question List. <https://www.texfaq.org/FAQ-xetex-luatex> (accessed 2020 July 27).

[13]	(1, 2, 3, 4, 5) “What Unicode characters does pdfLaTeX support with a minimal preamble?”, TeX - LaTeX Stack Exchange. Last modified 2020 July 27, <https://tex.stackexchange.com/q/555199> (accessed 2020 July 27).

[14]	(1, 2) “Why should I use \usepackage[T1]{fontenc}?”, TeX - LaTeX Stack Exchange. Last modified 2017 April 13, <https://tex.stackexchange.com/a/677> (accessed 2020 July 27).

[15]	Wikipedia contributors, “LuaTeX,” Wikipedia, The Free Encyclopedia. <https://en.wikipedia.org/w/index.php?title=LuaTeX&oldid=965669811> (accessed 2020 July 27).

[16]	XeTeX - Unicode-based TeX. <http://xetex.sourceforge.net> (accessed 2020 July 27).

Setting Default Option Values from Config Files with Click

2020-07-17T00:00:00-04:00

When developing a command-line application in Python with Click, you may find yourself wanting to use a configuration file to set the parameters’ default values at runtime and to be able to specify such a file with a --config option. There are various libraries of varying quality that provide such functionality in different ways; however, implementing such functionality on one’s own is simple enough (as this document will show), and knowing the basics will help you develop more advanced implementations when the libraries out there don’t meet your needs.

All code in this document has been tested against Click version 7.1.2.

`default_map`

The key feature of Click that will allow us to set parameter values from a configuration file is the default_map attribute of click.Context, documented at [1]. Any values set in this dict before parameters are processed will become the parameters’ new default values. Any extra keys that do not correspond to defined parameters are ignored.

Moreover, Click will map values in default_map through the parameters’ normal type conversion & validation routines. This means that, if you have an option --foo defined with type=int, you can set ctx.default_map["foo"] = "5", and the value will be converted to an int by the time it’s passed to the command callback. This even works for boolean flags: a value set for a boolean flag option will be processed by the click.BOOL type, which maps boolean-like strings to bool values. This eases your workload when reading from a configuration file type like INI where values don’t come with type information; just pass the values straight to Click, and it’ll do the conversion for you the same as it does for values on the command line.

Areas to be careful in include parameters defined with multiple=True. The default value for such parameters (whether declared with default= in the parameter’s decorator or set in default_map) must be a list or tuple; setting such a default to a string will cause the string to be interpreted as a list of single-character strings. Also requiring special attention are options defined with nargs or with a tuple of types; correct handling for all of these is left as an exercise for the reader.

Implementing a `--config` option

In order to define a --config FILE option that reads from FILE and sets other parameters’ default values, the option first of all needs to be eager so that it can modify ctx.default_map before the other options read it, and it needs to be defined with a callback that does the actual work. Everything else after that is straightforward.

Here is a sample Python script with a --config option that reads from a given config file (or from config.ini in the current directory if no config file is given). The config file must be an INI file, and the values for the options are read from the [options] section. The command callback simply dumps out its arguments so you can see what’s being passed to it.

from   configparser import ConfigParser
import json
import click

DEFAULT_CFG = 'config.ini'

def configure(ctx, param, filename):
    cfg = ConfigParser()
    cfg.read(filename)
    try:
        options = dict(cfg['options'])
    except KeyError:
        options = {}
    ctx.default_map = options

@click.command()
@click.option(
    '-c', '--config',
    type         = click.Path(dir_okay=False),
    default      = DEFAULT_CFG,
    callback     = configure,
    is_eager     = True,
    expose_value = False,
    help         = 'Read option defaults from the specified INI file',
    show_default = True,
)
@click.option('--integer', type=int, default=42)
@click.option('--flag/--no-flag', default=False)
@click.option('--str', default='foo')
@click.option('--choice', type=click.Choice(['red', 'green', 'blue']))
def main(**kwargs):
    print(json.dumps(kwargs, sort_keys=True, indent=4))

if __name__ == '__main__':
    main()

If we run this script with no options when config.ini does not exist or is empty, we get the parameters’ built-in default values:

$ python3 config01.py
{
    "choice": null,
    "flag": false,
    "integer": 42,
    "str": "foo"
}

That’s boring! Try populating example.ini with the below text:

[options]
integer = 23
flag = yes
str = bar
choice = green

… and then run with --config example.ini:

$ python3 config01.py --config example.ini
{
    "choice": "green",
    "flag": true,
    "integer": 23,
    "str": "bar"
}

Note that the values set for the flag and integer options have been converted to their appropriate types.

Of course, options set in the config file are overridden by command-line options, no matter where the options occur in relation to --config:

$ python3 config01.py --integer 17 --config example.ini --str glarch
{
    "choice": "green",
    "flag": true,
    "integer": 17,
    "str": "glarch"
}

What if a value in the config file is invalid? Try saving the following text to bad.ini:

[options]
choice = mauve

The script will then error when passed this config file:

$ python3 config01.py --config bad.ini
Usage: config01.py [OPTIONS]
Try 'config01.py --help' for help.

Error: Invalid value for '--choice': invalid choice: mauve. (choose from red, green, blue)

Not the best possible error message (It doesn’t tell us the bad value was in the config file), but it’s better than a stack trace.

Note that, with this code, parameters in the config file must be named using the same name & spelling as the parameter’s corresponding argument to the command callback. For example, the --integer option must be written integer, not --integer or -i or i; any entries in the config file with an invalid spelling will be ignored. For options with medial hyphens on the command line, like --log-level, the hyphens must become underscores in the configuration file, like log_level. If you want to support the spelling log-level as well, insert the following line after cfg = ConfigParser() to make the ConfigParser object convert hyphens in option names to underscores:

cfg.optionxform = lambda s: s.replace('-', '_')

Configuring command groups

default_map supports passing values to subcommands in command groups in a very simple way: if the main command has a subcommand named “foo”, then ctx.default_map["foo"] can be set to a dict of parameter names & values for foo. For example, the following assignment:

ctx.default_map = {
    "color": "red",
    "foo": {
        "speed": "high",
    },
    "bar": {
        "speed": "low",
        "baz": {
            "time": "late",
        },
    },
}

sets the default value for the main command’s --color option to red, the default value of the foo subcommand’s --speed option to high, the default value of the bar subcommand’s --speed option to low, and the default value of the bar baz sub-subcommand’s --time option to late. As you can see, this comes with one major drawback: a command can’t have a subcommand with the same name as one of its parameters.

Here is a sample Python script with command groups that reads configuration from an INI file. Settings in the [options] section are applied to the top-level command, settings in the [options.CMD] section are applied to the subcommand CMD, settings in [options.CMD1.CMD2] are applied to the CMD2 sub-subcommand of the CMD1 subcommand, and so forth. As above, each command prints out the parameters it receives.

from   configparser import ConfigParser
import json
import click

DEFAULT_CFG = 'config.ini'

def configure(ctx, param, filename):
    cfg = ConfigParser()
    cfg.read(filename)
    ctx.default_map = {}
    for sect in cfg.sections():
        command_path = sect.split('.')
        if command_path[0] != 'options':
            continue
        defaults = ctx.default_map
        for cmdname in command_path[1:]:
            defaults = defaults.setdefault(cmdname, {})
        defaults.update(cfg[sect])

@click.group(invoke_without_command=True)
@click.option(
    '-c', '--config',
    type         = click.Path(dir_okay=False),
    default      = DEFAULT_CFG,
    callback     = configure,
    is_eager     = True,
    expose_value = False,
    help         = 'Read option defaults from the specified INI file',
    show_default = True,
)
@click.option('--integer', type=int, default=42)
@click.option('--flag/--no-flag', default=False)
@click.option('--str', default='foo')
@click.option('--choice', type=click.Choice(['red', 'green', 'blue']))
def main(**kwargs):
    print('* main')
    print(json.dumps(kwargs, sort_keys=True, indent=4))

@main.command()
@click.option('--speed', type=click.Choice(['low', 'medium', 'high', 'ludicrous']), default='medium')
def foo(**kwargs):
    print('* foo')
    print(json.dumps(kwargs, sort_keys=True, indent=4))

@main.group(invoke_without_command=True)
@click.option('--speed', type=click.Choice(['low', 'medium', 'high', 'ludicrous']), default='medium')
def bar(**kwargs):
    print('* bar')
    print(json.dumps(kwargs, sort_keys=True, indent=4))

@bar.command()
@click.option('--time', type=click.Choice(['early', 'late', 'exact']), default='early')
def baz(**kwargs):
    print('* baz')
    print(json.dumps(kwargs, sort_keys=True, indent=4))

if __name__ == '__main__':
    main()

Set config.ini to the following:

[options]
integer = 23
flag = yes
str = bar
choice = green

[options.foo]
speed = high

[options.bar]
speed = low

[options.bar.baz]
time = late

… and then invoke some commands to see the results:

$ python3 config02.py
* main
{
    "choice": "green",
    "flag": true,
    "integer": 23,
    "str": "bar"
}
$ python3 config02.py foo
* main
{
    "choice": "green",
    "flag": true,
    "integer": 23,
    "str": "bar"
}
* foo
{
    "speed": "high"
}
$ python3 config02.py bar
* main
{
    "choice": "green",
    "flag": true,
    "integer": 23,
    "str": "bar"
}
* bar
{
    "speed": "low"
}
$ python3 config02.py bar baz
* main
{
    "choice": "green",
    "flag": true,
    "integer": 23,
    "str": "bar"
}
* bar
{
    "speed": "low"
}
* baz
{
    "time": "late"
}
$ python3 config02.py --choice red foo --speed medium
* main
{
    "choice": "red",
    "flag": true,
    "integer": 23,
    "str": "bar"
}
* foo
{
    "speed": "medium"
}

Knowledge Bits

Retrieving Logs for a Single Service Run from systemd

Basic journalctl(1) Usage

Systemd vs. Service Entries

Invocation IDs

Getting Logs for an Invocation ID

Process Groups, Sessions, and Controlling Terminals

Concepts

Process Groups at the Shell

Querying & Manipulating Process Groups & Sessions via System Calls

Creating a Background Process Without a Controlling Terminal (Daemonization)

Getting a Terminal’s Default Foreground & Background Colors

Getting the Current Cursor Position from a Terminal

The BitTorrent Encryption Protocol

Converting Lists to Simple Tables and Back with rs

Converting a List of Lines to a Simple Table

Converting a Simple Table to a List of Lines

Basics of Writing Unix Man Pages

Implementing Boolean Negation Flags with Clap

Python Asynchronous Programming Fundamentals

Creating Multi-Value Enums in Python

Skipping Pytest Tests Unless an Option is Given

Option 1: Use a Hook to Attach a skip Marker to Marked Tests

Option 2: Use @pytest.mark.skipif

Option 3: Use a Pre-Existing Plugin

Notable Features Introduced in Each Python Version

Special Version Control Files

Integrating auto with bump2version

Set up bump2version

Integrate with auto

Integrate with GitHub Actions

Using Package Data in Python Projects with Setuptools

Running Extra Steps after Releasing with auto in GitHub Actions

Common Python Packaging Mistakes

The Sum of Raising the First \(n\) Integers to a Given Power

All About reStructuredText Hyperlinks

Unicode and LaTeX

pdfTeX Engine (pdfLaTeX)

Non-Latin Alphabets

Cyrillic Alphabet

Greek Alphabet

Other Alphabets

XeTeX Engine (XeLaTeX) and LuaTeX Engine (LuaLaTeX)

References

Setting Default Option Values from Config Files with Click

default_map

Implementing a --config option

Configuring command groups

Basic `journalctl(1)` Usage

Option 1: Use a Hook to Attach a `skip` Marker to Marked Tests

Option 2: Use `@pytest.mark.skipif`

Set up `bump2version`

Integrate with `auto`

`default_map`

Implementing a `--config` option