What are these files again?

__init__.py files are often used to mark directories as Python packages. For example, a file structure like this:

my_package/
    __init__.py
    some_module.py

will allow you to run:

import my_package.some_module
# or
from my_package import some_module

What you might not know is that in modern Python, you can omit the __init__.py file and still be able to run the same import! So why not get rid of these files altogether?

The benefits of being explicit

Imagine the following codebase without any __init__.py files:

services/
    component_a/
        one.py
    component_b/
        child/
            two.py
        three.py
    scripts/
        my_script.py

Encountering this structure, you might wonder which of these directories are meant to be packages, and which are just directories that happen to contain Python files.

This matters in non-obvious ways. Take the “services” directory for example. Are you meant to…

1. import services.component_a.one?
2. or component_a.one with “services” as the working directory?

The problem is: only one of these will actually work, because the package internals likely assume one or the other. For example, if one.py contains:

import component_b.three

then only option 2 will work.

Adding the proper __init__.py files takes away the guesswork and makes the structure clear:

services/
    component_a/
        __init__.py
        one.py
    component_b/
        __init__.py
        child/
            __init__.py
            two.py
        three.py
    scripts/
        my_script.py

Now, it’s immediately clear that component_a and component_b are packages, and services is just a directory. It also makes clear that “scripts” isn’t a package at all, and my_script.py isn’t something you should be importing. __init__.py files help developers understand the structure of your codebase.

Tooling needs to understand your package structure too

You might think: “I’m not convinced. I know my codebase well enough. And besides, I document how to import my packages in the README.”

What you may forget is that it’s not just humans that need to understand the package structure. Tools like mypy and ruff also need to understand what is a package and what isn’t, in order to work correctly. What makes it extra tricky is that you may not notice problems at first, but they can crop up later as your codebase grows. Fixing these issues can be a real headache, especially if you’re not aware of the intricacies of Python’s import system. By omitting __init__.py files, you may be putting a maintenance timebomb in your codebase.

What about implicit namespace packages?

When omitting __init__.py files, you’re actually creating what’s called an “implicit namespace package”. This has some benefits, like allowing you to split a package across multiple directories. If you use namespace packages for this purpose, you’re probably aware of the trade-offs, and you’ve likely already struggled with issues of tooling compatibility and developer confusion.

For this reason, implicit namespace packages are rare. So long as you don’t need the advanced features of implicit namespace packages, you should stick to using __init__.py files.

Other loose ends

• Although __init__.py files are often empty, they can also contain code. For more information, see the Python documentation.
• You can enforce the use of __init__.py files in your codebase using ruff or a flake8 plugin.

Recommendations

You should use __init__.py files to make it clear which directories are packages and which aren’t. This isn’t only helpful for other developers, it’s often necesssary for tools like mypy to work correctly.

💬 Discuss this post on Reddit or Hacker News.

What’s a pitfall?

Two notes before we start:

• Pitfalls aren’t bugs. They’re cases where datetime behaves in a way that is surprising or confusing. It’s always a bit subjective whether something is a pitfall or not.
• Many pitfalls exist simply because the authors couldn’t possibly anticipate all future needs. Adding big features over 20 years—without breaking compatibility—isn’t easy.

Libraries considered

With that out of the way, these are the third-party datetime libraries I’m looking at in this post:

• arrow — Probably the most historically popular datetime library. Its goal is to make datetime easier to use, and to add features that many people feel are missing from the standard library.
• pendulum — The only library that rivals arrow in popularity. It has similar goals, while explicitly improving on Arrow’s handling of Daylight Saving Time (DST).
• DateType — a library that allows type-checkers to distinguish between naïve and aware datetimes. It doesn’t change the runtime behavior of datetime.
• heliclockter — a young library that offers datetime subclasses for UTC, local, and zoned datetimes.

These libraries I’m not looking at:

• pytz and python-dateutil, which aren’t (full) datetime replacements
• delorean, maya, and moment which all appear abandoned

Now: on to the pitfalls!

1. Incompatible concepts are squeezed into one class

It’s an infamous pain point that a datetime instance can be either naïve or aware, and that they can’t be mixed. In any complex codebase, it’s difficult to be sure you won’t accidentally mix them without actually running the code. As a result, you end up writing redundant runtime checks, or hoping all developers diligently read the docstrings.

# Naïve or aware? No way to tell...
def plan_mission(launch_utc: datetime) -> None: ...

There’s also the question whether distinguishing aware and naïve is enough, since within the “aware” category there are actually several different kinds of datetimes. While compatible, the semantics of UTC/offset and IANA timezones are notably different when it comes to ambiguity, for example.

What’s being done about it?

• :heavy_check_mark: heliclockter has separate classes for local, zoned, and UTC datetimes.
• :heavy_check_mark: DateType allows type-checkers to distinguish naïve or aware datetimes
• :x: arrow and pendulum still have one class for naïve and aware.

2. Operators ignore Daylight Saving Time (DST)

Given that datetime supports timezones with DST transitions, you’d reasonably expect that the +/- operators would take them into account—but they don’t!

paris = ZoneInfo("Europe/Paris")
# On the eve of moving the clock forward
bedtime = datetime(2023, 3, 25, 22, tzinfo=paris)
wake_up = datetime(2023, 3, 26, 7, tzinfo=paris)

# It says 9 hours, but it's actually 8!
# (because we skipped directly from 2am to 3am due to DST)
sleep = wake_up - bedtime

What’s being done about it?

• :heavy_check_mark: pendulum explicitly fixes this issue
• :x: heliclockter, arrow, and DateType don’t address it

3. The meaning of “naïve” is inconsistent

In various parts of the standard library, “naïve” datetimes are interpreted differently. Ostensibly, “naïve” means “detached from the real world”, but in the datetime library it is often implicitly treated as local time. Confusingly, it is sometimes treated as UTC¹, while in other places it is treated as neither!

# a naïve datetime
d = datetime(2024, 1, 1)

# here: treated as a local time
d.timestamp()
d.astimezone(UTC)

# here: assumed UTC
d.utctimetuple()
email.utils.format_datetime(d)
datetime.utcnow()

# here: neither! (error)
d >= datetime.now(UTC)

What’s being done about it?

• :x: While pendulum and arrow do discourage using naïve datetimes, they still support the same inconsistent semantics.
• :x: DateType and heliclockter don’t address this

4. Non-existent datetimes pass silently

When the clock in a timezone is set forward, a “gap” is created. For example, if DST moves the clock forward from 2am to 3am, the time 2:30am is skipped. The standard library doesn’t warn you when you create such a non-existent time. As soon as you operate on these objects, you run into problems.

# This time doesn't exist on this date
d = datetime(2023, 3, 26, 2, 30, tzinfo=paris)

# No timestamp exists, so it takes another one from the future
t = d.timestamp()
datetime.fromtimestamp(t, tz=paris) == d  # False!?

What’s being done about it?

• :x: pendulum replaces the current silent behavior with another: it fast-forwards to a valid time without warning.
• :x: arrow, DateType and heliclockter don’t address this issue

5. Guessing in the face of ambiguity

When the clock in a timezone is set backwards, an ambiguity is created. For example, if DST sets the clock one hour back at 3am, the time 2:30am exists twice: before and after the change. The fold attribute was introduced to resolve these ambiguities

The problem is that there is no objective default value for fold: whether you want the “earlier” or “later” option will depend on the particular context. For backwards compatibility, the standard library defaults to 0, which has the effect of silently assuming that you want the earlier occurrence².

# Guesses your intent without warning
d = datetime(2023, 10, 29, 2, 30, tzinfo=paris)

What’s being done about it?

• :x: pendulum also guesses, but rather arbitrarily decides that 1 is the better default³.
• :x: arrow, DateType and heliclockter don’t address the issue.

6. Disambiguation breaks equality

Even though fold was introduced to disambiguate times, comparisons of disambiguated times between timezones always evaluate false due to backwards compatibility reasons.

# A properly disambiguated time...
d = datetime(2023, 10, 29, 2, 30, tzinfo=paris, fold=1)

d_utc = d.astimezone(UTC)
d_utc.timestamp() == d.timestamp()  # True: same moment in time
d_utc == d  # False!?

What’s being done about it?

• :x: None of the libraries addresses this issue

7. Inconsistent equality within timezone

In a mirror image of the previous pitfall, there is a false positive when comparing two datetimes with the exact same tzinfo object. In that case, they are compared by their “wall time”. This is mostly the same except when fold is involved…

# two times one hour apart (due to DST transition)
earlier = datetime(2023, 10, 29, 2, 30, tzinfo=paris, fold=0)
later = datetime(2023, 10, 29, 2, 30, tzinfo=paris, fold=1)

earlier.timestamp() == later.timestamp()  # false, as expected
earlier == later  # true!?

Remember I said exact same tzinfo object? If you compare with the same timezone, but you get its object from dateutil.tz instead of ZoneInfo, you’ll get a different result!

from dateutil import tz
later2 = later.replace(tzinfo=tz.gettz("Europe/Paris"))
earlier == later2  # now false

What’s being done about it?

• :x: None of the libraries addresses this issue

8. Datetime inherits from date

You may be surprised to know that datetime is a subclass of date. This doesn’t seem problematic at first, but it leads to odd behavior. Most notably, the fact that date and datetime cannot be compared violates basic assumptions of how subclasses should work. The datetime/date inheritance is now widely considered to be a design flaw in the standard library.

# Breaks on a datetime, even though it's a subclass
def is_future(d: date) -> bool:
    return d > date.today()

# Some methods inherited from `date` don't make sense
datetime.today()  # fun exercise: what does this return?

What’s being done about it?

• :heavy_check_mark: DateType was explicitly developed to fix this inheritance relationship at type-checking time.
• :x: arrow, pendulum, and heliclockter don’t address the issue. Their datetime classes all inherit from datetime (and thus also date).

9. datetime.timezone isn’t enough for timezone support

OK—so this is maybe something you learn once and then never forget. But it’s still confusing that datetime.timezone is only for fixed offsets, and you need ZoneInfo to express real-world timezone behavior with DST transitions. For beginners that don’t know the difference, this is an unfortunate trap.

from datetime import timezone, datetime, timedelta
from zoneinfo import ZoneInfo

# Wrong: it's a fixed offset only valid in winter!
paris_tz = timezone(timedelta(hours=1), "CET")

# Correct: accounts for all timezone changes
paris_tz = ZoneInfo("Europe/Paris")

• :heavy_check_mark: Both arrow and pendulum side-step this issue by specifying timezones as strings instead of requiring special class instance.
• :x: heliclockter and DateType don’t address this issue

10. The local timezone is DST-unaware

Calling astimezone() without arguments gives you the time in the local system timezone. However, it returns it as a fixed offset (datetime.timezone) instead of a full timezone (ZoneInfo) that knows about DST transitions. In Paris, for example, astimezone() returns a fixed offset of UTC+1 or UTC+2 (depending on whether it’s winter or summer) instead of the full Europe/Paris timezone.

# you think you've got the local timezone
my_tz = datetime(2023, 1, 1).astimezone().tzinfo
# but you actually only have the wintertime variant
print(my_tz)  # timezone(offset=timedelta(hours=1), "CET")
datetime(2023, 7, 1, tzinfo=my_tz)  # not valid for summer!

What’s being done about it?

• :heavy_check_mark: pendulum and arrow have methods to convert to the full local timezone.
• :x: heliclockter has a local datetime type with the same issue, although a fix is in the works.
• :x: DateType doesn’t address this issue

Datetime library scorecard

Below is a summary of how the libraries address the pitfalls (:heavy_check_mark:) or not (:x:).

Why should you care?

The pitfalls roughly fall into two categories: confusing design and surprising edge cases. Here is why you should care about both.

Confusing design

Confusing design is the larger problem, because it amplifies the biggest source of bugs: human error. While good design helps minimize the chance of mistakes, bad design introduces more opportunities for them. Looking at other languages, it’s clear that better designs are possible. Java, C#, and Rust all have distinct classes for naïve and aware datetimes (and more). We can also see that redesigns are worth the substantial effort: Java adopted Joda-Time, and JavaScript is modernizing as well. Will Python’s datetime be left behind?

Surprising edge cases

Because these pitfalls are rare, you may think they’re not worth worrying about. After all, DST transitions only represent about 0.02% of the year. While this sentiment is understandable, I’d argue that the opposite is true:

• Getting timezones right is one of the main reasons for existence of a datetime library. If it can’t do that reliably, what’s the point?
• Rare cases are the most dangerous: they are the ones you’re least likely to test, and allow bad actors to trip up your code.
• Rare is still too common for such a fundamental concept as time. Would you run your business on numpy if it had a 0.02% chance of returning the wrong result? Would you accept a language in which 1 in 4000 booleans would arbitrarily be flipped? There is no reason why these pitfalls shouldn’t be corrected.

Imagining a solution

Inspired by these findings, I created a new library to explore what a better datetime library could look like. Here is how it addresses the pitfalls:

1. It has distinct classes for the most common use cases:

   (note: the types have been updated since the original article)

   from whenever import (
       # In case you don't care about timezones
       Instant,
       # Simple localization sans DST
       OffsetDateTime,
       # Full-featured IANA timezones
       ZonedDateTime,
       # The current system timezone
       SystemDateTime,
       # 'Naive' local times without a timezone
       LocalDateTime,
   )
   

2. Addition and subtraction take DST into account.
3. Naïve is always naïve. UTC and local time have their own separate classes.
4. Creating non-existent datetimes raises an exception.

5. Ambiguous datetimes must be explicitly disambiguated.

   ZonedDateTime(
       2023, 1, 1, tz="Europe/Paris",
   )  # ok: not ambiguous
   ZonedDateTime(
       2023, 10, 29, 2, tz="Europe/Paris",
   )  # ERROR: ambiguous!
   ZonedDateTime(
       2023, 10, 29, 2, tz="Europe/Paris",
       disambiguate="later"
   )  # that's better!
   

6. Disambiguated datetimes work correctly in comparisons.

7. Aware datetimes are equal if they occur at the same moment. No exceptions.

   a == b
   # always equivalent to:
   a.instant() == b.instant()
   

8. The datetime classes don’t inherit from date.
9. IANA timezones are used everywhere, no separate classes are needed.
10. Local datetimes handle DST transitions correctly.

Feedback is welcome! :star2:

Changelog

See the git history for exact changes to this article since initial publication.

2024-02-01 18:14:00+01:00

• Clarified wording and code comments in pitfall #3.

2024-02-02 10:13:00+01:00

• Clarified wording around timezones and IANA tz database in pitfall #9, and throughout the article.
• Added reddit link

2024-02-13 08:40:00+01:00

• Clarified wording on distinguishing “aware” types in pitfall #1.
• Added note about RFC 5545 in pitfall #5.

2024-02-18 20:28:00+01:00

• Added Hacker News link
• Clarification in pitfall #4, fix code example
• Added non-emoji text to scorecard for systems that don’t support it

2024-02-18 21:10:00+01:00

• A better solution for emoji :tada:

2024-10-03 19:15:00+02:00

• Updated the types in the example code to match the current version of the library

Let’s imagine you’re dealing with some rocket launch data:

# some timestamps in milliseconds
marsrover = datetime(2020, 7, 30, 11, 50).timestamp() * 1000
pathfinder = datetime(1996, 12, 4, 6, 58).timestamp() * 1000
apollo_13 = datetime(1970, 4, 11, 19, 13).timestamp() * 1000

When we use Pydantic to load this data, we notice something strange…

from pydantic import BaseModel

class Mission(BaseModel):
    launch: datetime

Mission(launch=marsrover)   # 2020-07-30 11:50
Mission(launch=pathfinder)  # 1996-12-04 06:58
Mission(launch=apollo_13)   # 2245-11-14 00:40 ???

While the first timestamps are parsed correctly, the third is wildly different! How did this happen?

Let’s take a closer look at the timestamp values:

print(marsrover)   # 1596102600000
print(pathfinder)  # 849679080000
print(apollo_13)   # 8705580000

What jumps out is that the timestamp for Apollo 13 is a lot smaller. This makes sense as it’s closer to the Unix epoch of 1970-1-1, after all.

Pydantic draws a different conclusion: it’s small because…it probably represents seconds, not milliseconds. In other words: at some point in the seventies, it starts interpreting millisecond timestamps as seconds instead. At best, you’ll get a confusing error about out-of-bounds time data, but at worst, your data is drastically and silently transformed.

You might think: who cares? This is such a rare case — and it’s often helpful!

Yes, but:

1. It’s not uncommon for large companies to have data from the 70s, and milliseconds are frequently used to store timestamps.
2. Working with time is already complex and error-prone. We should be critical of adding another edge case.

Thankfully, the Pydantic team is quick to respond, and a solution is in the works.

The larger lesson here

Libraries have become so good at ingesting our data automagically¹, that we can forget to do proper software engineering. With basic research, you can often find out what your data looks like before you ingest it. And if you define an API, you dictate the data format!

Unless you’re dealing with unconstrained and erratic data, you most likely already know whether the timestamps you’re reading are in seconds or milliseconds. Don’t rely on a library to guess it correctly for you! Yes, it may take slightly more time to code — but your app will be safer, more predictable, and faster for it.

If you’re still unconvinced of the danger of automagical parsing, look no further than Microsoft Excel. Who among us hasn’t been tripped up by its notoriously overeager data inference? Let’s not repeat this mistake in Python. The Zen of Python already warns us:

In the face of ambiguity, refuse the temptation to guess.

Refuse the temptation of automagical parsing. Be explicit about data you ingest.

Show me!

I built a small tool, slotscheck, that scans a package for these mistakes. My hope is libraries can get a free mini performance boost from fixing their slots.

Here’s how to use it:

$ pip install slotscheck
$ pip install pandas  # or whatever library you'd like to check
$ slotscheck pandas
ERROR: 'SingleArrayManager' has slots but inherits from non-slot class
ERROR: 'Block' has slots but inherits from non-slot class
ERROR: 'NumericBlock' has slots but inherits from non-slot class
ERROR: 'DatetimeLikeBlock' has slots but inherits from non-slot class
ERROR: 'ObjectBlock' has slots but inherits from non-slot class
ERROR: 'CategoricalBlock' has slots but inherits from non-slot class
ERROR: 'BaseArrayManager' has slots but inherits from non-slot class
ERROR: 'BaseBlockManager' has slots but inherits from non-slot class
ERROR: 'SingleBlockManager' has slots but inherits from non-slot class

Fixing slots? What is this about?

Declaring __slots__ allow you to limit class attributes to a fixed set. With this information, Python can optimize the layout of the class¹. However, to get the full advantages of slots, all bases of a class also need to have it defined.

Let’s look at slots without inheritance:

# checks complete size of objects in memory
from pympler.asizeof import asizeof

class EmptyNoSlots: pass

class EmptyWithSlots: __slots__ = ()

class NoSlots:
    def __init__(self): self.a, self.b = 1, 2

class WithSlots:
    __slots__ = ("a", "b")
    def __init__(self): self.a, self.b = 1, 2

print(asizeof(EmptyNoSlots()))    # 152
print(asizeof(EmptyWithSlots()))  # 32
print(asizeof(NoSlots()))         # 328
print(asizeof(WithSlots()))       # 112 !!!

Looks like quite the difference! So what about inheritance?

class WithSlotsAndProperBaseClass(EmptyWithSlots):
    __slots__ = ("a", "b")
    def __init__(self): self.a, self.b = 1, 2

class NoSlotsAtAll(EmptyNoSlots):
    def __init__(self): self.a, self.b = 1, 2

class WithSlotsAndBadBaseClass(EmptyNoSlots):
    __slots__ = ("a", "b")
    def __init__(self): self.a, self.b = 1, 2

print(asizeof(WithSlotsAndProperBaseClass(1, 2)))  # 112
print(asizeof(NoSlotsAtAll(1, 2)))                 # 328
print(asizeof(WithSlotsAndBadBaseClass(1, 2)))     # 232 !!!

As you can see, bad __slots__ inheritance can really bloat your memory footprint!²

What did I find?

Having built slotscheck, I couldn’t wait to see what I could find. I didn’t have to look far: I found some missing slots in the standard library.

Also, a scan of the 5000 most popular PyPI packages showed several of them seem to have some classes with broken slots:

• libcst (144) (issue opened)
• dpkt (129)
• scapy (85)
• exchangelib (39)
• sqlalchemy (12) (issue opened)
• pandas (9) (issue opened)
• trafaret (9)
• acme (9)
• tensorflow_probability (6)
• torch (5)
• srsly (5)
• dynaconf (5)
• falcon (4)
• glom (4)
• aio_pika (4)
• returns (3) (solved)
• parso (3)
• autobahn (3)
• rx (3)
• pika (3)
• aiormq (3)
• boxsdk (3)
• pipenv (3)
• oauthlib (2)
• xmlschema (2)
• aiohttp (2)
• dagster (2)
• peewee (2)
• xlrd (2)
• fiona (2)
• zeroconf (2)
• parsimonious (2)
• wand (2)
• werkzeug (1)
• josepy (1)
• llvmlite (1)
• sphinx (1)
• markupsafe (1)
• tensorflow (1)
• Pathy (1)
• sanic (1)
• mongoengine (1)
• requests_html (1)

(Note that some may be false positives, or out of date since this post. The list is not exhaustive.)

I was actually surprised how many packages didn’t have issues. This is mostly due to them not having many __slots__ classes in the first place. For example, requests has 43 classes, none with slots; azure has 2391 classes, only 5 with slots. I hope tools like slotscheck help more libraries adopt slots!

What now?

The first version of slotscheck is available on PyPI. Include it in your CI pipeline to prevent slots mistakes from appearing again! Or, contribute to the community by fixing slots in any of the packages listed above. Check out the GitHub repo to follow further development, and leave a ⭐️ if you like.

The basics of logging

Where does formatting meet user input exactly? Let’s start with a basic logging setup.

import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

Logging a message:

logger.info("hello world")

Which prints:

INFO:__main__:hello world

Let’s see the string formatting¹ in action:

context = {'user': 'bob', 'msg': 'hello everybody'}
logger.info("user '%(user)s' commented: '%(msg)s'.", context)

This outputs the following:

INFO:__main__:user 'bob' commented: 'hello everybody'.

Simple injection

If you don’t sanitize your inputs, you may be vulnerable to log injection. Consider the following message:

"hello'.\nINFO:__main__:user 'alice' commented: 'I like pineapple pizza"

If logged with the previous template, this results in:

INFO:__main__:user 'bob' commented: 'hello'.
INFO:__main__:user 'alice' commented: 'I like pineapple pizza'.

As you can see, an attacker can not only corrupt logs, but also incriminate others.

Mitigation

We can mitigate this particular attack by escaping newline characters. But, beware that there are plenty of other evil unicode control characters which can mess up your logs. The safest solution is to simply not log untrusted text. If you need to store it for an audit trail, use a database. Alternatively, structured logging can prevent newline-based attacks.

Double formatting trouble

There’s another interesting vulnerability which is particular to Python. Because the old %-style formatting used by logging is often considered ugly, many people prefer to use f-strings:

logger.info(f"user '{user}' commented: '{msg}'.")

Admittedly this looks nicer, but it won’t stop logging from trying to format the resulting string itself. So if the msg is…

"%(foo)s"

…we are left with this after the f-string evaluates:

logger.info("user 'bob' commented: '%(foo)s'.")

So what does logging do? Does it try to look up foo and crash? Thankfully not. In the depths of the logging source code we find:

if self.args:
    msg = msg % self.args

No arguments, no formatting. Makes sense. But once there is an argument, things get interesting. Consider this:

logger.info(f"user '%(user)s' commented: '{msg}'.", context)

Of course, nobody is likely to mix formatting styles at first. But it is plausible that either:

• Someone would add the msg parameter to an existing log statement in this way;
• When refactoring to an f-string, someone forgot to remove the context argument;
• That log messages are passed through a user-defined function or logging filter which adds a context argument.

In this case we get an error like this:

--- Logging error ---
[...snip...]
KeyError: 'foo'
Call stack:
  File "example.py", line 29, in <module>
    logger.info(f"user '%(user)s' commented: '{msg}'.", context)
Message: "user '%(user)s' commented: '%(foo)s'."
Arguments: {'user': 'bob'}

Annoying to have in the logs? Yes. Dangerous? Not…yet. But by formatting an external string into our log message (which in turn gets formatted again by logging) we open the door to string formatting attacks. Thankfully, Python is a lot less vulnerable than C, but there are still ways to abuse it.

Padding a ton

One such case is abusing padding syntax. Consider this message:

"%(user)999999999s"

This will pad the user with almost a gigabyte of whitespace. Not only will this slow your log statement down to a crawl, it could also clog up your logging infrastructure.

Why is this a problem? Attackers being able to crash your server is bad enough. If they cripple your logging, you wouldn’t even know what hit you.

Leaky logs

Another potential risk is leaking sensitive information. In our example, if the context contains a "secret" key, an attacker could leak them into the logs with the following message:

"%(secret)s"

This is particularly dangerous when combined with the padding vulnerability, as the attacker could use timing to sniff out which keys are present.

On the flip side, we can be thankful that Python’s %-style formatting syntax is so limited. If logging used the new braces-style, an attacker wouldn’t even need sensitive data to be present in context, by using a message like:

"{0.__init__.__globals__['SECRET']}"

You might wonder what the big deal is if secrets land in the logs — it’s not in the open, right? The problem is that logs are usually a lot easier for an attacker to access than a credential store. Because of this, CWE ranks “Insertion of Sensitive Information into Log File” as number 39 on their list of most dangerous software weaknesses.

Mitigation

To eliminate these risks, you should always let logging handle string formatting. Don’t format log messages yourself with f-strings or otherwise ². Thankfully, there is a flake8 plugin that can check this for you. Also, once PEP675 is implemented, you could perhaps use a typechecker to check only literal strings are passed to the logger.

Recommendations

1. Don’t log untrusted text. Python’s logging library doesn’t protect you from newlines or other unicode characters which allow attackers to mess up — or even forge — logs.
2. Don’t format logs yourself (with f-strings or otherwise). In certain situations this could leave you vulnerable to denial-of-service attacks or even sensitive data exposure.

A full sample of the code used can be found here, so you can experiment for yourself.

You can discuss this post on reddit or Hacker News.

Update 2022-01-04

I’ve since created an issue on the Python bug tracker to document security risks in the logging docs, and perhaps even to create a more secure logger API.

Update 2022-02-19

The log formatting DoS vulnerability has been included in PEP675 as a potential use of for the string literal type. I’ve opened a discussion on discuss.python.org for improvements to the logging API.

Thanks

To Daan Debie for reviewing this post!