Skip to content

Significant performance degradation building packages #19635

@ngnpope

Description

@ngnpope

Summary

We first picked this up in CI where our Docker image builds were running significantly slower and identified the issue as an upgrade from uv 0.11.14 to 0.11.15.

Image

Looking at resource usage, it seems that we went from the following:

Image

To maxxing out CPU with significantly longer duration:

Image

Of note we see that arm64, although still affected, is less bad than amd64.

I made release builds of uv on my laptop and have bisected this to f82a2cf which is the result of the following issue and pull request:

The Dockerfile has a more complex command, but is effectively running uv pip install -p $(which python) /path/to/source. For ease of testing, from the top-level of the source tree I ran uv pip install -p $(which python) -t xxx . and was able to reproduce the issue without the docker build being involved.

These are the timings I got running on my laptop (amd64), ~3.4x slower:

# 3fdfdc7d / 0.11.14

real    1m24.855s
user    1m0.430s
sys     0m23.315s

# 3cffe97c / 0.11.15

real    4m41.362s
user    32m49.346s
sys     22m36.837s

# f8fdfe3b / 0.11.14+28

real    4m46.323s
user    32m35.007s
sys     24m9.523s

# 79b5b6a9 / 0.11.14+14

real    4m45.763s
user    33m3.741s
sys     23m0.286s

# 9b78b792 / 0.11.14+7

real    4m59.556s
user    37m33.998s
sys     20m32.525s

# f82a2cf7 / 0.11.14+3

real    4m57.137s
user    31m55.857s
sys     23m20.045s

# 390ae462 / 0.11.14+2

real    1m36.855s
user    1m1.224s
sys     0m23.989s

Here's the bisect log:

$ git bisect log
# bad: [3cffe97c2e48c9e49422c738da3af95919dd0bf5] Fix crates.io publish script lockfile (#19473)
# good: [3fdfdc7d4a63c9f283eb751823b7628b13116684] Bump version to 0.11.14 (#19377)
git bisect start '0.11.15' '0.11.14'
# bad: [f8fdfe3b44a71e7cb0b4c07b32d64dac8f55f693] Reject empty string as an invalid package name (#19435)
git bisect bad f8fdfe3b44a71e7cb0b4c07b32d64dac8f55f693
# bad: [79b5b6a9312895262a1db6c8082fa46b050df0bd] uv audit: JSON output (#19305)
git bisect bad 79b5b6a9312895262a1db6c8082fa46b050df0bd
# bad: [9b78b7926f5364dfffcfc78b2dd07ccd36efb79c] Bump `cyclonedx-bom` to remove duplicate `spdx`, `base64`, and `bitflags` crates (#19386)
git bisect bad 9b78b7926f5364dfffcfc78b2dd07ccd36efb79c
# bad: [f82a2cf7184dd551c314afc46b6b6601885faa70] Remove usages of `zip` crate (#19365)
git bisect bad f82a2cf7184dd551c314afc46b6b6601885faa70
# good: [390ae46203193ad34239a1aeca89cc5886f5e84e] Apply stricter validation to all wheel filename segments (#19364)
git bisect good 390ae46203193ad34239a1aeca89cc5886f5e84e
# first bad commit: [f82a2cf7184dd551c314afc46b6b6601885faa70] Remove usages of `zip` crate (#19365)

For perspective on the size: 27,113 directories, 101,363 files with the size of the installed files being 1.1GB. Unfortunately I can't really provide a reproducer, but I imagine that you could generate something of a similar size.

Platform

Debian 12 Bookworm amd64/arm64, Arch Linux amd64

Version

uv 0.11.15

Python version

Python 3.13.7

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions