Skip to content

Content-address distributions in the cache by distribution hash#16816

Open
charliermarsh wants to merge 11 commits into
mainfrom
charlie/content
Open

Content-address distributions in the cache by distribution hash#16816
charliermarsh wants to merge 11 commits into
mainfrom
charlie/content

Conversation

@charliermarsh

@charliermarsh charliermarsh commented Nov 22, 2025

Copy link
Copy Markdown
Member

Summary

The core here is to always compute at least a BLAKE3 hash for all wheels, then store unzipped wheels in a content-address location in the archive directory. This will help with disk space (since we'll avoid storing multiple copies of the same wheel contents) and cache reuse, since we can now reuse unzipped distributions from uv pip install in uv sync commands (which always require hashes already).

We use BLAKE3 since it's especially fast for files that are already present on disk via it's Rayon and memory-mapping features (\cc @oconnor663).

Closes #1061.

Closes #13995.

Closes #16786.

@charliermarsh

Copy link
Copy Markdown
Member Author

Still a few things I want to improve here.

Comment thread crates/uv-distribution/src/distribution_database.rs
@charliermarsh charliermarsh marked this pull request as ready for review November 22, 2025 14:46
@charliermarsh charliermarsh force-pushed the charlie/content branch 2 times, most recently from 084d601 to d111e5c Compare November 22, 2025 15:09
raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
File "/Users/example/.cache/uv/archive-v0/3783IbOdglemN3ieOULx2/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
File "/Users/example/.cache/uv/archive-v0/97de8790030bbd5c2d96b7ec782fc2f7820ef8dba6db909ccf95449f2d062d4b/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another risk here is that this is significantly longer which hurts path length.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could base64.urlsafe_b64encode it which would be ~43 characters (less than the 64 here, but more than the 21 we used before).

@zanieb zanieb Nov 22, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few ideas...

  1. base64 encoding seems reasonable
  2. we might want to store it as {:2}/{2:}? git and npm do this to shard directories. I guess we don't have that problem today but if we're changing it maybe we should consider it? It looks like you did in 3bf79e2 ?
  3. We could do a truncated hash with a package id for collisions? {:8}/{package-id} (I guess the package-id could come first?). We'd could persist the full hash to a file for a safety check too.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I did it as {:2}/{2:4}/{4:} in an earlier commit then rolled it back because it makes various things more complicated (e.g., for cache prune we have to figure out if we can prune the directories recursively). I can re-add it if it seems compelling.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could do a truncated hash with a package id for collisions?

I'd prefer not to couple the content-addressed storage to a concept like "package names" if possible. It's meant to be more general (e.g., we also use it for cached environments).

@charliermarsh charliermarsh Nov 22, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

({:2}/{2:4}/{4:} is what PyPI uses; it looks like pip does {:2}/{2:4}/{4:6}/{6:}?)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then rolled it back because it makes various things more complicated

Fair enough. I think people do it to avoid directory size limits (i.e., the number of items allowed in a single directory). I think we'd have had this problem already though if it was a concern for us? It seems fairly trivial to check both locations in the future if we determine we need it.

I'd prefer not to couple the content-addressed storage to a concept like "package names" if possible.

I think the idea that there's a "disambiguating" component for collisions if we truncate the hash doesn't need to be tied to "package names" specifically. The most generic way to do it would be to have /0, /1, ... directories with /{id}/HASH files and iterate over them? I sort of don't like that though :)

It's broadly unclear to me how much engineering we should do to avoid a long path length.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may not really matter. I can't remember the specifics but what ends up happening here is: we create a temp dir, unzip it, then we move the temp dir into this location and hardlink from this location. So I don't think we end up referencing paths within these archives?

@konstin konstin added enhancement New feature or improvement to existing functionality performance labels Dec 1, 2025
@zanieb

zanieb commented Dec 2, 2025

Copy link
Copy Markdown
Member

How does this relate to #888?

@konstin

konstin commented Dec 2, 2025

Copy link
Copy Markdown
Member

iirc The hash checking ideas of RECORD never materialized, pip doesn't check the RECORD and neither does uv, and there's plans to remove it (https://discuss.python.org/t/discouraging-deprecating-pep-427-style-signatures/94968). The consensus has shifted to using hashes and signature for the entire archive that are presented outside of the archive, on the index page, instead of being shipped with the archive.

@sandyharvie

Copy link
Copy Markdown

What's the likelihood of this being merged anytime soon @charliermarsh? Currently running into the following issue, which this would solve:

  • Node comes up, global cache is configured
  • Bunch of workers are scheduled all at once on the node, each of which proceed to set up their own virtual environment using this global cache
  • If N of these workers need a particular wheel, we often end up with N copies of that unzipped wheel in the cache

If the wheel happens to be, say, torch, this can cause the cache to balloon in size by ~2N GiB. When there are many such wheels and the node is sufficiently large (i.e., N is large), we see our cache instantly grow to upwards of 1 TiB!

@charliermarsh charliermarsh force-pushed the charlie/content branch 8 times, most recently from 8a87bf7 to 6d22a38 Compare March 26, 2026 02:32
@woodruffw

Copy link
Copy Markdown
Member

NB: If we do this, we might want to also remove our seahash dep and use BLAKE3 everywhere 🙂

@charliermarsh charliermarsh marked this pull request as draft May 30, 2026 14:06
@charliermarsh

Copy link
Copy Markdown
Member Author

I likely need to re-benchmark this prior to landing.

@charliermarsh charliermarsh marked this pull request as ready for review May 30, 2026 14:59
@astral-sh-bot

astral-sh-bot Bot commented May 30, 2026

Copy link
Copy Markdown

uv test inventory changes

This PR changes the tests when compared with the latest main baseline.

  • Added tests: 1
  • Removed tests: 0
  • Changed suites: 1
uv-distribution-types: +1 / -0

Added:

  • uv-distribution-types::hash::tests::algorithms_include_only_required_hashes

Removed: none

@charliermarsh

Copy link
Copy Markdown
Member Author

There doesn't seem to be much of an effect vs. main for local wheels (which is the case I'm worried about, since we now have to hash those in addition to unzipping them):

Wheel       Size    main mean    current mean    Result
━━━━━━━  ━━━━━━━━━  ━━━━━━━━━━━  ━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
tomli     12 KiB      44.6 ms         42.8 ms    current 1.04x faster
───────  ─────────  ───────────  ──────────────  ────────────────────────────────────
black    1.3 MiB      50.5 ms         50.3 ms    same
───────  ─────────  ───────────  ──────────────  ────────────────────────────────────
numpy     13 MiB     120.8 ms        122.7 ms    current 1.02x slower, within noise
───────  ─────────  ───────────  ──────────────  ────────────────────────────────────
torch     84 MiB    1225.8 ms       1223.1 ms    same

That said... I have no idea if this will generalize to other machines, since this is a pretty hefty MacBook Pro.

@zanieb

zanieb commented Jun 2, 2026

Copy link
Copy Markdown
Member

What's the story for compatibility with the existing archive storage?

I can try to do some benchmarking on smaller machines too.

@charliermarsh

Copy link
Copy Markdown
Member Author

What's the story for compatibility with the existing archive storage?

It "just works". If you have existing archives, we continue to use them. If you access a new archive, we generate a content-addressed ID, and those IDs never collide with our existing IDs.

@zanieb zanieb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — I'll report back with some more benchmarks tomorrow.

@charliermarsh

Copy link
Copy Markdown
Member Author

Thanks. I'm also having Codex explore whether there's a way to use blake3 hazmat to do something more clever to avoid reading the file twice. In general I'm nervous about the change as-is due to the performance implications for large local files...

@zanieb

zanieb commented Jun 2, 2026

Copy link
Copy Markdown
Member

The wheels you tested on a Namespace runner with restricted CPU counts

CPU tomli black numpy torch Geomean
2 0.987x 0.970x 1.010x 1.017x 0.996x
4 1.031x 0.916x 0.973x 0.935x 0.963x
8 0.989x 1.024x 0.998x 0.986x 0.999x
16 1.000x 0.982x 0.998x 1.022x 1.000x

And some synthetic wheel shapes

Shape Ratio
single stored 64 MiB 1.023x
single stored 256 MiB 1.026x
10k tiny files 1.007x
1k medium files 1.025x
single deflated zero 512 MiB 0.993x

And some synthetic "big file" cases

CPU 512 MiB 1 GiB
2 0.996x same 1.037x slower
4 1.034x slower 1.060x slower
8 1.039x slower 1.063x slower
16 1.083x slower 1.105x slower

So it looks like the worst-case is the 1 GiB / 16-core case which was about +27 ms / 10.5% slower.

I'm a bit confused it got comparatively slower on large files as the CPU count increased, so I'm going to look into that.

@charliermarsh

Copy link
Copy Markdown
Member Author

Just to double-confirm, these are local wheels already available on disk right?

@zanieb

zanieb commented Jun 2, 2026

Copy link
Copy Markdown
Member

Yep!

@charliermarsh charliermarsh changed the title Content-address distributions in the archive Content-address distributions in the cache by distribution hash Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or improvement to existing functionality performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Store distributions in cache with content addressed keys Store symlinked directories under their SHA weird behaviour of uv sync and the uv cache

5 participants