Support Git LFS with opt-in by samypr100 · Pull Request #16143 · astral-sh/uv

samypr100 · 2025-10-06T21:13:10Z

Summary

Follow up to #15563
Closes #13485

This is a first-pass at adding support for conditional support for Git LFS between git sources, initial feedback welcome.

e.g.

[tool.uv.sources]
test-lfs-repo = { git = "https://github.com/zanieb/test-lfs-repo.git", lfs = true }

For context previously a user had to set UV_GIT_LFS to have uv fetch lfs objects on git sources. This env var was all or nothing, meaning you must always have it set to get consistent behavior and it applied to all git sources. If you fetched lfs objects at a revision and then turned off lfs (or vice versa), the git db, corresponding checkout lfs artifacts would not be updated properly. Similarly, when git source distributions were built, there would be no distinction between sources with lfs and without lfs. Hence, it could corrupt the git, sdist, and archive caches.

In order to support some sources being LFS enabled and other not, this PR adds a stateful layer roughly similar to how subdirectory works but for lfs since the git database, the checkouts and the corresponding caching layers needed to be LFS aware (requested vs installed). The caches also had to isolated and treated entirely separate when handling LFS sources.

Summary

Adds lfs = true or lfs = false to git sources in pyproject.toml
Added lfs=true query param / fragments to most relevant url structs (not parsed as user input)
- In the case of uv add / uv tool, --lfs is supported instead
- UV_GIT_LFS environment variable support is still functional for non-project entrypoints (e.g. uv pip)
direct-url.json now has an custom git_lfs entry under VcsInfo (note, this is not in the spec currently -- see caveats).
git database and checkouts have an different cache key as the sources should be treated effectively different for the same rev.
sdists cache also differ in the cache key of a built distribution if it was built using LFS enabled revisions to distinguish between non-LFS same revisions. This ensures the strong assumption for archive-v0 that an unpacked revision "doesn't change sources" stays valid.

Caveats

pylock.toml import support has not been added via git_lfs=true, going through the spec it wasn't clear to me it's something we'd support outside of the env var (for now).
direct-url struct was modified by adding a non-standard git_lfs field under VcsInfo which may be undersirable although the PEP 610 does say Additional fields that would be necessary to support such VCS SHOULD be prefixed with the VCS command name which could be interpret this change as ok.
There will be a slight lockfile and cache churn for users that use UV_GIT_LFS as all git lockfile entries will get a lfs=true fragment. The cache version does not need an update, but LFS sources will get their own namespace under git-v0 and sdist-v9/git hence a cache-miss will occur once but this can be sufficient to label this as breaking for workflows always setting UV_GIT_LFS.

Test Plan

Some initial tests were added. More tests likely to follow as we reach consensus on a final approach.

For IT test, we may want to move to use a repo under astral namespace in order to test lfs functionality.

Manual testing was done for common pathological cases like killing LFS fetch mid-way, uninstalling LFS after installing an sdist with it and reinstalling, fetching LFS artifacts in different commits, etc.

PSA: Please ignore the docker build failures as its related to depot OIDC issues.

codspeed-hq · 2025-10-17T22:11:35Z

CodSpeed Performance Report

Merging #16143 will not alter performance

_{Comparing samypr100:conditional_lfs_support (dc1b6ba) with main (5947fb0)}

Summary

✅ 6 untouched

geofft

Thanks, and sorry for the delay in the re-review. I'm not totally following some of the code changes here but the behavior does seem to match what we want, i.e., sharing the db/* directory and only having separate checkouts/*/* directories. I might take a look at that later and send you something for review but that doesn't block merging this.

geofft · 2025-11-18T19:33:29Z

There is one theoretical backwards-incompatibility here: with current uv, you only have to specify UV_GIT_LFS=1 once and the checkout and built wheel gets cached with the LFS artifacts included. This PR (correctly) identifies that as a risk to cache consistency, but I think in practice it may work fine for users with git-lfs installed and configured properly, because the .ok file won't get created if the LFS fetch fails or otherwise leaves you in an inconsistent state, and I am worried that people might be relying on that in practice.

With current uv:

$ UV_CACHE_DIR=/tmp/cache/q2-old UV_GIT_LFS=1 uvx git+https://github.com/astral-sh/lfs-cowsay -v
    Updated https://github.com/astral-sh/lfs-cowsay (44b8e65c8ecd376e4482a68b29312227879a226d)
      Built lfs-cowsay @ git+https://github.com/astral-sh/lfs-cowsay@44b8e65c8ecd376e4482a68b29312227879a226d
Installed 1 package in 2ms
6.1
$ UV_CACHE_DIR=/tmp/cache/q2-old uvx git+https://github.com/astral-sh/lfs-cowsay -v
6.1

With the code from this PR:

$ UV_CACHE_DIR=/tmp/cache/q2-new UV_GIT_LFS=1 target/debug/uvx git+https://github.com/astral-sh/lfs-cowsay -v
    Updated https://github.com/astral-sh/lfs-cowsay (44b8e65c8ecd376e4482a68b29312227879a226d)
      Built lfs-cowsay @ git+https://github.com/astral-sh/lfs-cowsay@44b8e65c8ecd376e4482a68b29312227879a226d#lfs=true
Installed 1 package in 19ms
6.1
$ UV_CACHE_DIR=/tmp/cache/q2-new target/debug/uvx git+https://github.com/astral-sh/lfs-cowsay -v
      Built lfs-cowsay @ git+https://github.com/astral-sh/lfs-cowsay@44b8e65c8ecd376e4482a68b29312227879a226d
Installed 1 package in 18ms
Traceback (most recent call last):
  File "/tmp/cache/q2-new/archive-v0/UHbaC9QcSG9WzgquNtN5V/bin/lfs-cowsay", line 6, in <module>
    from lfs_cowsay.__main__ import cli
  File "/tmp/cache/q2-new/archive-v0/UHbaC9QcSG9WzgquNtN5V/lib/python3.14/site-packages/lfs_cowsay/__init__.py", line 4, in <module>
    from .main import (
    ...<13 lines>...
    )
  File "/tmp/cache/q2-new/archive-v0/UHbaC9QcSG9WzgquNtN5V/lib/python3.14/site-packages/lfs_cowsay/main.py", line 4, in <module>
    from .characters import CHARS
  File "/tmp/cache/q2-new/archive-v0/UHbaC9QcSG9WzgquNtN5V/lib/python3.14/site-packages/lfs_cowsay/characters.py", line 2
    oid sha256:4a003717d1e1997c56ef4128f54eb92c72831e44cda4e3d49e93a32ae0cc9222
               ^
SyntaxError: invalid decimal literal

(This does not affect UV_GIT_LFS=1 uv tool install blah + uv tool upgrade, though - we correctly persist the ?lfs=true in the URL. It only affects two independent parses of the same git repo/commit that do not share persistent state but share the cache.)

Again, I don't think this is a problem in practice / don't think this should block merging.

crates/uv-distribution-types/src/requirement.rs

crates/uv-git-types/src/lib.rs

crates/uv-installer/src/satisfies.rs

docs/concepts/projects/dependencies.md

We're not very consistent in adding spans for git operations, but we should at least for new functions.

konstin

Thank you!

Red-Eyed · 2025-12-03T06:17:02Z

Thank you!
Having worked with LFS, and I wonder how to deal with different LFS endpoints and credentials?
For example, at work we have GitHub enterprise and external git LFS server (not github)

samypr100 · 2025-12-03T14:34:53Z

Thank you! Having worked with LFS, and I wonder how to deal with different LFS endpoints and credentials? For example, at work we have GitHub enterprise and external git LFS server (not github)

@Red-Eyed That would still be configured as part of your git configuration. Since LFS objects have the originating url on them, as long as git itself can authenticate, fetch, and materialize the objects you shouldn't have issues with uv.

gsemet · 2025-12-06T10:15:56Z

Great ! Is there a way to use it when doing « uv tool install git+https//… »?

samypr100 · 2025-12-06T12:08:39Z

Great ! Is there a way to use it when doing « uv tool install git+https//… »?

uv tool install --lfs

samypr100 force-pushed the conditional_lfs_support branch 8 times, most recently from b34deec to 52cbd5c Compare October 13, 2025 19:37

samypr100 force-pushed the conditional_lfs_support branch 2 times, most recently from 6b19653 to 17d5d16 Compare October 17, 2025 21:51

samypr100 force-pushed the conditional_lfs_support branch 6 times, most recently from ce1cfb1 to f222e06 Compare October 21, 2025 15:55

samypr100 marked this pull request as ready for review October 21, 2025 15:59

samypr100 force-pushed the conditional_lfs_support branch 11 times, most recently from 8d3c6f0 to 50cf0e5 Compare October 29, 2025 16:20

samypr100 force-pushed the conditional_lfs_support branch from 50cf0e5 to 81d1024 Compare October 29, 2025 22:00

samypr100 force-pushed the conditional_lfs_support branch from 3c0b7fb to 63883ef Compare November 14, 2025 05:14

samypr100 added the test:macos Enable macOS tests for a pull request label Nov 14, 2025

samypr100 force-pushed the conditional_lfs_support branch 3 times, most recently from b0ab2da to 38b3990 Compare November 15, 2025 00:43

geofft approved these changes Nov 18, 2025

View reviewed changes

messerli-wallace approved these changes Nov 18, 2025

View reviewed changes

zanieb mentioned this pull request Nov 21, 2025

fixed git submodule bug when using relative paths #12156

Open

konstin reviewed Nov 28, 2025

View reviewed changes

zanieb and others added 10 commits December 1, 2025 23:51

Add lfs = true support in Git source declarations

6102f52

chore: initial cleanup

4fa3e77

fix: ensure lfs state is propagated in the git database and checkouts

1a9cb16

fix: support stateful lfs fragments and indicators

070c2e7

fix: proper caching lfs-based built wheels

f34b1ac

tests: initial uv-sync test adjustments

511c8b1

feat: support --lfs in uv add

0ce00eb

feat: support --lfs in uv tool run/install

bd397b2

docs: add documentation for lfs usage and configuration

70bd58b

Add span for LFS git fsck

d90ebc7

We're not very consistent in adding spans for git operations, but we should at least for new functions.

konstin mentioned this pull request Dec 2, 2025

Add a check for accidental cache modifications #16927

Closed

Final touch-ups

dc1b6ba

konstin approved these changes Dec 2, 2025

View reviewed changes

BrewTestBot mentioned this pull request Dec 3, 2025

uv 0.9.15 Homebrew/homebrew-core#256853

Merged

zanieb mentioned this pull request Dec 13, 2025

Add lfs = true support in Git source declarations #15563

Closed

traversaro mentioned this pull request Feb 11, 2026

Smudge errors when installing a package from git prefix-dev/pixi#2000

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Git LFS with opt-in#16143

Support Git LFS with opt-in#16143
konstin merged 11 commits intoastral-sh:mainfrom
samypr100:conditional_lfs_support

samypr100 commented Oct 6, 2025 •

edited

Loading

Uh oh!

codspeed-hq bot commented Oct 17, 2025 •

edited

Loading

Uh oh!

geofft left a comment

Uh oh!

geofft commented Nov 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

konstin left a comment

Uh oh!

Red-Eyed commented Dec 3, 2025 •

edited

Loading

Uh oh!

samypr100 commented Dec 3, 2025

Uh oh!

gsemet commented Dec 6, 2025

Uh oh!

samypr100 commented Dec 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

samypr100 commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Uh oh!

codspeed-hq bot commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #16143 will not alter performance

Summary

Uh oh!

geofft left a comment

Choose a reason for hiding this comment

Uh oh!

geofft commented Nov 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

konstin left a comment

Choose a reason for hiding this comment

Uh oh!

Red-Eyed commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samypr100 commented Dec 3, 2025

Uh oh!

gsemet commented Dec 6, 2025

Uh oh!

samypr100 commented Dec 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

samypr100 commented Oct 6, 2025 •

edited

Loading

codspeed-hq bot commented Oct 17, 2025 •

edited

Loading

Red-Eyed commented Dec 3, 2025 •

edited

Loading