Git-Xet: LFS custom transfer agent with Xet protocol by seanses · Pull Request #425 · huggingface/xet-core

seanses · 2025-07-24T22:11:04Z

This PR builds a Git integration called git-xet that enables users to upload files using the Xet protocol as part of a standard git push.

This integration builds on the Git LFS custom transfer adapter protocol, the same mechanism we now use to handle Git LFS uploads for files larger than 5 GB through multipart PUT.
To enable uploads to Xet, users run git-xet install, which writes the following configuration to the Git config file at a selected scope [--system, --global (default), or --local]:

[lfs "customtransfer.xet"]
	path = git-xet
	args = transfer
	concurrent = true

This setup registers a new transfer adapter named xet, allowing Git to delegate LFS file transfers to the git-xet binary when applicable.

On the server side, support is rolled out in two stages:

Stage 1 (Upload): The Git LFS batch API for the "upload" operation is updated.

If a repo is Xet enabled but users didn't run git-xet install, moon-landing rejects the request when users initiated git push and returns an instruction to install git-xet. (see demo in screenshot 1).

If a repo is Xet enabled and users have git-xet configured correctly, moon-landing accepts the request and replies with CAS server URL and access token, which git-xet will use to upload files to Xet. (see demo in screenshot 2 & 3).

If a repo is NOT Xet enabled, upload goes through the LFS path.

Stage 2 (Download): The same logic is applied to the Git LFS batch API for the "download" operation.

Server side changes are at https://github.com/huggingface-internal/moon-landing/pull/14572

hf_xet_git_integration/Cargo.toml

git_xet/src/errors.rs

.github/workflows/ci.yml

git_xet/src/test_utils/mod.rs

git_xet/src/lfs_agent_protocol.rs

git_xet/src/lfs_agent_protocol/protocol_spec.rs

git_xet/src/hub_client.rs

git_xet/src/app/xet_agent.rs

…d style

jgodlew

Overall, looks good, I mostly have nits.

git_xet/src/git_url.rs

git_xet/src/hub_client.rs

git_xet/src/lfs_agent_protocol/agent_state.rs

git_xet/src/lfs_agent_protocol/progress_updater.rs

git_xet/src/lfs_agent_protocol.rs

git_xet/src/app/xet_agent.rs

git_xet/src/lfs_agent_protocol/progress_updater.rs

git_xet/src/app.rs

git_xet/src/auth.rs

assafvayner · 2025-08-28T17:19:37Z

git_xet/src/auth.rs

+//     user for a password to fetch public repository data.
+//  2. The Git Remote URL, which should be something like "https://git.com/repo.git" This URL is used for the Git
+//     Credential Helper. This way existing https Git remote credentials can be re-used for LFS.
+pub fn get_creds(repo: &GitRepo, remote_url: &GitUrl, operation: Operation) -> Result<Arc<dyn CredentialHelper>> {


should we have debug/info logs for the selected credential helper?

git_xet/src/auth.rs

assafvayner · 2025-08-28T17:23:10Z

git_xet/src/auth.rs

+    }
+
+    // 6. check Git credential helper
+    Ok(GitCredentialHelper::new(repo.git_path()?, &derived_host_url)?)


in case this fails as in the git credential returns nothing, maybe log that we failed to use the git credential command.

git_xet/src/app/xet_agent.rs

assafvayner · 2025-08-28T17:40:52Z

git_xet/src/app.rs

+struct CliOverrides {
+    /// Increase verbosity of output (-v, -vv, etc.)
+    #[clap(long, short = 'v', action = ArgAction::Count)]
+    pub verbose: u8,


we need to set up logging somewhere, unless it's set up already? we should also make distinction for if we respect RUST_LOG over the -v flag

also set a max/min maybe? I'm guessing it's:
0 - error
1 - warn
2 - info
3 - debug
4 - trace
so min 0, max 4

rajatarya

@seanses I am only through first 15 files of this PR, will review the remaining files later today.

Overall I LOVE the tests and the comments - my suggestions are mostly around adding more comments.

git_xet/src/app/install.rs

git_xet/src/app/xet_agent.rs

rajatarya · 2025-08-29T18:24:52Z

git_xet/src/app/xet_agent.rs

+    }
+
+    async fn terminate(&mut self) -> Result<()> {
+        Ok(())


Should we log something here? Is terminate() expected to be called by someone? I am wondering if a debug/trace log message makes sense here.

git_xet/src/app/xet_agent.rs

git_xet/src/lfs_agent_protocol/protocol_spec.rs

git_xet/src/app.rs

rajatarya

This is overall looking good. Most of my comments are to add documentation/comments and maybe restructure code for easier readability.

The child process part needs some additional testing.

Great work @seanses! This is a big feature and it is great to see its implementation really close to completed!

git_xet/src/auth.rs

git_xet/src/git_url.rs

git_xet/src/hub_client.rs

git_xet/Cargo.toml

git_xet/src/lfs_agent_protocol.rs

git_xet/src/lib.rs

git_xet/src/lfs_agent_protocol/progress_updater.rs

hoytak

Some minor things, but overall looks good to me. I'm assuming it works overall. I suspect there will be a lot to tweak when it gets used more widely, but it looks solid.

rajatarya

Looking good! I think this is ready to merge now, and any suggestions could come in subsequent PRs.

Love the comments added and the additional tests.

git_xet/src/app/install.rs

git_xet/src/app/xet_agent.rs

git_xet/src/auth.rs

git_xet/src/auth/git.rs

git_xet/src/auth/ssh.rs

LFS custom transfer agent with Xet protocol

bba2d6b

seanses marked this pull request as draft July 24, 2025 22:19

seanses added 3 commits July 24, 2025 16:22

fix linting errors

59d2557

fix fmt

a1bad5b

fix test

a656f13

assafvayner reviewed Jul 28, 2025

View reviewed changes

hf_xet_git_integration/Cargo.toml Show resolved Hide resolved

seanses added 5 commits August 6, 2025 11:26

checkpoint

ddab889

rename crate; restructure; tests; progress updater

9995218

token refresher

e2e91c5

Merge branch 'main' into di/lfs-integration

acd8588

and everything

ccee1a5

seanses changed the title ~~LFS custom transfer agent with Xet protocol~~ Git-Xet: LFS custom transfer agent with Xet protocol Aug 21, 2025

seanses marked this pull request as ready for review August 21, 2025 17:40

seanses requested review from bpronan, hoytak, jgodlew and rajatarya August 21, 2025 17:51

seanses added 3 commits August 21, 2025 12:30

fix fmt

dfe3194

set up git test environment

a6c4e04

Merge branch 'main' into di/lfs-integration

2de6bba

assafvayner reviewed Aug 21, 2025

View reviewed changes

seanses added 7 commits August 21, 2025 19:09

"git config set" is a very new feature since 2.46.0, let's use the ol…

818c7b1

…d style

upload session use the progress config; fix typo

5a75d96

GitCredentialHelper uses repo path

b896796

update hf_xet Cargo.lock

91c4d6f

more credential helper tests

c0219fd

use the remote provided in the init event

e253994

Merge branch 'main' into di/lfs-integration

a3aa0f9

jgodlew approved these changes Aug 27, 2025

View reviewed changes

assafvayner reviewed Aug 28, 2025

View reviewed changes

rajatarya reviewed Aug 29, 2025

View reviewed changes

rajatarya reviewed Aug 30, 2025

View reviewed changes

seanses added 5 commits September 2, 2025 14:13

hub_client as a separate crate

5ed6cc6

uninstall; more comments; more tests; restructure; code style

c59aa7c

set up git-lfs in ci

4af0c3c

fix git-lfs set up; remove lint step on Windows

436ef1d

git for windows comes with git-lfs

d1546a1

hoytak reviewed Sep 5, 2025

View reviewed changes

git_xet/src/lfs_agent_protocol/progress_updater.rs Outdated Show resolved Hide resolved

hoytak reviewed Sep 5, 2025

View reviewed changes

git_xet/src/lfs_agent_protocol/progress_updater.rs Outdated Show resolved Hide resolved

hoytak approved these changes Sep 5, 2025

View reviewed changes

seanses added 3 commits September 8, 2025 12:37

clean code; use fetch_max

ba641dc

fix fmt

182ebda

Merge branch 'main' into di/lfs-integration

b0123e6

rajatarya approved these changes Sep 8, 2025

View reviewed changes

change the install command concurrency option to accept u32

bae4e56

seanses merged commit 0e1f9f4 into main Sep 8, 2025
6 checks passed

seanses deleted the di/lfs-integration branch September 8, 2025 23:08

Conversation

seanses commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgodlew left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

assafvayner Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

assafvayner Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

assafvayner Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

rajatarya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rajatarya Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rajatarya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hoytak left a comment

Choose a reason for hiding this comment

Uh oh!

rajatarya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

seanses commented Jul 24, 2025 •

edited

Loading