XetSession API by seanses · Pull Request #657 · huggingface/xet-core

seanses · 2026-02-13T18:56:05Z

This PR introduces a new xet_session crate that provides a session-based hierarchical API: Users create a XetSession to manage runtime and configuration, then batch uploads into UploadCommit objects and downloads into DownloadGroup objects — each of which runs transfers in the background by the inner XetRuntime.

All pub functions are exposed as sync functions - making them easy to use in other languages, e.g. Python, C, ...

Will expose another interface for Rust async native environment.

Example

let session = XetSession::new(XetConfig::new(...));

let upload_commit = session.new_upload_commit();

let handle = upload_commit.upload_file_from_path(file_path); // return immediately
let handle = upload_commit.upload_bytes(bytes); // return immediately

let progress = upload_commit.get_progress(); // return a collection of upload progress

let result = upload_commit.commit(); // blocking wait: finalizing upload, pushes all data to remote, return a collection of file metadata (hash and size)

let download_group = session.new_download_group();
let handle = download_group.download_file_to_path(file_hash, dest_path); // return immediately

let progress = download_group.get_progress(); // return a collection of download progress

let result = download_group.finish() // blocking wait: return a collection of download results (metadata)

Next steps

Make progress reporting pull-based not push-based: introduces a get_process() function to FileUploadSession and FileDownloadSession so XetSession can pull.
Plug into Xet migration service to test.

… tracking

hoytak

Per discussion, I think we should get this in and then iterate rather than hold it up longer. Many todos, but let's merge and then update it.

…leanup

assafvayner

Some comments.

I do have 2 additional requests for this revision, then more content can go in in the future.

Add a README.md in xet_session, should be easy just generate and review. This should describe the general interface exposed and the motivation.
^ should fix this, If you run cargo doc on this crate, the first page is pretty empty, so we should fill something in there, doesn't have to be final/release version, just something.

xet_session/Cargo.toml

xet_session/src/session.rs

xet_session/src/download_group.rs

xet_session/src/errors.rs

rajatarya · 2026-03-03T23:14:07Z

Code Review Summary

Well-structured session API with a clean hierarchical design (XetSession → UploadCommit/DownloadGroup). The sync-over-async approach is appropriate for FFI consumers.

Existing review feedback already covers: inner struct visibility/Deref pattern, builder pattern for XetSession, std::sync::Mutex usage, and README/docs. These findings are omitted below.

P0 — Must Fix

Example will deadlock — #[tokio::main] + commit() (which calls external_run_async_task → block_on) deadlocks because you can't block_on inside a Tokio runtime.
handle_commit/handle_finish abandon remaining tasks on first error — ?? propagates immediately, dropping un-awaited JoinHandles. Remaining tasks continue running after commit()/finish() returns.

P1 — Should Fix

HashMap::drain() returns results in non-deterministic order — commit()/finish() return Vec<FileMetadata>/Vec<DownloadResult> in arbitrary order. Use BTreeMap<Ulid> (time-ordered) or document.
abort() fails partway on PoisonError — If status.lock()? fails mid-iteration, remaining tasks are left un-cancelled. XetSession::abort() also returns early on the first failing sub-abort.
file_size field assigned wrong value in download — result? is n_bytes downloaded, not the logical file size. Should use file_info.file_size.

P2 — Nice to Fix

crate-type includes cdylib/staticlib — triples compile time unless FFI is needed now
Double Arc wrapping of XetRuntime — XetRuntime is already internally Arc-wrapped
Silent progress drops on poisoned locks in register_updates
Download concurrency semaphore moved to caller with only a comment as the contract

See inline comments for exact locations of P0/P1 issues.

rajatarya

Inline comments on high-priority issues.

xet_session/examples/example.rs

xet_session/src/upload_commit.rs

xet_session/src/download_group.rs

xet_session/src/upload_commit.rs

xet_session/src/download_group.rs

rajatarya

Don't block on feedback from Claude - but I think the P0/P1 feedback sounds reasonable to address - either prior to merging or after merging in a subsequent PR.

… unit tests

seanses and others added 4 commits February 13, 2026 10:49

session interface draft

e618d30

Streaming data writer.

4ac2baa

checkpoint

7a0eff1

Merge branch 'main' into di/session-interface

c9839f1

seanses changed the title ~~draft~~ XetSession API Feb 25, 2026

update

248f584

seanses force-pushed the di/session-interface branch from 210aced to 248f584 Compare February 25, 2026 01:19

seanses added 3 commits February 25, 2026 16:28

change tracking ids from Arc<str> to ulid

27e9b7f

update

46fb034

merge branch di/change_tracking_id_to_ulid and plumb through progress…

22f3726

… tracking

seanses changed the base branch from main to di/change_tracking_id_to_ulid February 26, 2026 20:57

fix cargo machete

c3bbaee

seanses marked this pull request as ready for review February 26, 2026 21:03

seanses requested review from assafvayner, hoytak and rajatarya February 26, 2026 21:04

hoytak approved these changes Feb 27, 2026

View reviewed changes

seanses added 4 commits February 27, 2026 18:00

use external_run_async_task and evaludate eagerly

17276ce

impl todos; more accurate status reporting; better unit tests; code c…

f1db3f9

…leanup

round-trip tests; code cleanup; fix comments

afd1b8d

Merge branch 'main' into di/session-interface

45b1fb7

seanses changed the base branch from di/change_tracking_id_to_ulid to main March 3, 2026 01:30

fix test

7710c68

seanses mentioned this pull request Mar 3, 2026

Change progress tracking ids from Arc<str> to Ulid #667

Closed

assafvayner approved these changes Mar 3, 2026

View reviewed changes

rajatarya reviewed Mar 3, 2026

View reviewed changes

rajatarya approved these changes Mar 3, 2026

View reviewed changes

address feedback; update crate doc; better group status handling; add…

da947c0

… unit tests

seanses merged commit c4a56f8 into main Mar 4, 2026
7 checks passed

seanses deleted the di/session-interface branch March 4, 2026 04:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

XetSession API#657

XetSession API#657
seanses merged 15 commits intomainfrom
di/session-interface

seanses commented Feb 13, 2026 •

edited

Loading

Uh oh!

hoytak left a comment

Uh oh!

assafvayner left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rajatarya commented Mar 3, 2026

Uh oh!

rajatarya left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rajatarya left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

seanses commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example

Next steps

Uh oh!

hoytak left a comment

Choose a reason for hiding this comment

Uh oh!

assafvayner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rajatarya commented Mar 3, 2026

Code Review Summary

P0 — Must Fix

P1 — Should Fix

P2 — Nice to Fix

Uh oh!

rajatarya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rajatarya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

seanses commented Feb 13, 2026 •

edited

Loading