Summary

Wrap download progress updaters in AggregatingProgressUpdater to eliminate GIL contention when Python callers provide per-file progress callbacks.

The upload path has had this aggregation since v1.1.3 (PR #340), but the download path was missed. Without aggregation, each XORB chunk triggers a spawn_blocking + Python::with_gil() callback. With many concurrent file downloads, this causes severe GIL contention — measured as a 4x throughput reduction (3000 MB/s → 750 MB/s on a 25 Gbps link).

The fix wraps the caller-provided TrackingProgressUpdater in an AggregatingProgressUpdater (200ms flush interval) inside download_file_with_updater(), matching the pattern already used by FileUploadSession. This reduces Python callback frequency from thousands/sec to ~5/sec per file while preserving progress bar feedback.

Root cause

When huggingface_hub calls hf_xet.download_files(), it passes a per-file Python callback for progress bar updates. On the Rust side, each callback invocation goes through:

report_bytes_written() / report_transfer_progress()
  → tokio::spawn(register_updates())
    → spawn_blocking(Python::with_gil(callback))

With the detailed download progress tracking added in PR #645 (hf-xet v1.3.0), both report_bytes_written and report_transfer_progress fire per chunk, roughly doubling callback frequency. With 8+ concurrent file downloads, each spawning dozens of concurrent XORB streams, the GIL becomes a severe bottleneck.

History

The problem has existed since xet download support was introduced, but worsened over time:

PR #340 explicitly noted: "each [update] has to acquire a global GIL lock. This negatively affects the upload speed on fast connections" — the same problem, but only the upload side was addressed.

Benchmarks

Downloading 3 safetensors files (16.1 GB total) from Qwen/Qwen3.5-35B-A3B on a 25 Gbps machine:

Progress callback overhead drops from 4x slowdown to <1%.