Skip to content

okaybeydanol/hf-model-pull

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 
 
 

Repository files navigation

hf-model-pull.sh

Fast Hugging Face model repository downloader powered by aria2c.

hf-model-pull.sh downloads complete Hugging Face model repositories with one command using aria2c segmented downloads. It is designed for large LLM repositories where standard download methods may underuse available bandwidth.

Instead of manually copying every .safetensors, .gguf, tokenizer, config, or shard filename, the script automatically discovers all repository files, preserves subdirectories, supports resume, and can use HF_TOKEN for gated/private repositories.

When installed locally, the script can be used as the hf-model-pull command.

Quick Start

sudo apt update
sudo apt install -y curl aria2 python3

chmod +x hf-model-pull.sh
./hf-model-pull.sh Qwen/Qwen3-0.6B

The model is downloaded into the current working directory.

Example output:

./Qwen3-0.6B/

Why this exists

Large Hugging Face model repositories often contain many files, including huge model shards, tokenizer files, configs, .safetensors, .gguf, README files, and metadata.

Downloading these repositories manually can be slow and error-prone.

Standard Hugging Face download methods can also underuse available internet bandwidth when downloading very large model repositories.

hf-model-pull.sh was created to make full Hugging Face model repository downloads faster and simpler by using aria2c segmented downloads.

The script uses multiple connections per file:

-x 16
-s 16
-j 1

This helps large model files download much closer to your available connection speed, while still preserving the original repository folder structure and supporting resume for interrupted downloads.

Features

  • Download complete Hugging Face model repositories
  • Improve large model download throughput with aria2c segmented connections
  • Automatically discover repository files from the Hugging Face API
  • No need to manually write .safetensors, .gguf, tokenizer, config, or shard filenames
  • Preserve repository subdirectories locally
  • Resume interrupted downloads cleanly
  • Support public, gated, and private repositories
  • Keep Hugging Face tokens out of the script
  • Works both as a direct script and as a local terminal command

Requirements

Ubuntu/Debian/Lubuntu:

sudo apt update
sudo apt install -y curl aria2 python3

Required commands:

bash
curl
python3
aria2c

Check:

curl --version
python3 --version
aria2c --version

Run Directly

chmod +x hf-model-pull.sh
./hf-model-pull.sh Qwen/Qwen3-0.6B

Another example:

./hf-model-pull.sh unsloth/Qwen3.6-35B-A3B-MTP-GGUF

The script downloads the model into the current working directory.

Example:

./hf-model-pull.sh Qwen/Qwen3-0.6B

Creates:

./Qwen3-0.6B/

Install Locally

You can install it as a local command:

mkdir -p ~/.local/bin
cp hf-model-pull.sh ~/.local/bin/hf-model-pull
chmod +x ~/.local/bin/hf-model-pull

Make sure ~/.local/bin is in your PATH:

echo $PATH | grep -o "$HOME/.local/bin"

If it is missing, add this to ~/.bashrc or ~/.zshrc:

export PATH="$HOME/.local/bin:$PATH"

Then reload your shell:

source ~/.bashrc
# or
source ~/.zshrc

Run:

hf-model-pull Qwen/Qwen3-0.6B

Usage

hf-model-pull repo_id

Examples:

hf-model-pull Qwen/Qwen3-0.6B
hf-model-pull Qwen/Qwen3.6-35B-A3B
hf-model-pull unsloth/Qwen3.6-35B-A3B-MTP-GGUF

Repository IDs must use the Hugging Face format:

owner/repo

Valid:

Qwen/Qwen3-0.6B
unsloth/Qwen3.6-35B-A3B-MTP-GGUF

Invalid:

Qwen3-0.6B
https://huggingface.co/Qwen/Qwen3-0.6B

Gated or Private Repositories

Public repositories work without a token.

For gated or private Hugging Face repositories, export your token first:

export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxx"

Then run:

hf-model-pull meta-llama/Some-Gated-Model

Do not hardcode your Hugging Face token inside the script.

Output Behavior

The model is downloaded into the current working directory.

The output folder name is generated from the repository name.

Example:

hf-model-pull Qwen/Qwen3-0.6B

Creates:

./Qwen3-0.6B/

If the repository contains subdirectories, they are preserved locally.

Example Hugging Face file:

subfolder/model-00001-of-00004.safetensors

Becomes:

./Qwen3-0.6B/subfolder/model-00001-of-00004.safetensors

Resume Interrupted Downloads

Downloads are resumable.

If a download is interrupted, run the same command again:

hf-model-pull Qwen/Qwen3-0.6B

aria2c continues partially downloaded files instead of starting from zero.

How It Works

The script:

  1. Takes a Hugging Face repository ID:
Qwen/Qwen3-0.6B
  1. Queries the Hugging Face model tree API:
https://huggingface.co/api/models/Qwen/Qwen3-0.6B/tree/main?recursive=1
  1. Extracts every file path from the repository metadata.

  2. Builds an aria2c input list.

  3. Downloads each file from:

https://huggingface.co/<owner>/<repo>/resolve/main/<file>
  1. Saves the files into a local folder named after the repository.

aria2 Settings

The script uses fixed high-throughput aria2c settings:

-x 16
-s 16
-j 1

Meaning:

  • -x 16: use up to 16 connections per file
  • -s 16: split each file into 16 parts
  • -j 1: download one file at a time

Hugging Face often behaves better with one active file and many connections instead of many parallel files at once.

Other important options:

-c
--continue=true
--file-allocation=none
--retry-wait=5
--max-tries=0
--allow-overwrite=true
--auto-file-renaming=false

These options help resume downloads, avoid duplicate renamed files, and keep retrying when the connection is unstable.

Notes

  • The script downloads from the main revision.
  • The script itself can live in ~/.local/bin.
  • Model files are downloaded into the directory where you run the command.
  • Downloads do not go into ~/.local/bin unless you run the command from there.
  • Public repositories do not need HF_TOKEN.
  • Gated/private repositories require HF_TOKEN.
  • The script rejects unsafe repository paths such as absolute paths or paths containing ...
  • Temporary files are created under /tmp and removed automatically when the script exits.

Troubleshooting

aria2c is not installed

Install it:

sudo apt update
sudo apt install -y aria2

curl is not installed

Install it:

sudo apt update
sudo apt install -y curl

python3 is not installed

Install it:

sudo apt update
sudo apt install -y python3

Repository file list is empty

Possible reasons:

  • The repository ID is wrong
  • The repository is private
  • The repository is gated
  • HF_TOKEN is missing
  • The token does not have access to the repository

Set your token:

export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxx"

Then retry:

hf-model-pull owner/repo

Invalid repo id

The script expects:

owner/repo

Use:

hf-model-pull Qwen/Qwen3-0.6B

Not:

hf-model-pull Qwen3-0.6B

Download was interrupted

Run the same command again:

hf-model-pull Qwen/Qwen3-0.6B

The download will continue.

Example Workflow

Create a models directory:

mkdir -p ~/models
cd ~/models

Download a model:

hf-model-pull Qwen/Qwen3-0.6B

Result:

~/models/Qwen3-0.6B/

Download a GGUF repository:

hf-model-pull unsloth/Qwen3.6-35B-A3B-MTP-GGUF

Result:

~/models/Qwen3.6-35B-A3B-MTP-GGUF/

License

MIT

About

Fast Hugging Face model repository downloader using aria2c segmented downloads for better bandwidth usage and resume support.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages