The Unsloth provider enables 2x faster training with 20% less memory through optimized CUDA kernels and efficient memory management.

Unsloth is a specialized library that patches HuggingFace Transformers to use optimized implementations for:

• Flash Attention 2
• Fused optimizer kernels
• Efficient gradient checkpointing
• Optimized LoRA implementations

✅ 2x faster training compared to standard HuggingFace
✅ 20% memory reduction for the same batch size
✅ Zero code changes - same API as HuggingFace
✅ Supports popular architectures: Llama, Mistral, Qwen, Gemma, Phi
✅ Compatible with all strategies: SFT, QLoRA, RLHF, DPO

⚠️ Important: Unsloth is NOT supported on Apple Silicon Macs. The optimized CUDA kernels are not compatible with Apple's MPS backend. If you're on macOS with Apple Silicon, use the HuggingFace provider instead.

pip install unsloth

See Windows Installation Guide.

FROM nvidia/cuda:12.6.0-devel-ubuntu22.04
RUN pip install unsloth

python -c "import unsloth; print('Unsloth version:', unsloth.__version__)"

{
  "provider": "unsloth",
  "model_name": "meta-llama/Llama-3.2-3B",
  "max_seq_length": 2048,
  "task": "text-generation",
  "strategy": "sft",
  "num_train_epochs": 3,
  "lora_r": 16,
  "lora_alpha": 32
}

⚠️ CRITICAL: When using Unsloth, you MUST specify a fixed max_seq_length. Auto-inference (-1) is NOT supported.

Valid:

{
  "provider": "unsloth",
  "max_seq_length": 2048  // ✅ Fixed value
}

Invalid:

{
  "provider": "unsloth",
  "max_seq_length": -1  // ❌ NOT supported
}

Common values:

• 512 - Short sequences, lower memory
• 1024 - Medium sequences
• 2048 - Standard (recommended)
• 4096 - Long contexts, more memory
• 8192 - Very long contexts, high memory

1. Go to Training tab
2. Select Provider: unsloth
3. Set Max Sequence Length: 2048 (or your preferred value)
4. Configure other settings
5. Start training

curl -X POST http://localhost:8000/api/start_training \
  -H "Content-Type: application/json" \
  -d '{
    "provider": "unsloth",
    "model_name": "meta-llama/Llama-3.2-3B",
    "max_seq_length": 2048,
    "task": "text-generation",
    "strategy": "sft",
    "dataset": "/path/to/dataset.jsonl",
    "num_train_epochs": 3
  }'

• Llama (1, 2, 3, 3.1, 3.2)

  • meta-llama/Llama-3.2-1B
  • meta-llama/Llama-3.2-3B
  • meta-llama/Llama-3.1-8B

• Mistral

  • mistralai/Mistral-7B-v0.1
  • mistralai/Mistral-7B-Instruct-v0.3

• Qwen

  • Qwen/Qwen2-1.5B
  • Qwen/Qwen2-7B

• Gemma

  • google/gemma-2b
  • google/gemma-7b

• Phi

  • microsoft/phi-2
  • microsoft/phi-3-mini

• BART - Some optimizations not available
• T5 - Not recommended with Unsloth

Setup: Llama-3.2-3B, 1000 examples, NVIDIA RTX 3090

Setup: Llama-3.2-7B, batch_size=4, seq_length=2048

Setup: Llama-3.2-3B, batch_size=8

{
  "provider": "unsloth",
  "model_name": "meta-llama/Llama-3.2-3B",
  "max_seq_length": 2048,
  "strategy": "qlora",
  "use_4bit": true,
  "bf16": true,
  "gradient_checkpointing": true,
  "per_device_train_batch_size": 8,
  "gradient_accumulation_steps": 4,
  "lora_r": 64,
  "lora_alpha": 16,
  "lora_dropout": 0.1
}

For limited VRAM:

{
  "provider": "unsloth",
  "max_seq_length": 1024,  // Reduce sequence length
  "per_device_train_batch_size": 2,
  "gradient_accumulation_steps": 8,
  "gradient_checkpointing": true,
  "use_4bit": true
}

For maximum speed:

{
  "provider": "unsloth",
  "max_seq_length": 2048,
  "per_device_train_batch_size": 16,
  "gradient_accumulation_steps": 1,
  "bf16": true,
  "optim": "adamw_8bit"
}

Unsloth auto-detects optimal LoRA target modules, but you can override:

{
  "provider": "unsloth",
  "target_modules": [
    "q_proj",
    "k_proj",
    "v_proj",
    "o_proj",
    "gate_proj",
    "up_proj",
    "down_proj"
  ]
}

Unsloth uses optimized gradient checkpointing:

{
  "provider": "unsloth",
  "gradient_checkpointing": true  // Automatically optimized
}

Problem: Provider error when selecting Unsloth

Solution: Install Unsloth:

pip install unsloth

Problem: Running on native Windows

Solution: Use WSL or Docker. See Windows Installation.

Problem: Auto-inference not supported

Solution: Set a fixed value:

{
  "max_seq_length": 2048
}

Problem: OOM errors during training

Solutions:

1. Reduce max_seq_length: 2048 → 1024
2. Reduce per_device_train_batch_size: 8 → 4
3. Enable gradient_checkpointing: true
4. Use 4-bit quantization: use_4bit: true

Problem: Specific model doesn't work with Unsloth

Solution: Fall back to HuggingFace provider:

{
  "provider": "huggingface"
}

Problem: Flash Attention 2 compatibility issues

Solution: Disable Flash Attention:

export UNSLOTH_DISABLE_FLASH_ATTN=1
modelforge

• Training on Linux or WSL with NVIDIA GPU
• Using supported models (Llama, Mistral, etc.)
• Need faster training times
• Have limited VRAM
• Training large models (7B+)

• Running on native Windows (use HuggingFace)
• Running on macOS with Apple Silicon (use HuggingFace)
• Using unsupported models (BART, T5)
• Debugging issues (HuggingFace has better error messages)
• Need maximum compatibility

Switching is simple - just change the provider:

Before:

{
  "provider": "huggingface",
  "model_name": "meta-llama/Llama-3.2-3B",
  ...
}

After:

{
  "provider": "unsloth",
  "model_name": "meta-llama/Llama-3.2-3B",
  "max_seq_length": 2048,  // Add this!
  ...
}

All other settings remain the same!

• Provider Overview - Compare all providers
• HuggingFace Provider - Standard provider docs
• Configuration Guide - All config options
• Performance Optimization - Get the best results

Unsloth: Train faster, use less memory! 🚀