EAGLE-2 Speculative Decoding Support for llama.cpp - Phase 1 Submission

Overview

This PR introduces EAGLE-2 (Extrapolation Algorithm for Greater Language-model Efficiency) speculative decoding support for llama.cpp. EAGLE-2 is an advanced speculative decoding technique that uses a smaller draft model to accelerate inference of larger target models.

Paper: EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees
Official Repository: https://github.com/SafeAILab/EAGLE

Implementation Status

✅ EAGLE-2 is fully implemented and functional - complete with model conversion, loading, inference, and speculative decoding algorithm.

For maintainability and review efficiency, we’re submitting the implementation in multiple focused phases rather than as one large PR.

Phase 1: Model Conversion Infrastructure

This submission focuses on the model conversion component - enabling EAGLE-2 draft models to be converted from safetensors to GGUF format.

Why Phased Submission?

We've chosen a phased submission approach for the following reasons:

1. Review Efficiency: Smaller, focused PRs are easier for maintainers to review thoroughly
2. Testing & Validation: Allows community testing of each component before building on it
3. Incremental Integration: Reduces risk of conflicts and integration issues
4. Code Quality: Enables focused feedback on specific functionality

Submission Roadmap

• Phase 1 (Current): Model conversion and GGUF format support
• Phase 2 (Next): Draft model loading and computation graph implementation
• Phase 3 (Final): Core speculative decoding algorithm and inference logic

All phases are already implemented and tested - this roadmap represents submission phases only.

Current Submission Features

Enhanced Model Conversion

• ✅ Support for EAGLE-2 draft model architecture in convert_hf_to_gguf.py
• ✅ Proper handling of draft model specific layers and weights
• ✅ Seamless integration with existing GGUF conversion pipeline

Model Size Optimization

EAGLE-2 draft models offer significant size advantages:

Draft models achieve extreme compression as they extract and optimize single decoder layers from the original model.

Demonstrated Performance

The complete EAGLE-2 implementation has been tested on NVIDIA RTX 4080 with few data come from ShareGPT dataset:

Qwen2-7B-Instruct + EAGLE-Qwen2-1.4B Draft Model

Performance Notes:

• Mixed sampling includes temperature + top_k + top_p
• Results demonstrate up to 2.1x acceleration
• Performance matches expectations from original Python implementation

Technical Implementation

Architecture Support

• Extended convert_hf_to_gguf.py with EAGLE-2 model detection
• Proper weight mapping for draft model specific layers
• Full compatibility with existing llama.cpp infrastructure

Memory Efficiency

• Draft models use single decoder layers, dramatically reducing memory footprint

Testing Resources

Pre-converted Models

For immediate testing and validation:

EAGLE-Qwen2 Draft Model (F16):

• 🤗 Safetensor Models & HuggingFace Model
• Size: 2.7GB
• Compatible with: Qwen2-7B-Instruct target models

Testing the Conversion

# Convert EAGLE-2 draft model to GGUF
python convert_hf_to_gguf.py /path/to/eagle-draft-model --outfile eagle-draft.gguf

Current Implementation Characteristics

Operational Parameters

1. Platform Performance: Optimized for GPU inference; CPU inference optimization pending.
2. Sequence Processing: Single-sequence inference (consistent with original EAGLE-2 research design)
3. Model Support: Currently validated with Qwen2; architecture supports LLaMA 2/3, Vicuna, etc.

Architecture Design

• EAGLE-2 algorithm is designed for GPU acceleration scenarios
• Implementation follows original research specifications
• Full compatibility with llama.cpp ecosystem

Future Submissions

The remaining implementation components will be submitted in subsequent PRs:

• Phase 2: Draft model loading, computation graph, and memory management
• Phase 3: Complete speculative decoding algorithm and inference engine

Acknowledgments

• Original EAGLE-2 research team at SafeAI Lab
• llama.cpp community for the robust infrastructure
• Early adopters and testers providing valuable feedback

This Phase 1 submission provides the model conversion foundation for EAGLE-2 speculative decoding in llama.cpp. The complete implementation demonstrates significant inference acceleration while maintaining compatibility with the llama.cpp ecosystem.