Judging by the description of this PR, I believe many models with multiple-token prediction also have the same strategy of reusing hidden features from the main model.

It can be quite interesting to generalize this features to support other models. I would expect some kind of sub-llama_context that allow both the main and draft models to share the same cgraph, avoiding the need of explicitly passing the intermediate embedding through the host memory.

It can be quite interesting to generalize this features to support other models.

I will definitely be looking at refactoring the implementation to become more generic before merging it. The initial results in terms of performance are really great, but we'll need to work on cleaning up the code and reduce the special-casing in several places. I'll try to provide insights how to do that in the next days.

It can be quite interesting to generalize this features to support other models.

I will definitely be looking at refactoring the implementation to become more generic before merging it. The initial results in terms of performance are really great, but we'll need to work on cleaning up the code and reduce the special-casing in several places. I'll try to provide insights how to do that in the next days.

Thanks @ggerganov @ngxson for your inputs. Definitely, looking forward to hearing your feedback and improving this PR.

Bumping, is there any progress on this? It's probably one of the more coveted features to have right now.

Bumping, is there any progress on this?

I'm currently side-tracked by some graph reallocation optimizations. Will probably come back to this after that.

Eagle3 checkpoints for the Qwen3 series (including both dense and MoE models) are now supported, see the updated PR description for details.
Although these Eagle3 checkpoints are from third party, they can still deliver a 1–2× speedup.
Speculative decoding performance for MoE models is not as good as dense models, which is expected, since more experts are invoked during the parallel verification phase than during the target model’s decoding phase.

One question: it seems that CUDA Graph is disabled when the input n_tokens > 1. During the target model verification stage of speculative decoding, CUDA Graph is always disabled for the target model, since it’s only used for verification with multiple draft tokens > 1. However, we can fix the number of draft tokens (e.g., by using padding) to make it constant and thus enable CUDA Graph (may need to remove n_tokens > 1 constraint)? @ggerganov

Context: I’m testing GPT-OSS-120B Eagle3 with llama.cpp, and I found that even with Eagle3 (accept rate 86%), the performance is worse than the naive llama-cli. After profiling, I discovered that CUDA Graph is consistently disabled for the target model during speculative decoding, whereas it remains enabled in llama-cli. This results in the target model’s verification(prefiling) phase being roughly >5× times slower compared to normal autoregressive decoding step.
After disabling CUDA graphs for llama-cli using GGML_CUDA_DISABLE_GRAPHS=1, the eagle3 achieved roughly a 1.5× speedup.

I’ve only observed this performance issue with GPT-OSS-120B Eagle3. For other models, even without CUDA Graph enabled for target model in Eagle3 speculative decoding, the performance remains great.

Speculative decoding performance for MoE models is not as good as dense models, which is expected, since more experts are invoked during the parallel verification phase than during the target model’s decoding phase.

I think the small-batch mul_mat_id could be improved in the CUDA backend. AFAIR there the performance for batch sizes (1, 8] is not optimal atm. Need double check.

However, we can fix the number of draft tokens (e.g., by using padding) to make it constant and thus enable CUDA Graph (may need to remove n_tokens > 1 constraint)? @ggerganov

Possibly, but to me this sounds like second-order optimization. Optimizing the mul_mat_id for small batches should bring more generic benefits and would likely have larger impact for speculative decoding compared to enabling CUDA graphs.

After disabling CUDA graphs for llama-cli using GGML_CUDA_DISABLE_GRAPHS=1, the eagle3 achieved roughly a 1.5× speedup.

Hm, this is a bit surprising observation. Can you run a llama-batched-bench test on your system with and without CUDA graphs using the commands from #18308 (comment) and share the results. We are interested in batch sizes [1, 4]. So something like this:

llama-batched-bench -m [gpt-oss-120b] -c 65536 -b 2048 -ub 512 -npp 1024 -ntg 32 -npl 1,2,3,4,5,6,7,8

Thanks very much for your inputs! @ggerganov

After disabling CUDA graphs for llama-cli using GGML_CUDA_DISABLE_GRAPHS=1, the eagle3 achieved roughly a 1.5× speedup.

Hm, this is a bit surprising observation. Can you run a llama-batched-bench test on your system with and without CUDA graphs using the commands from #18308 (comment) and share the results. We are interested in batch sizes [1, 4]. So something like this:

I double-checked the run today. The previous statement about cuda graph was incorrect due to instability and concurrent CPU activity in my test environment, sorry about that! Currently, enabling or disabling CUDA Graphs doesn’t have much impact in llama-cli for GPT-OSS-120B model. (I am testing on DGX Spark)

• with cuda graph enabled: [ Prompt: 120.1 t/s | Generation: 47.2 t/s ]
• without cuda graph enabled: [ Prompt: 119.2 t/s | Generation: 45.7 t/s ]

Also, the results for llama-batched-bench:

• with cuda graph enabled

• without cuda graph enabled

Possibly, but to me this sounds like second-order optimization. Optimizing the mul_mat_id for small batches should bring more generic benefits and would likely have larger impact for speculative decoding compared to enabling CUDA graphs.

I agree. CUDA graphs could be second-order optimization.
Here are the eagle3 GPT-OSS-120B test results on DGX spark: (I will also test this on other hardwares)

For MoE models, prefilling becomes the main performance bottleneck because more active experts are involved. As a result, the assumption that “processing multiple draft tokens concurrently is as fast as processing a single token” no longer holds, which is an important condition for effective speculative decoding. I also saw that as the draft token length increases, the verification cost of the target model also rises.
This explains the results shown in the table above, in some cases, Eagle3 can even degrade performance. To observe improvements, the accept rate must exceed a certain lower bound.

Do you have any rough ideas that how much performance gain we can get through imporving mul_mat_id?

The llama-batched-bench results are actually better than I expected. In the previous reported numbers there was a sharp dip at BS = 2. Here the TG performance steadily increases with the batch size which is good, though it is not as linear as we want it to be.

I suppose the explanation is that for MoE models, at low batch sizes the amount of data we need to read from the weights for each batch increases linearly with the batch size (i.e. each extra token in the batch activates more experts and at small batch size the experts for each token are very likely different from each other). So it's probably normal that TG for MoE does not scale as well as TG for dense models as a function of the batch size.

As a result, the assumption that “processing multiple draft tokens concurrently is as fast as processing a single token” no longer holds, which is an important condition for effective speculative decoding.

Yeah, that's my guess as well. Do we have some references to cross-check this? Does the Eagle3 authors discuss it's performance for MoE models? Do we have sample numbers for gpt-oss-120 with Eagle3 using vllm, trrt?

Do you have any rough ideas that how much performance gain we can get through imporving mul_mat_id?

Hm, not sure. Thinking about it now, I feel like mul_mat_id is unlikely to scale good enough due to the increasing data for each new token.

The following eagle3 models should also work out of the box, though they haven’t been tested yet:
Qwen3-235B-A22B-EAGLE3

I tested the Baichuan-M3-235B model that was released yesterday (draft here). It's a finetune of the Qwen3 model above. It quantized successfully but failed due to having a different tensor shape (even in the original weights):

load_tensors: EAGLE3 using d2t mapping (draft_vocab_size = 32000)
llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected  8192,  8192, got  8192, 16384,     1,     1
llama_model_load_from_file_impl: failed to load model
failed to load EAGLE3 draft model

I haven't looked into how often this to happen in finetunes of the same model, especially in the context of eagle3.

However, the shapes of the tensors changing might be something to account for in the implementation (in this case Qwen3). Unless those will be treated as completely new models, in which case please disregard this comment.

The following eagle3 models should also work out of the box, though they haven’t been tested yet:
Qwen3-235B-A22B-EAGLE3

I tested the Baichuan-M3-235B model that was released yesterday (draft here). It's a finetune of the Qwen3 model above. It quantized successfully but failed due to having a different tensor shape (even in the original weights):

load_tensors: EAGLE3 using d2t mapping (draft_vocab_size = 32000)
llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.attn_q.weight' has wrong shape; expected  8192,  8192, got  8192, 16384,     1,     1
llama_model_load_from_file_impl: failed to load model
failed to load EAGLE3 draft model

I haven't looked into how often this to happen in finetunes of the same model, especially in the context of eagle3.

However, the shapes of the tensors changing might be something to account for in the implementation (in this case Qwen3). Unless those will be treated as completely new models, in which case please disregard this comment.

I spent some time analyzing the Baichuan-EAGLE3 draft model. It has a slightly different architecture compared to the standard Qwen3-EAGLE3 model.
The main difference is in the self_attn.q_proj.weight tensor shape:

• Standard Qwen3-EAGLE3 : [8192, 8192] — outputs Q only
• Baichuan-EAGLE3: [16384, 8192] — outputs Q + Gate (2× the size)

This is because Baichuan-EAGLE3 uses an Attention Output Gate mechanism, which is not present in the standard EAGLE3 model. In this variant:

• The Q projection outputs both query vectors and gate vectors
• After attention computation, the output is element-wise multiplied by sigmoid(gate) before the output projection

This is essentially a variant architecture of EAGLE3, not just a tensor shape difference. Supporting this variant would require:

• Detecting the gate mechanism during model loading
• Modifying the graph construction to split Q/Gate and apply the gating after attention
• Adding the ggml_sigmoid operation in the attention path

I would suggest we focus this PR on the standard EAGLE3 model first. Once merged, we can consider adding support for this gated variant in a follow-up PR.

Have you tested the standard Qwen3-EAGLE3 model as well? Does it work well with the current implementation? If yes, could you please share the t/s and speedup you got with eagle3? @arch-btw

Since EAGLE3 can vary quite a lot for each model, maybe a better way is to consider it as an adapter (the same logic as lora adapter), instead of a dedicated arch?

That way, it can hook into existing models more easily, making internal data like KV state, gate, etc, accessible to the draft model.

Since EAGLE3 can vary quite a lot for each model, maybe a better way is to consider it as an adapter (the same logic as lora adapter), instead of a dedicated arch?

That way, it can hook into existing models more easily, making internal data like KV state, gate, etc, accessible to the draft model.

Good point. However, Eagle3 doesn’t vary much across models. So far, except for Baichuan-Eagle3, all other models essentially use the same Eagle3 architecture. Please refer to the supported models listed in the PR description. I’d say the majority of models share the same Eagle3 architecture, with only a few exceptions. This standalone Eagle3 architecture strategy is also adopted in TensorRT-LLM, vLLM, and SGLang.

I doubt that. In theory, nothing prevent them or another team from making a variant of eagle3 that get the state of more than 3 layers, or even reuse the KV state from earlier layers. Possibilities are endless, and that's why it's important to think about the bigger picture instead of just trying to make it work with one single existing architecture.

I think a more model-agnostic approach via adapter API (or another API based on that form) will likely be the way ultimately. It will allow computing both the next token + draft token in one pass, allowing even higher performance than this approach.

I doubt that. In theory, nothing prevent them or another team from making a variant of eagle3 that get the state of more than 3 layers, or even reuse the KV state from earlier layers. Possibilities are endless, and that's why it's important to think about the bigger picture instead of just trying to make it work with one single existing architecture.

Could you please share some examples or real-world use cases of this? I’d like to better understand how such an approach might be applied in practice.

The main problem with this PR and #15225 is that both assumes that the MTP (multi-token prediction) to work this way:

• main LLM generates first tokens + hidden_state from a list of selected layers
• hidden_state is then forwarded to the speculative model to generate N next tokens

(Note: the dash line is to tell that it's may not be the case for all models; some only use the last hidden state)

While it does work for the moment, this approach doesn't address the true nature of MTP models. In other words, it is not truly model-agnostic. The main drawbacks is that you must manually pass the embeddings between 2 models, so you must know where to get the embeddings, its shapes, etc.

Instead, we should look at MTP models as a normal LLM with multiple output heads:

In this POV, it's not matter what is the implementation of the mtp_head. From the outside world, the model will just output N next tokens given one input token.

In practice, the mtp_head(s) can be:

• A single encoder model in eagle3 case
• Some extra N layers in GLM/deepseek case, N+1 layer use output from N-th layer (serial setup)
• Some extra N layers in Xiaomi Mimo case, but all layers use the same input embeddings (parallel setup) - see llama : add Xiaomi Mimo (with proper MTP - multi token predict) #13236 for more info

Now, returning to your question:

Could you please share some examples or real-world use cases of this? I’d like to better understand how such an approach might be applied in practice.

If you already get the idea above, then consider gemma3n: the model has 30 layers, but only 20 layers has KV projection. The last 10 layers reuse the KV from the 20-th layer. Some models also implement this idea, notably GLM, bailing.

The same idea can be apply to MTP layers. Future models may has MTP layers to not just reuse the layer output hidden state, but also the projected KV inside the layer. While there is no models in the wild currently doing that, Baichuan-EAGLE3 (as you shown), already someway heading towards this direction by exposing both the Q+gate to the MTP model.

(I have to split up my comment otherwise it's too long)

My proposal is that we must design this function + the API in a way that it is flexible enough for future models.

For EAGLE3, the MTP model is technically a mtp_head shipped as an extension to the main model (note that the eagle3 repo only contains the extra tensors, but does not contain the main LLM), it can be viewed as an adapter, much like how LoRA works.

For the API, we must avoid leaking the information about the implementation under the hood. The downstream code must only know about how many tokens can be generated, they don't need to know how to generate these extra tokens.

So, an array of API as follow should be enough:

• llama_model_load_mtp: load the mtp as a llama_adapter_lora or maybe we can add a new struct for it
• llama_mtp_set_n_draft: set the max number of draft tokens to be generated in the next llama_decode; set to 0 for verification pass
• llama_mtp_get_n_draft_max: get max number of draft tokens that the MTP head can generate
• llama_mtp_get_logits_ith: get logits at for i-th token in batch, returns array of float with size n_vocab*n_draft

All the info about embeddings and the draft model must be kept private.

CC @ggerganov maybe this is helpful for you

Should be good to merge after review of the Python code.

I notice some of the more recent uploads on huggingface (like https://huggingface.co/nvidia/Kimi-K2.6-Eagle3) don't use any of the listed archs, instead using Eagle3DeepseekV2ForCausalLM

Is there any way we can understand Eagle3 at the start of the arch as an eagle3 draft checkpoint and route it the same way?

I notice some of the more recent uploads on huggingface (like https://huggingface.co/nvidia/Kimi-K2.6-Eagle3) don't use any of the listed archs, instead using Eagle3DeepseekV2ForCausalLM

Is there any way we can understand Eagle3 at the start of the arch as an eagle3 draft checkpoint and route it the same way?

Oh, thanks for sharing this. I wasn’t aware that we have a new Kimi 2.6 Eagle3 model checkpoint. (This is great, eagle3 ecosystem is still growing, so it is great we have it in llama.cpp now).
The config looks different, and I haven’t had a chance to look into it yet. I’m not sure how they implemented Eagle3 here, maybe they used a decoder layer from DeepSeekV2?
All the Eagle3 checkpoints I’ve seen so far are based on Llama’s decoder layer, which matches the original paper. Maybe this one is different.

I also found another Kimi 2.6 Eagle3 config that seems aligned with our current approach: https://huggingface.co/lightseekorg/kimi-k2.6-eagle3/blob/main/config.json

Okay, would probably prefer the property to live in llama.py since Eagle is the only arch that uses it, but I'll let @CISC decide that one :)

That will change though, so makes sense that it's in base.

Okay, would probably prefer the property to live in llama.py since Eagle is the only arch that uses it, but I'll let @CISC decide that one :)

Thanks for approval!
Makes sense. I kept it in ModelBase because target_model_dir is passed through the common converter constructor, even though Eagle3 is the only current consumer.
DFlash would be another consumer in the future :) #22105

Thanks @bartowski1182 @pwilkin @CISC for the review! I guess this PR is good to merge :) CC @ggerganov

Let's see if we win the \r\n lottery today...

Edit: Yay, green: https://github.com/ggml-org/llama.cpp/actions/runs/27377588058/job/80906815068?pr=18039

i cannot find qwen3.6 eagle3 if it exist can someone please link it.

i cannot find qwen3.6 eagle3 if it exist can someone please link it.

Btw, direct GGUF files would be greatly appreciated 😅

Thank you for introducing Eagle3 !

Here are benchmarks from ruixiang63 -> #18039 (comment)

I have did additional benchmarks And here are results FreeBSD + 7900 xtx :

* [RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3)
* [gemma-4-26B-A4B-it-GGUF Q4_K_M)[https://huggingface.co/ggml-org/gemma-4-26B-A4B-it-GGUF]
* [Assistant IT Q8_0] (https://huggingface.co/google/gemma-4-26B-A4B-it-assistant)

reasoning off

According to the above, the Assistant MTP issue was resolved in PR #24277. However, I’m still seeing a similar performance regression on Eagle 3 — while working with an agentic tool like claude-code, I’m observing a significant drop in draft acceptance, ranging from 9% to 33% together with performance drop.

Advice is appreciated

It crashes on load with -sm tensor for me using gemma 4 31b and redhat's eagle3 model converted to bf16.

0.57.900.481 I srv    load_model: initializing slots, n_slots = 4
0.58.099.454 I common_speculative_impl_draft_eagle3: adding speculative implementation 'draft-eagle3'
0.58.099.462 I common_speculative_impl_draft_eagle3: - n_max=8, n_min=0, p_min=0.500000
/home/local/downloads/llama.cpp/ggml/src/ggml-backend-meta.cpp:728: GGML_ASSERT(src_ss[0].axis != GGML_BACKEND_SPLIT_AXIS_1) failed
/home/local/downloads/llama.cpp/build/bin/libggml-base.so.0(+0x18665) [0x7f8670d5f665]
/home/local/downloads/llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x1df) [0x7f8670d5fa3f]
/home/local/downloads/llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x11e) [0x7f8670d5fbce]
/home/local/downloads/llama.cpp/build/bin/libggml-base.so.0(+0x41f2f) [0x7f8670d88f2f]
/home/local/downloads/llama.cpp/build/bin/libggml-base.so.0(+0x39a75) [0x7f8670d80a75]
/home/local/downloads/llama.cpp/build/bin/libggml-base.so.0(+0x42cac) [0x7f8670d89cac]
/home/local/downloads/llama.cpp/build/bin/libggml-base.so.0(+0x45420) [0x7f8670d8c420]
/home/local/downloads/llama.cpp/build/bin/libggml-base.so.0(ggml_gallocr_alloc_graph+0x483) [0x7f8670d74c23]
/home/local/downloads/llama.cpp/build/bin/libggml-base.so.0(ggml_backend_sched_alloc_graph+0x111) [0x7f8670d7af11]
/home/local/downloads/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0xbd) [0x7f86704d291d]
/home/local/downloads/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x368) [0x7f86704d83d8]
/home/local/downloads/llama.cpp/build/bin/libllama.so.0(llama_decode+0xb) [0x7f86704d9fbb]
/home/local/downloads/llama.cpp/build/bin/libllama-common.so.0(_Z25common_context_can_seq_rmP13llama_context+0xc8) [0x7f86709e3b08]
/home/local/downloads/llama.cpp/build/bin/libllama-server-impl.so(_ZN19server_context_impl10load_modelER13common_params+0xcd4) [0x7f8671572314]
/home/local/downloads/llama.cpp/build/bin/libllama-server-impl.so(_Z12llama_serveriPPc+0x2ece) [0x7f86714c57de]
/lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7f8670e35ca8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f8670e35d65]
./.llama-cpp/llama-server(+0x11b1) [0x55c5ab6f21b1]
Aborted