Replies: 2 comments 7 replies
-
|
there are a bunch of MTP PR's but they don't seem to be making much progress and seem to be EAGLE 3 focused. I'm also watching this diffusion draft approach closely. |
Beta Was this translation helpful? Give feedback.
-
|
yeah thats why i didnt want to crowd the issues tab with another draft feature request although this one seems pretty neat (; |
Beta Was this translation helpful? Give feedback.
-
|
For anyone interested in DFlash, please take a look at this PR: #22105 Now it comes to llama.cpp :) |
Beta Was this translation helpful? Give feedback.
-
|
I’ve added dflash support for Qwen3.5/3.6 MoE. However, please note that the performance is currently not optimal. This is mainly due to the MoE architecture combined with the hybrid structure used in Qwen3.5/3.6 MoE, which is not yet well supported by llama.cpp. For more details on the performance limitations and underlying issues, please refer to my shared PR: #22105 |
Beta Was this translation helpful? Give feedback.
-
|
@ruixiang63 Thanks for the update! Performance issue: noted on the MoE + hybrid architecture limitations with llama.cpp. Will keep an eye on #22105 for progress. Suggestion: saw that the ggml team has been working on refactoring the codebase (e.g., CMake glob changes and bias tensor renaming). It might be easier for the ggml team if the PR is rebased on the latest upstream master to avoid conflicts with those structural changes. |
Beta Was this translation helpful? Give feedback.
-
|
Don’t see it mentioned here yet, there's also DDTree for accelerated speculative decoding on top of DFlash. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, thanks. My thought is to get the Eagle3 PR #18039 merged first, since the API refactoring also needs to happen there. My DFlash PR is based on Eagle3, so once that’s merged, I’ll rebase the DFlash PR on top of it. |
Beta Was this translation helpful? Give feedback.
-
Does it also work for dense models? like qwen3.6 27B |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Just saw this new repo on r/locallama and its seems promising, they (z-lab) claim to have a decent speedup even over eagle3, which is still a draft pr here. The idea is basically to not do a autoregressive draft model but a diffusion one to speed things up and ofc the stats vary but for Qwen3.5-35B-A3B for example they claim a 2-2.8x speedup over normal token generation with an acceptance length mostly beating even mtp other models are at times even better like qwen3.5 9b with up to 3.5x, as can be seen in their repo and huggingface. Im not sure if the devs here think its worth it rn with eagle3 having a draft pr already so ive created this as a discussion to just bring this up (;
The repo is available here
and the already available draft models can be found here
Beta Was this translation helpful? Give feedback.
All reactions