[MODEL] support qwen3.5 series #19468
Conversation
|
@JJJYmmm no worries, glad if it helped even a little :) |
ngxson
left a comment
There was a problem hiding this comment.
I haven't reviewed the actual implementation, but I assume it's mostly qwen3 next + imrope (please correct if I'm wrong)
Don't hesitate to ping if you need helps to add the vision support!
It definitely helped a lot 🙏 |
ggerganov
left a comment
There was a problem hiding this comment.
The libllama changes are good. I will follow up with a PR to deduplicate and optimize the delta net stuff across the Qwen family.
|
|
||
| // if head keys and value keys are different, repeat to force tensors into matching shapes | ||
| if (num_k_heads != num_v_heads) { | ||
| GGML_ASSERT(num_v_heads % num_k_heads == 0); | ||
| int64_t repeat_factor = num_v_heads / num_k_heads; | ||
|
|
||
| // repeat interleave: reshape to (repeat part, 1, remaining part), do repeat, then reshape back | ||
| ggml_tensor * q_reshaped = ggml_reshape_3d(ctx0, q_conv, head_k_dim, 1, num_k_heads * n_seq_tokens * n_seqs); | ||
| ggml_tensor * k_reshaped = ggml_reshape_3d(ctx0, k_conv, head_k_dim, 1, num_k_heads * n_seq_tokens * n_seqs); | ||
|
|
||
| // Repeat along the third dimension (the new dimension with size 1) | ||
| ggml_tensor * q_repeated = | ||
| ggml_repeat_4d(ctx0, q_reshaped, head_k_dim, repeat_factor, num_k_heads * n_seq_tokens * n_seqs, 1); | ||
| ggml_tensor * k_repeated = | ||
| ggml_repeat_4d(ctx0, k_reshaped, head_k_dim, repeat_factor, num_k_heads * n_seq_tokens * n_seqs, 1); | ||
|
|
||
| // Reshape back to merge the head and repeat dimensions | ||
| // From [head_dim, num_k_heads, repeat_factor, n_seq_tokens * n_seqs] | ||
| // Back to [head_dim, num_k_heads * repeat_factor, n_seq_tokens, n_seqs] | ||
| q_conv = ggml_reshape_4d(ctx0, q_repeated, head_k_dim, num_k_heads * repeat_factor, n_seq_tokens, n_seqs); | ||
| k_conv = ggml_reshape_4d(ctx0, k_repeated, head_k_dim, num_k_heads * repeat_factor, n_seq_tokens, n_seqs); | ||
| } | ||
|
|
There was a problem hiding this comment.
Btw, this is an issue inherited from the Qwen3 Next implementation. Basically we are doing redundant repeats here because the V weights are not arranged correctly during conversion. Ideally this whole section should not be needed and we should only rely on broadcasts.
Basically, we want to broadcast Q and K into V. With the current order of the V heads, we need an interleaved broadcast. However the ggml binary ops perform tiled broadcast. Hence the need for this rearrangement.
There was a problem hiding this comment.
If it is too difficult to figure out how to fix this for the new Qwen3.5 models, we'll have to do it in a follow-up PR. But it's going to be a breaking change. Or maybe we can try to gate it with some flag.
There was a problem hiding this comment.
I would try to fix it in the qwen3.5 gguf conversion since it is not released.
There was a problem hiding this comment.
Since it's a new arch we should not need a flag as such then.
There was a problem hiding this comment.
just reorder the related v weights to avoid repeat_interleave🫡 4248c93
There was a problem hiding this comment.
Nice. Just make sure results are still as expected after this change.
From here, I think we will be able to remove the explicit repeats - this is something for next PR. Thanks a lot!
There was a problem hiding this comment.
Yes, I tested it with the new generated weights, and it works just as before. 🫡
ngxson
left a comment
There was a problem hiding this comment.
The changes outside of cgraph looks good to me (sorry don't have too much time to look deeper, but I believe the review from @ggerganov and @CISC should already be enough)
Looking forward to the vision support PR then!
@ngxson This pr includes the vision part and I test it too! (qwen3.5 uses the same vit as qwen3vl and deepstack was removed temporally)🤗 |
|
@JJJYmmm ah yeah right, thanks for the clarification. Yes I already acknowledge that the deep stack is removed (confirmed by Look good to me then, thanks again! |
|
It seems to me that all review comments are resolved, and @JJJYmmm also confirmed that the model is still working fine with last commit. So, I'll let @ggerganov to decide when to merge this. |
* support qwen3.5 series * remove deepstack for now, and some code clean * code clean * add FULL_ATTENTION_INTERVAL metadata * code clean * reorder v heads for linear attention to avoid expensive interleaved repeat
* support qwen3.5 series * remove deepstack for now, and some code clean * code clean * add FULL_ATTENTION_INTERVAL metadata * code clean * reorder v heads for linear attention to avoid expensive interleaved repeat
This PR adds model support for the upcoming Qwen3.5 models, including both dense and MoE variants. It has been verified with preview checkpoints from the Qwen Team in both vision and pure text modes (with or without mmproj file).
As mentioned here, this PR was planned to be a follow-up PR after #19435 or #19456 by @pwilkin. However, it seems there is still a lot to do without the real checkpoints (such as vision support, vocab, weight mapping, and forward logic), so I'm sharing my implementation here for reference. 🫡
Some differences compared to #19435 or #19456:
@pwilkin Sorry for the work overlap, and I really appreciate your work on Qwen3Next! It has been a huge stepping stone for this PR.
Reference HF implementation: huggingface/transformers#43830