Qwen3.5 support series

This is part of a 5-PR series adding Qwen3.5-VL support, split for review clarity.

Dev PRs (this series):

• [1/5] MTP packed-seq CP+THD fix — [Dev] fix(mtp): use padded cu_seqlens in MTP roll for THD with CP #4494
• [2/5] FSDP DTensor Bridge checkpoint compatibility — [dev] [2/5] Qwen3.5 support: FSDP DTensor Bridge checkpoint compatibility #4748
• [3/5] SharedExpertMLP meta init — [dev] [3/5] Qwen3.5 support: SharedExpertMLP meta init #4749
• [4/5] Interleaved MRoPE layout — [dev] [4/5] Qwen3.5 support: Interleaved MRoPE layout #4750 ← this PR
• [5/5] Qwen3.5-VL training example — [dev] [5/5] Qwen3.5 support: Qwen3.5-VL training example #4751

Main PRs (corresponding mirrors):

• [1/5] fix(mtp): use padded cu_seqlens in MTP roll for THD with CP #4495
• [2/5] [main] [2/5] Qwen3.5 support: FSDP DTensor Bridge checkpoint compatibility #4753
• [3/5] [main] [3/5] Qwen3.5 support: SharedExpertMLP meta init #4754
• [4/5] [main] [4/5] Qwen3.5 support: Interleaved MRoPE layout #4755
• [5/5] [main] [5/5] Qwen3.5 support: Qwen3.5-VL training example #4756

Summary

Add the interleaved MRoPE layout used by Qwen3.5-VL, gated by a new config flag.

• New TransformerConfig.mrope_interleaved: bool = False.
• New MultimodalRotaryEmbedding(interleaved_mrope=...) argument.
• New helper _apply_interleaved_mrope(freqs, mrope_section) that converts the per-channel outer-product layout (3, bs, seq_len, dim) into the interleaved single-tensor layout (bs, seq_len, dim).
• GPTModel passes config.mrope_interleaved through to the embedding.

Why

MultimodalRotaryEmbedding currently produces the section-based T/H/W cycling used by Qwen2-VL. Qwen3.5-VL (and the matching HuggingFace Qwen3VLTextRotaryEmbedding.apply_interleaved_mrope unified on 2026-02-24) use a different layout where:

• T freqs occupy stride-3 positions {0, 3, 6, ...}
• H freqs occupy {1, 4, 7, ...}
• W freqs occupy {2, 5, 8, ...}

Both layouts are now supported; the flag picks between them.

Risk

mrope_interleaved defaults to False, so existing Qwen2-VL / GPT users see no behavior change.

Test plan

• Existing MRoPE tests still pass with mrope_interleaved=False (default).
• New layout matches HF apply_interleaved_mrope reference on a fixed (position_ids, mrope_section) input.

Notes

This is the core prerequisite for an upcoming examples/multimodal_dev/ PR that adds Qwen3.5-VL.

🤖 Generated with Claude Code