Current Status

There has been some prior efforts on context parallelism:

• Refactor and unify parallel states for CP refactor context parallel state  #17213
• Prefill CP for DSA models (Dpsk v32/glm 5) (1/n)support context parallel with deepseekv3.2-DSA #12065 [DeepSeek v3.2] opt Context Parallelism: support fused moe, multi batch and fp8 kvcache #13959
• Prefill CP for MHA/GQA model (Qwen3-MoE + FA3 attention backend) Support Qwen3 MoE context parallel #18233

However, the current support only covers a small number of models. To generalize the usage of CP, we need to work on following features and tasks

Prefill CP

• Compatibility with allreduce fusion Support allreduce fusion with cp #21249
• Support attention cp size != moe_dp size Support moe_dp_size = 1 for various attention_cp_size #22003
• Prefill CP for MLA models (Dpsk v3/Kimi-K2.5) [Feature] Prefill Context Parallelism Support for MLA Models #22896 [Bug] KIMI-K2.5 can't use context parallel #22692 @kpham-sgl
• Prefill CP for more MHA/GQA attention backends (flashinfer/trtllm-mha)
• Optimize all-to-all communication pattern for CP+EP with hybrid EP @Fridge003
• Support batch size > 1 Support batch size > 1 when enable CP #23269
• Support sharded kv cache between CP ranks, can be any of the following: @Shunkangz
  • Sharding KV cache by layers dim (https://z.ai/blog/scaling-pain)
  • Sharding KV cache by sequence dim (probably with Ring Attention)
  • Sharding KV cache by flattened(page_size x num_heads x hidden_state) dim
• Unifying the usage of communicators and util functions, for deduplication of messy codes in DSA CP. [Refactor] Deduplicate NSA utils.py into cp_utils.py for context parallel #22914

Decode CP

• Helix Parallelism [Feature] Helix (TP + CP) Parallelism: A2A Communication + Decoupled Attention/FFN Parallelism #19436 [Feature] DCP: Decode Context Parallelism with A2A and FA3 Backend Support #21637
• Decode CP for MLA models [feature] implement dcp for deepseek_v2 #14194
• Decode CP for MHA/GQA models [Feature] Add DCP support for GQA with flashinfer #14982
• Decode CP for DSA models (Dpsk v32/GLM 5) [Feature] Add DCP support for DeepSeek v3.2 #18167

@Fridge003 @Shunkangz @ShangmingCai @ch-wan @kpham-sgl