Teammates: @Makcum888e @OrangeRedeng @e-martirosian @ssshinigami

Foundational Capability Development

initial npu support for SGLang-Diffusion was added in #13662

• support NPU

  • support NPUPlatformBase (python\sglang\multimodal_gen\runtime\platforms\npu.py)
  • support models
    • Wan/Qwen-Image/FLUX [NPU][diffusion] model: support WAN/FLUX/Qwen-Image/Qwen-Image-edit on Ascend #13662
    • MOVA @LLThomas [Diffusion][NPU] Add support for MOVA #21633
    • Hunyuan3D @e-martirosian [Diffusion][NPU] Add support for Hunyuan3D #20352
      • rasterize_image_npu
    • BAGEL
    • GLM-Image
    • ERNIE-Image
    • Hunyuan-Image/Video
    • ERNIE-Image
    • GPT-Image 2.0
    • Happy Horse 1.0

• Tools

  • profiling [Diffusion] [NPU] Enable profiler on NPU #17807
  • benchmark [diffusion] benchmark: Add SLO metric for SGL-Diffusion #18907

• documentation

  • get started [Diffusion] [NPU] [Doc] Add NPU documentation for sglang-diffusion #18894
  • basic usage
  • advanced features
  • developer guide
  • cookbook

• CI/CD & Observability

  • PR-test design(include testcase and corresponding time)
  • nightly-test design(include testcase and corresponding time)
  • Improve CI monitor workflow
  • More tests
    • Different parallelism
    • ulysses+ring+tp
    • ulysses+ring
    • tp+ring

• Models

• accuracy test

  • module test
    • v-bench

Performance Optimization

P0 means High priority, P1 means Medium priority, P2 means low priority

• Graph optimization

  • [P1] support ACLGraph [in progress] @Alisehen
  • support torch.compile @Alisehen [NPU][diffusion] npu support enable_torch_compile for torchair backend on diffusion models  #20687

• Advanced attention

  • USPAttention
  • LocalAttention
  • UlyssesAttention
  • [P2] UlyssesAttention_VSA
    • Video Sparse Attention P2
  • MinimalA2AAttnOp
    • [P2] Sparse Linear attention backend
    • [P2] Sage Sparse Linear attention backend
  • [P2] SAGEAttention
  • [P0] RainFusion Attention @Napkin-AI @Svoloch2940194
  • Block Sparse Attention
    • [P0] Initial support @Napkin-AI @Svoloch2940194
    • [P1] Improved version support
  • Laser Attention
    • [P0] Initial support @Napkin-AI @Svoloch2940194
    • [P1] Improved version support
  • Ulysses Anything Attention
  • Support Cross attention skipping for Ulysses SP

• Cache

  • Cache-DiT
  • TeaCache
  • [P1] AttentionCache

• memory optimization

  • CPU offload (need support for quant models)

• parallelism

  • TP
  • Sequence Parallel(SP)
  • DP
  • Image-model SP(with padding)
  • VAE Decoding Parallel @gxxx-hum [Feature] [NPU] Support for Diffusion VAE Parallel Decoding #20764
  • VAE tiling Parallel encode
  • PipelineStage Parallelism
  • Ring Sequence Parallel [NPU] Update torch and torch_npu version for NPU #20013 [diffusion] [NPU] support ring attention on NPU with FA #21383 [diffusion][doc]: add ring sp performance benchmark page #20998
  • Ulysses Sequence Parallel
  • CFG parallelism
  • Expert Parallelism

• Other optimizations

  • Support fast_layernorm from MindIE-SD @Napkin-AI @Svoloch2940194
  • Support BF16 datatype in vae
  • NZ for MoE support

• quantization and compression @OrangeRedeng

  • modelslim -

    • Wan2.2 [Diffusion] [NPU] Wan2.2-T2V-A14B-Diffusers modelslim quantization support #17996
    • Wan2.1 - need to test with [Diffusion] [NPU] Wan2.2-T2V-A14B-Diffusers modelslim quantization support #17996
    • SD3
    • Open-Sora-Plan v1.2
    • Flux.1
    • HunyuanVideo 
    • Qwen-Image-Edit

  • nunchaku

  • TurboDiffusion

• Scheduling & Serving

  • ComfyUI web serving
  • LoRA support
  • Batch Scheduler

Advanced features

• Real time diffusion
• Postprocess
  • Upscaling [diffusion] Postprocess: frame interpolation and upscaling #18327
  • Frame interpolation [diffusion] Postprocess: implement frame interpolation using RIFE #19384

Refactoring

• Refactor Custom Op
• Refactor Rotary Embedding: Custom Op for RoPE