Here is the roadmap for refactoring SGLang's memory caching system (mem cache v2).

Design Goals

Support arbitrary feature combinations regarding the KV cache management sub-system.

• Prefix caching
• Hierarchical caching
• PD disaggregation
• Request retraction/abortion
• Hybrid and Sparse attention
• Spec Decode

Road Map

• Release Initial Code
  • Existing code cleaning
    • Clean match_prefix and prepare_for_extend for mem cache V2 #11200
    • Refactor kv cache free #11351
    • Unify memory management across (overlap, non-overlap) x (page>=1) x (spec, non-spec, spec v2) x (retract, finished) #12224
  • New interface + chunk cache + triton backend
• Prefix Cache Reusing
  • Radix Cache
  • SWA Radix Cache
  • Mamaba Radix Cache
• KV cache transfer
  • PD disaggregation
  • Hierarchical KV Cache
  • KV cache offloading for Retraction
• More attention to backend integration