Motivation

The current radix cache implementations (RadixCache, MambaRadixCache, SWARadixCache, etc.) share a large amount of logic but are maintained as separate, diverged copies. This leads to code duplication, inconsistent behavior, and high maintenance burden when extending cache functionality to new model types (e.g., hybrid linear models, sliding window attention models).

The goal of this refactor is to unify the hybrid radix cache hierarchy around a common base interface, making it easier to:

• Add new cache variants without duplicating logic
• Ensure consistency across all cache implementations
• Enable future extensions (e.g., HiCache, PD disaggregation, new model architectures)

@hzh0425 @yizhang2077 @pansicheng @ispobock

Progress

Stage 0: Unify Radix Tree Interface

• [RadixTree][1/N Refactor]: Support unified match_prefix params #17142 @hzh0425
• [RadixTree][2/N Refactor]: swa cache init tiny refactor #17397 @yizhang2077
• [RadixTree][3/N Refactor]：Support unified insert/evict params #17401 @hzh0425
• [RadixTree][4/N Refactor]: Move available_and_evictable_str to individual radix cache classes #17852 @pansicheng
• [RadixTree][5/N Refactor]: Introduce pre- and post-processing methods for key matching #18147 @hzh0425
• [RadixTree][6/N Refactor]: Refactor SWARadixTree to simplify the computation and alignment of bigram keys. #19427 @hzh0425
• [RadixTree][7/N Refactor]: Refactor mamba radix tree, release dup kvcache in insert func #19429 @hzh0425
• [RadixTree][8/N Refactor]: unify lock interface #20330 @pansicheng
• [RadixTree][9/N Refactor]: Support unified init_load_back params #20590 @hzh0425

Stage 1: Support Unified HybridRadixTree V2

• [RaidxTree Refactor]: Support Unified HybridRadixTree V2 #21206 @hzh0425 @ispobock @yizhang2077 @pansicheng
  • Move UnifiedRadixTree into the main mem_cache directory and place reusable components under a separate dedicated directory.
  • provide detailed Markdown documentation, including each api's purpose, inputs and outputs, mutation behavior, and time complexity.
  • Review all tree APIs carefully, ensure that core tree operations remain linear time cost
• Add HiCache support, including offloading support for mamba, swa, and full components. @hzh0425 @ispobock @pansicheng
  • [UnifiedRadixTree]: Support HiCache Framework for UnifiedRadixTree  #23316
  • Support swa HiCache for unified radix cache #23391
  • [UnifiedTree]: Support HiCache For DeepSeek_V4 #24691
• Mamba Cache V2 Support Bigram key @yizhang2077 @hzh0425
• Add Session Cache Support @pansicheng @hnyls2002
  • integrate streaming session into UnifiedRadixCache #23145
  • [core] Always-on StreamingSession in UnifiedRadixCache #23202
• LRU Optimization [Perf] UnifiedRadixCache LRU optimization proposals #24072
• Remove swa mapping and update all backends to align with the new unified tree design（low priority). @ispobock
• Once this phase is stable, remove all other tree implementations（including RadixTree、SwaRadixTree、MambaRadixTree） and keep only UnifiedRadixTree.
• Compatibility with other features (PD, Spec)

Stage 2: Tree Interface Cleanup and Optimization

• Clarify whether req should be mutable and immutable across the entire tree interface. @ispobock @hzh0425
• Redesign confusing APIs such as cache_finished and cache_unfinished so their names accurately reflect their behavior. @hnyls2002
• Separate the responsibilities of match_prefix and match_prefix_and_split more clearly, especially around whether they mutate the tree. @pansicheng
• Explore tree compression strategies, including leaf-node merging and block-size-based merging. Evaluate whether merge logic can be integrated into match_prefix without making the API semantics harder to understand. @hzh0425

Stage 3: Long-Term Rewrite

• Replace List [int] with a continuous array in memory for token storage, as a prerequisite for Rust migration
• Plan for a Rust rewrite once the tree design has sufficiently converged and the API semantics are stable.
• Use the stabilized unified design and cleaned-up interfaces as the foundation for the Rust implementation.

Related Issues

• [Feature] Memory Cache System Refactoring Road Map (Mem Cache V2) #12587
• Refactor swa_radix_cache.py #13742
• [Feature] HiCache for Hybrid and Sparse LLMs #12826