Motivation

I am deeply grateful to @xiezhq-hermann for implementing the Hierarchical Caching feature, which has expanded the storage capacity of the KV cache. However, the version only supports MHA and not MLA. This PR introduces support for Hierarchical Caching in the context of MLA.
At present, there might currently be a bug when tp > 1, and @xiezhq-hermann is fixing it.

Modifications

I have abstracted a base class named BaseTokenToKVPoolHost and implemented two classes: MHATokenToKVPoolHost and MLATokenToKVPoolHost.

Benchmark

enable-hierarchical-cache

CUDA_VISIBLE_DEVICES=1 python -m sglang.launch_server --model-path /data00/models/DeepSeek-V2-Lite-Chat --port 30000 --enable-hierarchical-cache --trust-remote-code --mem-fraction-static 0.4

disable-hierarchical-cache

CUDA_VISIBLE_DEVICES=1 python -m sglang.launch_server --model-path /data00/models/DeepSeek-V2-Lite-Chat --port 30000 --trust-remote-code --mem-fraction-static 0.4

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.