Enable IndexCache for DeepSeek V3.2 by jinyouzhi · Pull Request #21405 · sgl-project/sglang

jinyouzhi · 2026-03-25T11:59:56Z

Motivation

Modifications

Port https://github.com/THUDM/IndexCache
add ut for deepseek-v3.2

Accuracy Tests

python3 -m sglang.launch_server   --model-path /ssd/hf_models/DeepSeek-V3.2-Exp --tp 8 --mem-fraction-static=0.9 --tool-call-parser deepseekv32  --reasoning-parser deepseek-v3 --json-model-override-args '{"index_topk_freq": 4}'

lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:30000/v1/completions,model=/ssd/hf_models/DeepSeek-V3.2-Exp,num_concurrent=100,tokenized_requests=False" --tasks gsm8k

gsm8k with DeepSeek-V3.2-Exp with this PR:

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9583|±  |0.0055|
|     |       |strict-match    |     5|exact_match|↑  |0.9575|±  |0.0056|

main

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9598|±  |0.0054|
|     |       |strict-match    |     5|exact_match|↑  |0.9575|±  |0.0056|

Benchmarking and Profiling

Throghput improved ~ +6.4%
TTFT improved ~ -5.4%
TPOT improved ~ -5.5%

python3 -m sglang.bench_serving   \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model /ssd/hf_models/DeepSeek-V3.2-Exp \
  --dataset-name random \
  --num-prompts 500 \
  --random-input-len 1024 \
  --random-output-len 512

this PR:

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 not set
Successful requests:                     500
Benchmark duration (s):                  113.18
Total input tokens:                      253168
Total input text tokens:                 253168
Total generated tokens:                  131674
Total generated tokens (retokenized):    129011
Request throughput (req/s):              4.42
Input token throughput (tok/s):          2236.90
Output token throughput (tok/s):         1163.42
Peak output token throughput (tok/s):    1811.00
Peak concurrent requests:                500
Total token throughput (tok/s):          3400.32
Concurrency:                             267.13
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   60465.83
Median E2E Latency (ms):                 60908.48
P90 E2E Latency (ms):                    103027.63
P99 E2E Latency (ms):                    111428.53
---------------Time to First Token----------------
Mean TTFT (ms):                          47792.35
Median TTFT (ms):                        47176.86
P99 TTFT (ms):                           102048.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          49.13
Median TPOT (ms):                        49.81
P99 TPOT (ms):                           65.31
---------------Inter-Token Latency----------------
Mean ITL (ms):                           48.31
Median ITL (ms):                         35.71
P95 ITL (ms):                            116.53
P99 ITL (ms):                            158.38
Max ITL (ms):                            1811.18
==================================================

main

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 not set
Successful requests:                     500
Benchmark duration (s):                  120.48
Total input tokens:                      253168
Total input text tokens:                 253168
Total generated tokens:                  131674
Total generated tokens (retokenized):    129011
Request throughput (req/s):              4.15
Input token throughput (tok/s):          2101.31
Output token throughput (tok/s):         1092.90
Peak output token throughput (tok/s):    1617.00
Peak concurrent requests:                500
Total token throughput (tok/s):          3194.21
Concurrency:                             265.43
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   63957.86
Median E2E Latency (ms):                 64404.77
P90 E2E Latency (ms):                    108928.34
P99 E2E Latency (ms):                    118216.62
---------------Time to First Token----------------
Mean TTFT (ms):                          50530.24
Median TTFT (ms):                        49860.22
P99 TTFT (ms):                           107822.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          52.01
Median TPOT (ms):                        52.70
P99 TPOT (ms):                           68.16
---------------Inter-Token Latency----------------
Mean ITL (ms):                           51.18
Median ITL (ms):                         38.67
P95 ITL (ms):                            121.57
P99 ITL (ms):                            160.03
Max ITL (ms):                            1821.74
==================================================

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review Process

Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
After green CI and required approvals, ask Merge Oncalls to merge.

gemini-code-assist · 2026-03-25T12:00:20Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates the IndexCache mechanism into DeepSeek V3.2 models within the SGLang framework. The primary goal is to enhance inference performance by intelligently skipping redundant top-k index computations in attention layers, leveraging a technique ported from THUDM. This optimization leads to notable improvements in throughput and latency, while maintaining accuracy, and is supported by new unit tests to ensure stability across different distributed processing configurations.

Highlights

IndexCache Enablement: Enabled IndexCache for DeepSeek V3.2 models to significantly improve inference performance.
Core Logic Integration: Integrated the IndexCache mechanism, originally from THUDM, into the SGLang framework, introducing logic to conditionally skip top-k index computation in attention layers and reuse previous layer's indices.
Performance Improvements: Demonstrated significant performance gains, including approximately 6.4% increase in throughput, 5.4% decrease in Time to First Token (TTFT), and 5.5% decrease in Time per Output Token (TPOT).
New Unit Tests: Added comprehensive unit tests for DeepSeek V3.2 with IndexCache, covering various configurations such as DP8, TP8, and both frequency-based and pattern-based skipping.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces an "IndexCache" mechanism for the DeepSeekV2 model, enabling certain attention layers to reuse topk_indices from previous layers instead of recomputing them. This behavior is controlled by skip_topk and next_skip_topk flags, configurable via index_topk_freq or index_topk_pattern in the model's configuration. The changes involve modifying attention forward methods to accept and return topk_indices, updating decoder layers to propagate these indices, and adding a new test file to validate the functionality and performance of the IndexCache across different configurations. A minor issue was noted regarding a typo in an arXiv reference year within the deepseek_v2.py file.

Fridge003 · 2026-03-25T19:13:11Z

Thanks for supporting! Will review on it soon

baonudesifeizhai · 2026-03-25T22:48:10Z

HI could you also provide Traffic request rate: at 1 5 10 20 ? thanks!

ChefWu551 · 2026-03-27T02:38:44Z

I adapted this codebase for NPU, and also validated the corresponding accuracy. The CEval accuracy score is 0.9198, and the E2E latency shows about 1.2x speedup. For details, please refer to PR #21502.

jinyouzhi · 2026-03-27T06:48:06Z

HI could you also provide Traffic request rate: at 1 5 10 20 ? thanks!

Nice advice. thanks

Fridge003 · 2026-03-30T08:27:37Z

@jinyouzhi Have you tried how it works with MTP/CP? Also, how this feature affects harder tests like GPQA or AIME25?

Fridge003 · 2026-03-31T06:03:17Z

+from sglang.test.test_utils import ModelLaunchSettings
+from sglang.test.tool_call_test_runner import ToolCallTestParams
+
+register_cuda_ci(est_time=5400, suite="nightly-8-gpu-common", nightly=True)


Please don't add such a huge nightly test for index_cache.
Rather, add a single per-commit test similar to test/registered/8-gpu-models/test_deepseek_v32_basic.py

jinyouzhi · 2026-03-31T14:34:23Z

HI could you also provide Traffic request rate: at 1 5 10 20 ? thanks!

FYI.

Request Rate	Throughput	TTFT	TPOT
1 - Base	794.16	139.23	19.79
1 - Freq=4	796.27 (+0.27%)	138.75 (-0.34%)	15.83 (-20.01%)
5 - Base	3047.37	8356.24	52.49
5 - Freq=4	3167.40 (+3.94%)	7105.00 (-14.97%)	50.57 (-3.66%)
10 - Base	3117.71	29061.12	53.71
10 - Freq=4	3296.38 (+5.73%)	26511.61 (-8.78%)	53.17 (-1.01%)
20 - Base	3018.98	43372.26	55.35
20 - Freq=4	3363.01 (+11.39%)	36536.14 (-15.76%)	50.63 (-8.54%)

python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model /ssd/hf_models/DeepSeek-V3.2-Exp \
  --dataset-name random \
  --num-prompts 500 \
  --random-input-len 1024 \
  --random-output-len 512 \
  --request-rate 1/5/10/20

Fridge003 · 2026-04-02T07:33:22Z

@jinyouzhi Which GPU are you testing on?

jinyouzhi · 2026-04-02T07:40:15Z

@jinyouzhi Which GPU are you testing on?

H20x8

zRzRzRzRzRzRzR · 2026-04-03T11:45:46Z

This works on GLM-5-FP8, and I think it can serve as a tutorial in the SGLang CookBook for users attempting their own deployment, though we cannot guarantee zero precision loss.

Fridge003 · 2026-04-04T03:42:41Z

/tag-and-rerun-ci

Fridge003 · 2026-04-04T09:23:08Z

/rerun-test test_kimi_linear_models.py

Fridge003 · 2026-04-05T06:17:49Z

/rerun-stage stage-c-test-8-gpu-h200

Fridge003 · 2026-04-05T06:17:56Z

/rerun-stage stage-c-test-4-gpu-b200

github-actions · 2026-04-05T06:18:18Z

✅ Triggered stage-c-test-8-gpu-h200 to run independently (skipping dependencies). View workflow run

github-actions · 2026-04-05T06:18:22Z

✅ Triggered stage-c-test-4-gpu-b200 to run independently (skipping dependencies). View workflow run

Fridge003 · 2026-04-05T06:44:14Z

/rerun-stage stage-c-test-deepep-8-gpu-h200

github-actions · 2026-04-05T06:44:41Z

✅ Triggered stage-c-test-deepep-8-gpu-h200 to run independently (skipping dependencies). View workflow run

The index cache feature makes forward_core return a tuple (hidden_states, topk_indices) for NSA models. The TBO overlapped operations path (op_core) didn't unpack this tuple, causing 'tuple' object has no attribute 'shape' errors in DeepEP tests with DeepSeek V3.2. Fix: - Unpack the tuple in op_core, discarding topk_indices (TBO path doesn't propagate it between layers) - Fall back to computing topk when prev_topk_indices is None (i.e., in TBO mode) even if skip_topk is set, to avoid using None as topk indices Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>

dongyibo · 2026-04-07T13:32:03Z

Hello, I understand that the larger the index_topk_freq setting is (i.e., the more S layers there are), the greater the impact on output accuracy. Is that correct? @jinyouzhi

jinyouzhi · 2026-04-07T13:34:39Z

Hello, I understand that the larger the index_topk_freq setting is (i.e., the more S layers there are), the greater the impact on output accuracy. Is that correct? @jinyouzhi

Yes

Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>

jinyouzhi added 3 commits March 25, 2026 17:22

add indexcache support for dsa models

ac2522c

add ut for dsv32

e14b367

cleanup debug msg

723c6f0

jinyouzhi requested review from Fridge003, ch-wan, fzyzcjy, ispobock and merrymercy as code owners March 25, 2026 11:59

github-actions Bot added the deepseek label Mar 25, 2026

gemini-code-assist Bot reviewed Mar 25, 2026

View reviewed changes

Comment thread python/sglang/srt/models/deepseek_v2.py

ChefWu551 mentioned this pull request Mar 27, 2026

[NPU] enable index Cache for npu #21502

Merged

5 tasks

Fridge003 self-assigned this Mar 27, 2026

Fridge003 added the high priority label Mar 27, 2026

Merge branch 'main' into ic

7c208a9

Fridge003 mentioned this pull request Mar 30, 2026

[Feature Request] Integrate IndexCache to optimize DSA indexer overhead #21663

Closed

Merge branch 'main' into ic

adde6f1

Fridge003 requested changes Mar 31, 2026

View reviewed changes

mmangkad mentioned this pull request Apr 2, 2026

[WIP] Implement IndexCache for GLM-5/DeepSeek V3.2 #21925

Closed

ChefWu551 mentioned this pull request Apr 3, 2026

[NPU] pre-enable index Cache for npu #21991

Closed

5 tasks

zRzRzRzRzRzRzR mentioned this pull request Apr 3, 2026

update with stable version(default method) sgl-project/sgl-cookbook#224

Merged

upd test

5546973

github-actions Bot added the run-ci label Apr 4, 2026

Fridge003 approved these changes Apr 4, 2026

View reviewed changes

Fridge003 added 2 commits April 4, 2026 01:07

fix bug

779396c

fix bug

7645e6b

Merge branch 'main' into ic

6080eb1

Fridge003 and others added 2 commits April 5, 2026 00:11

Merge remote-tracking branch 'origin/main' into ic

efd008d

Fridge003 merged commit 5a35316 into sgl-project:main Apr 5, 2026
240 of 278 checks passed

Fridge003 added a commit that referenced this pull request Apr 5, 2026

Enable IndexCache for DeepSeek V3.2 (#21405)

07f57fc

Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>

JustinTong0323 pushed a commit to JustinTong0323/sglang that referenced this pull request Apr 7, 2026

Enable IndexCache for DeepSeek V3.2 (sgl-project#21405)

f5267f9

Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>

Fridge003 added a commit that referenced this pull request Apr 7, 2026

Enable IndexCache for DeepSeek V3.2 (#21405)

86c1c44

Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>

xiezhq-hermann pushed a commit to antgroup/sglang that referenced this pull request Apr 7, 2026

Enable IndexCache for DeepSeek V3.2 (sgl-project#21405)

3278af8

Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>

Fridge003 mentioned this pull request Apr 8, 2026

[Tracking] GLM-5 (G)B200 Functional Support and Optimizations #19380

Open

caijixueIT pushed a commit to caijixueIT/sglang that referenced this pull request Apr 11, 2026

Enable IndexCache for DeepSeek V3.2 (sgl-project#21405)

f957f09

Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>

yhyang201 pushed a commit to yhyang201/sglang that referenced this pull request Apr 22, 2026

Enable IndexCache for DeepSeek V3.2 (sgl-project#21405)

9e4a664

Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>

sunway513 mentioned this pull request Apr 30, 2026

Hand-off: DSV4-Pro Sprint 6/7 — what to use, what to skip, and how to continue multi-req debug (#37) sunway513/ATOM#60

Open

Conversation

jinyouzhi commented Mar 25, 2026

Motivation

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Review Process

Uh oh!

gemini-code-assist Bot commented Mar 25, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Fridge003 commented Mar 25, 2026

Uh oh!

baonudesifeizhai commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChefWu551 commented Mar 27, 2026

Uh oh!

jinyouzhi commented Mar 27, 2026

Uh oh!

Fridge003 commented Mar 30, 2026

Uh oh!

Fridge003 Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jinyouzhi commented Mar 31, 2026

Uh oh!

Fridge003 commented Apr 2, 2026

Uh oh!

jinyouzhi commented Apr 2, 2026

Uh oh!

zRzRzRzRzRzRzR commented Apr 3, 2026

Uh oh!

Fridge003 commented Apr 4, 2026

Uh oh!

Fridge003 commented Apr 4, 2026

Uh oh!

Fridge003 commented Apr 5, 2026

Uh oh!

Fridge003 commented Apr 5, 2026

Uh oh!

github-actions Bot commented Apr 5, 2026

Uh oh!

github-actions Bot commented Apr 5, 2026

Uh oh!

Fridge003 commented Apr 5, 2026

Uh oh!

github-actions Bot commented Apr 5, 2026

Uh oh!

Uh oh!

dongyibo commented Apr 7, 2026

Uh oh!

jinyouzhi commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

baonudesifeizhai commented Mar 25, 2026 •

edited

Loading