[vllm adapter] num_lmcache_cached_tokens by aeon-x · Pull Request #2670 · LMCache/LMCache

aeon-x · 2026-03-02T22:10:16Z

What this PR does / why we need it:
Add num_lmcache_cached_tokens to response of kv_transfer_params

Set up:

test.yaml

extra_config:
  enable_cache_usage_details_in_response: true

LMCACHE_CONFIG_FILE=test.yaml \
vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --no-enable-prefix-caching \
  --kv-transfer-config  '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'

Sample Response:

{"id":"cmpl-a546036e085f8d78", ... , "kv_transfer_params":{"num_lmcache_cached_tokens":1792}}

If applicable:

this PR contains user facing changes - docs added
this PR contains unit tests

gemini-code-assist · 2026-03-02T22:10:30Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request extends the VLLM adapter's functionality by enabling the request_finished method to provide additional insights into caching performance. It introduces the capability to retrieve the number of tokens cached by VLLM and LMCache, which can be instrumental for detailed monitoring and optimization of the caching mechanisms.

Highlights

VLLM Adapter Enhancement: The request_finished method in the VLLM adapter now supports returning vllm_cached_tokens and lmcache_cached_tokens.
Conditional Metric Return: These new caching metrics are included in the return_params dictionary only when the ret_num_cached_toks parameter is specified in the request.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

lmcache/integration/vllm/vllm_v1_adapter.py
- Modified the request_finished function to conditionally add vllm_cached_tokens and lmcache_cached_tokens to the return parameters based on the ret_num_cached_toks flag.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request extends the request_finished method to return the number of cached tokens from vLLM and LMCache. However, the implementation introduces potential Denial of Service (DoS) vulnerabilities due to unsafe dictionary and list access. Specifically, accessing self.load_specs with a key that may have been popped can lead to a KeyError, and accessing request._output_token_ids without an empty check can cause an IndexError. Both issues, triggerable by user-supplied parameters in kv_transfer_params, could crash the vLLM worker or scheduler. A fix has been provided to prevent these crashes and improve robustness.

Signed-off-by: aeon-x <talexcao@gmail.com>

sammshen

LGTM

royyhuang

LGTM

* init Signed-off-by: aeon-x <talexcao@gmail.com> * update Signed-off-by: aeon-x <talexcao@gmail.com> * update Signed-off-by: aeon-x <talexcao@gmail.com> * update Signed-off-by: aeon-x <talexcao@gmail.com> --------- Signed-off-by: aeon-x <talexcao@gmail.com> Signed-off-by: shaoxiawjc <wjc2800@163.com>

* init Signed-off-by: aeon-x <talexcao@gmail.com> * update Signed-off-by: aeon-x <talexcao@gmail.com> * update Signed-off-by: aeon-x <talexcao@gmail.com> * update Signed-off-by: aeon-x <talexcao@gmail.com> --------- Signed-off-by: aeon-x <talexcao@gmail.com> Signed-off-by: Aaron Wu <aaron.wu@dell.com>

* init Signed-off-by: aeon-x <talexcao@gmail.com> * update Signed-off-by: aeon-x <talexcao@gmail.com> * update Signed-off-by: aeon-x <talexcao@gmail.com> * update Signed-off-by: aeon-x <talexcao@gmail.com> --------- Signed-off-by: aeon-x <talexcao@gmail.com>

gemini-code-assist Bot reviewed Mar 2, 2026

View reviewed changes

Comment thread lmcache/integration/vllm/vllm_v1_adapter.py Outdated

Comment thread lmcache/integration/vllm/vllm_v1_adapter.py Outdated

aeon-x added 2 commits March 3, 2026 15:33

init

a9b92b6

Signed-off-by: aeon-x <talexcao@gmail.com>

update

0a86a0f

Signed-off-by: aeon-x <talexcao@gmail.com>

aeon-x force-pushed the num_cached_tokens branch from a4a0a89 to 0a86a0f Compare March 3, 2026 23:33

aeon-x changed the title ~~[vllm adapter] vllm_cached_tokens and lmcache_cached_tokens~~ [vllm adapter]num_lmcache_cached_tokens Mar 3, 2026

aeon-x changed the title ~~[vllm adapter]num_lmcache_cached_tokens~~ [vllm adapter] num_lmcache_cached_tokens Mar 3, 2026

aeon-x added 2 commits March 4, 2026 03:36

update

1c79900

Signed-off-by: aeon-x <talexcao@gmail.com>

update

dbf36e1

Signed-off-by: aeon-x <talexcao@gmail.com>

sammshen approved these changes Mar 6, 2026

View reviewed changes

sammshen requested a review from deng451e March 6, 2026 03:30

royyhuang approved these changes Mar 6, 2026

View reviewed changes

Merge branch 'dev' into num_cached_tokens

0b6dbb4

deng451e approved these changes Mar 6, 2026

View reviewed changes

aeon-x added 3 commits March 7, 2026 06:30

Merge branch 'dev' into num_cached_tokens

a227391

Merge branch 'dev' into num_cached_tokens

0969c80

Merge branch 'dev' into num_cached_tokens

faf16b8

sammshen enabled auto-merge (squash) March 9, 2026 22:40

github-actions Bot added the full Run comprehensive tests on this PR label Mar 9, 2026

aeon-x added 4 commits March 10, 2026 07:54

Merge branch 'dev' into num_cached_tokens

e392a2c

Merge branch 'dev' into num_cached_tokens

d24917a

Merge branch 'dev' into num_cached_tokens

05ef7b7

Merge branch 'dev' into num_cached_tokens

e214edb

sammshen merged commit f192189 into LMCache:dev Mar 11, 2026
22 of 28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vllm adapter] num_lmcache_cached_tokens #2670

[vllm adapter] num_lmcache_cached_tokens #2670
sammshen merged 12 commits intoLMCache:devfrom
aeon-x:num_cached_tokens

aeon-x commented Mar 2, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Mar 2, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

sammshen left a comment

Uh oh!

royyhuang left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

aeon-x commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Mar 2, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

sammshen left a comment

Choose a reason for hiding this comment

Uh oh!

royyhuang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

aeon-x commented Mar 2, 2026 •

edited

Loading