[LMCache CLI] Design and implementation of `lmcache kvcache` by KuntaiDu · Pull Request #2827 · LMCache/LMCache

KuntaiDu · 2026-03-19T22:33:02Z

The initial design of lmcache kvcache. Please refer to the changed files for detail.

If applicable:

this PR contains user facing changes - docs added
this PR contains unit tests

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

gemini-code-assist · 2026-03-19T22:33:19Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a comprehensive design for a new lmcache kvcache command-line interface. The proposed CLI aims to empower users with fine-grained control over the Key-Value (KV) cache state for individual requests, facilitating operations such as inspecting, clearing, pinning, compressing, and ending sessions. By adhering to principles of HTTP-based communication and script-friendly output, this design provides a foundational framework for robust cache management and debugging within the LMCache ecosystem.

Highlights

New CLI Command Design: Introduced the design for the lmcache kvcache command-line interface, focusing on per-request KV cache management.
Core Sub-commands Defined: Defined five key sub-commands: info, clear, pin, compress, and end-session, each targeting specific KV cache operations.
Design Principles Established: Established principles including HTTP-only communication for management, per-request targeting, and script-friendly output with clear exit codes.
API Gaps and Implementation Plan: Identified necessary new HTTP endpoints for several sub-commands and outlined a phased implementation plan for the CLI.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a design document for the lmcache kvcache CLI command. The document is well-structured, comprehensive, and clearly outlines the new functionality, including subcommands for inspecting, clearing, pinning, compressing, and ending sessions for KV caches on a per-request basis. The design thoughtfully considers script-friendliness with features like JSON output and specific exit codes. My feedback includes a couple of suggestions to further improve the scriptability of the JSON output and the user experience of the compress command.

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

ApostaC

LGTM overall! Just wondering what sub-command we already support for now? I suppose only clear?

Other small comments:

End-session should only be used by the serving engine, otherwise it may cause internal state inconsistency
Can we add a user-facing doc (docs/src/mp) for LMCache CLI as well?

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

sammshen · 2026-03-21T07:44:49Z

+Common Patterns
+---------------
+
+**Check if a server is reachable before clearing:**


is using the destructive clear as a reachability check a good pattern?

Here the clear command is not intended to perform reachability check. The goal is that, in case where the clear command fails due to connectivity issue, the return value reflects this. I just updated the doc.

sammshen

LGTM! two small nits

royyhuang · 2026-03-23T21:56:12Z

+
+Every sub-command requires one of these to identify the target KV cache:
+
+- **`--request-id <id>`** (required) — identifies the request whose KV cache


Since request id is required all the time, I feel it would be more convenient to just have lmcache kvcache <subcommand> <req_id>.

lmcache kvcache clear does not take in request id. I will make that clear in the doc.

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

Signed-off-by: Kuntai Du <kuntai@uchicago.edu>

royyhuang

LGTM!

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

…into kuntai-kvcache

maobaolong

LGTM. Thanks for this great feature.

…#2827) * initial design of lmcache kvcache Signed-off-by: KuntaiDu <kuntai@uchicago.edu> * changing of file Signed-off-by: KuntaiDu <kuntai@uchicago.edu> * add lmcache kvcache -h Signed-off-by: KuntaiDu <kuntai@uchicago.edu> * clarify that lmcache kvcache info design is temporary Signed-off-by: KuntaiDu <kuntai@uchicago.edu> * initial implementation of lmcache kvcache Signed-off-by: KuntaiDu <kuntai@uchicago.edu> * UX update Signed-off-by: KuntaiDu <kuntai@uchicago.edu> * remove end-session Signed-off-by: KuntaiDu <kuntai@uchicago.edu> * add user-facing docs Signed-off-by: KuntaiDu <kuntai@uchicago.edu> * update doc and fix comments Signed-off-by: KuntaiDu <kuntai@uchicago.edu> * let request-id be append argument instead of --request-id Signed-off-by: KuntaiDu <kuntai@uchicago.edu> --------- Signed-off-by: KuntaiDu <kuntai@uchicago.edu> Signed-off-by: Kuntai Du <kuntai@uchicago.edu> Co-authored-by: Roy Huang <roy.y.huang@gmail.com>

KuntaiDu added 2 commits March 19, 2026 08:18

initial design of lmcache kvcache

c22f7f6

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

changing of file

651927f

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

gemini-code-assist Bot reviewed Mar 19, 2026

View reviewed changes

Comment thread docs/design/cli/kvcache-command.md

Comment thread docs/design/cli/kvcache-command.md

KuntaiDu added 3 commits March 20, 2026 01:12

add lmcache kvcache -h

9440686

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

clarify that lmcache kvcache info design is temporary

56ccbf1

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

initial implementation of lmcache kvcache

9a84900

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

ApostaC approved these changes Mar 20, 2026

View reviewed changes

UX update

d15f703

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

KuntaiDu changed the title ~~[LMCache CLI][Design] the design of lmcache kvcache~~ [LMCache CLI] Design and implementation of lmcache kvcache Mar 20, 2026

KuntaiDu added 2 commits March 20, 2026 22:19

remove end-session

099ec0c

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

add user-facing docs

a7c81d5

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

sammshen reviewed Mar 21, 2026

View reviewed changes

Comment thread docs/source/cli/kvcache.rst Outdated

sammshen reviewed Mar 21, 2026

View reviewed changes

royyhuang reviewed Mar 23, 2026

View reviewed changes

update doc and fix comments

c957376

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

KuntaiDu requested review from ApostaC, royyhuang and sammshen March 23, 2026 22:31

KuntaiDu added 2 commits March 23, 2026 22:41

let request-id be append argument instead of --request-id

48ddb43

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

Merge branch 'dev' into kuntai-kvcache

bbb60f3

Signed-off-by: Kuntai Du <kuntai@uchicago.edu>

royyhuang approved these changes Mar 24, 2026

View reviewed changes

Merge branch 'dev' into kuntai-kvcache

64e8b71

royyhuang enabled auto-merge (squash) March 24, 2026 21:09

github-actions Bot added the full Run comprehensive tests on this PR label Mar 24, 2026

KuntaiDu added 2 commits March 24, 2026 23:01

fix isort

72c7ac5

Signed-off-by: KuntaiDu <kuntai@uchicago.edu>

Merge branch 'kuntai-kvcache' of https://github.com/KuntaiDu/LMCache …

b445e1a

…into kuntai-kvcache

maobaolong approved these changes Mar 25, 2026

View reviewed changes

Merge branch 'dev' into kuntai-kvcache

76d401b

royyhuang merged commit 130db2b into LMCache:dev Mar 25, 2026
33 of 34 checks passed

KuntaiDu deleted the kuntai-kvcache branch March 25, 2026 22:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[LMCache CLI] Design and implementation of `lmcache kvcache`#2827

[LMCache CLI] Design and implementation of `lmcache kvcache`#2827
royyhuang merged 15 commits intoLMCache:devfrom
KuntaiDu:kuntai-kvcache

KuntaiDu commented Mar 19, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Mar 19, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

ApostaC left a comment

Uh oh!

Uh oh!

sammshen Mar 21, 2026

Uh oh!

KuntaiDu Mar 23, 2026

Uh oh!

sammshen left a comment

Uh oh!

royyhuang Mar 23, 2026

Uh oh!

KuntaiDu Mar 23, 2026

Uh oh!

royyhuang left a comment

Uh oh!

maobaolong left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants


		Every sub-command requires one of these to identify the target KV cache:

		- `--request-id <id>` (required) — identifies the request whose KV cache

Conversation

KuntaiDu commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Mar 19, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

ApostaC left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sammshen Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

KuntaiDu Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

sammshen left a comment

Choose a reason for hiding this comment

Uh oh!

royyhuang Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

KuntaiDu Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

royyhuang left a comment

Choose a reason for hiding this comment

Uh oh!

maobaolong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

KuntaiDu commented Mar 19, 2026 •

edited

Loading