Add lookup api to support dynamic recreate lookup client/server by maobaolong · Pull Request #2625 · LMCache/LMCache

maobaolong · 2026-02-21T03:13:46Z

What this PR does / why we need it:

This pull request enhances the system's operational capabilities by introducing a suite of API endpoints that allow for the dynamic management and reconfiguration of lookup clients and servers. This change provides the ability to update and switch lookup strategies on the fly, improving system adaptability and reducing downtime associated with configuration changes.

Special notes for your reviewers:

If applicable:

this PR contains user facing changes - docs added
this PR contains unit tests

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

gemini-code-assist · 2026-02-21T03:14:01Z

Summary of Changes

Hello @maobaolong, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the system's operational capabilities by introducing a suite of API endpoints that allow for the dynamic management and reconfiguration of lookup clients and servers. This change provides the ability to update and switch lookup strategies on the fly, improving system adaptability and reducing downtime associated with configuration changes.

Highlights

New API Endpoints for Lookup Management: Introduced new API endpoints: GET /lookup/info, POST /lookup/close, POST /lookup/create, and POST /lookup/recreate, enabling dynamic management of lookup clients and servers.
Dynamic Reconfiguration: The new endpoints allow for runtime reconfiguration of the lookup mechanism (client for scheduler, server for worker) without requiring a service restart, enhancing operational flexibility.
Manager Logic Implementation: Added corresponding methods within the LMCacheManager to handle closing, creating, recreating, and providing information about lookup clients and servers.
Documentation and Testing: New API endpoints are thoroughly documented in vllm_apis.rst and covered by comprehensive unit tests in test_lookup_api.py.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

docs/source/internal_api_server/vllm_apis.rst
- Documented new API endpoints for dynamic lookup client/server management, including usage examples and important configuration notes.
lmcache/v1/internal_api_server/vllm/lookup_api.py
- Added new FastAPI routes for /lookup/info, /lookup/close, /lookup/create, and /lookup/recreate to expose dynamic lookup management functionality.
- Implemented helper function _json_response for consistent JSON responses.
lmcache/v1/manager.py
- Implemented close_lookup_client and close_lookup_server methods to gracefully shut down lookup components.
- Implemented create_lookup_client and create_lookup_server methods to instantiate new lookup components based on current configuration, supporting a dryrun option.
- Implemented recreate_lookup_client and recreate_lookup_server methods as a convenience for closing and then creating lookup components.
- Added _get_lookup_type_str helper to provide detailed type information for lookup clients/servers, including wrapper chains.
- Implemented get_lookup_info to retrieve the current status and type of lookup client and server.
tests/v1/internal_api_server/test_lookup_api.py
- Added a new test file with comprehensive unit tests for all new /lookup/* API endpoints.
- Included tests for scheduler and worker roles, dryrun functionality, error handling, and full integration flows.

Activity

maobaolong created this pull request to add dynamic lookup API support.
New API endpoints and their underlying logic were implemented.
Documentation for the new API was added.
Extensive unit tests were written to ensure the correctness and robustness of the new features.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

The pull request introduces a set of internal APIs to manage the lifecycle of lookup clients and servers at runtime. This allows for dynamic reconfiguration (e.g., enabling/disabling scheduler bypass) without restarting the service. The implementation includes the necessary manager methods, FastAPI routes, documentation, and unit tests. The overall structure is sound, but there are minor inconsistencies in the documentation examples and an opportunity to improve the maintainability of the type-string generation logic.

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: maobaolong <307499405@qq.com>

sammshen

LGTM

chunxiaozheng

lgtm

…che#2625) * Add lookup api to support dynamic recreate lookup client/server Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Update lmcache/v1/internal_api_server/vllm/lookup_api.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: maobaolong <307499405@qq.com> * Update lmcache/v1/manager.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: maobaolong <307499405@qq.com> --------- Signed-off-by: baoloongmao <baoloongmao@tencent.com> Signed-off-by: maobaolong <307499405@qq.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…che#2625) * Add lookup api to support dynamic recreate lookup client/server Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Update lmcache/v1/internal_api_server/vllm/lookup_api.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: maobaolong <307499405@qq.com> * Update lmcache/v1/manager.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: maobaolong <307499405@qq.com> --------- Signed-off-by: baoloongmao <baoloongmao@tencent.com> Signed-off-by: maobaolong <307499405@qq.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: shaoxiawjc <wjc2800@163.com>

…che#2625) * Add lookup api to support dynamic recreate lookup client/server Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Update lmcache/v1/internal_api_server/vllm/lookup_api.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: maobaolong <307499405@qq.com> * Update lmcache/v1/manager.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: maobaolong <307499405@qq.com> --------- Signed-off-by: baoloongmao <baoloongmao@tencent.com> Signed-off-by: maobaolong <307499405@qq.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Aaron Wu <aaron.wu@dell.com>

Add lookup api to support dynamic recreate lookup client/server

e4ebad8

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

gemini-code-assist Bot reviewed Feb 21, 2026

View reviewed changes

Comment thread lmcache/v1/internal_api_server/vllm/lookup_api.py Outdated

Comment thread lmcache/v1/manager.py Outdated

maobaolong and others added 2 commits February 21, 2026 11:26

Update lmcache/v1/internal_api_server/vllm/lookup_api.py

fc0eab9

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: maobaolong <307499405@qq.com>

Update lmcache/v1/manager.py

d18205a

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: maobaolong <307499405@qq.com>

maobaolong added the full Run comprehensive tests on this PR label Feb 21, 2026

sammshen approved these changes Feb 22, 2026

View reviewed changes

maobaolong enabled auto-merge (squash) February 22, 2026 02:50

chunxiaozheng approved these changes Feb 24, 2026

View reviewed changes

maobaolong merged commit e411ed7 into LMCache:dev Feb 24, 2026
32 of 33 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add lookup api to support dynamic recreate lookup client/server#2625

Add lookup api to support dynamic recreate lookup client/server#2625
maobaolong merged 3 commits intoLMCache:devfrom
maobaolong:iapi

maobaolong commented Feb 21, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Feb 21, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

sammshen left a comment

Uh oh!

chunxiaozheng left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

maobaolong commented Feb 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Feb 21, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

sammshen left a comment

Choose a reason for hiding this comment

Uh oh!

chunxiaozheng left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

maobaolong commented Feb 21, 2026 •

edited

Loading