[Southbound]: Create a Native Protocol for MP and non-MP by sammshen · Pull Request #2642 · LMCache/LMCache

sammshen · 2026-02-26T10:53:43Z

Implementation of #2632 as a generalization of the strategies employed in #2541

gemini-code-assist · 2026-02-26T10:54:09Z

Summary of Changes

Hello @sammshen, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the LMCache project by establishing a robust and extensible architecture for native storage backend integrations. By abstracting common patterns like threading, eventfd-based notifications, and submission/completion queues into a base class, it simplifies the development of new high-performance connectors. The existing Redis client serves as a concrete example of this new framework, showcasing how to leverage C++ for efficient, non-blocking I/O operations while seamlessly integrating with Python's asynchronous capabilities.

Highlights

Native Protocol Abstraction: Introduced a new, generalized C++ framework for building high-performance native storage backend connectors, promoting modularity and reusability.
Redis Client Refactoring: The existing Redis client implementation has been refactored to conform to the new native connector framework, demonstrating its application.
Python Client Generalization: The Python RESPClient was generalized into a ConnectorClientBase, allowing any new native C++ connector to easily integrate with Python's asyncio event loop.
Improved Documentation: Added a comprehensive README.md detailing the process for implementing new native storage backends using the provided framework.
Build System Updates: Modified setup.py to correctly build the refactored Redis C++ extension under the new directory structure.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

csrc/redis/pybind.cpp
- Removed old Redis Python binding.
csrc/redis/resp.cpp
- Removed old Redis C++ implementation.
csrc/redis/resp.h
- Removed old Redis C++ header definitions.
csrc/storage_backends/README.md
- Added documentation explaining the new native backend interface and implementation steps.
csrc/storage_backends/connector_base.h
- Added a templated base class ConnectorBase providing core logic for threading, eventfd, and request/completion queues.
csrc/storage_backends/connector_interface.h
- Added an abstract interface IStorageConnector defining the contract for native storage connectors.
csrc/storage_backends/connector_pybind_utils.h
- Added utilities and a macro (LMCACHE_BIND_CONNECTOR_METHODS) to simplify pybind11 integration with GIL release.
csrc/storage_backends/connector_types.h
- Added common data structures (Op, BatchState, Request, Completion) for inter-thread communication.
csrc/storage_backends/redis/connector.cpp
- Added the new Redis connector implementation, inheriting from ConnectorBase.
csrc/storage_backends/redis/connector.h
- Added the header for the new Redis connector.
csrc/storage_backends/redis/pybind.cpp
- Added the new Python binding for the Redis connector, utilizing connector_pybind_utils.h.
examples/kv_cache_reuse/remote_backends/resp/benchmark_resp_client.py
- Updated import path for RESPClient to reflect its new location.
lmcache/v1/storage_backend/connector/redis_connector.py
- Updated import path for RESPClient to reflect its new location.
lmcache/v1/storage_backend/native_clients/connector_client_base.py
- Created a generic Python client base to handle native C++ connector interactions.
lmcache/v1/storage_backend/native_clients/resp_client.py
- Added the specific Python RESPClient inheriting from ConnectorClientBase.
lmcache/v1/storage_backend/resp_client.py
- Renamed to lmcache/v1/storage_backend/native_clients/connector_client_base.py and refactored into a generic base class.
setup.py
- Updated source file paths and include directories for the lmcache_redis CppExtension build.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a well-designed abstraction for native storage backends, refactoring the existing Redis client into a more generic and extensible "connector" architecture. This is an excellent improvement for maintainability and will make it easier to add new storage backends in the future. The new ConnectorBase provides a solid foundation, and the new README.md is very helpful for developers. My review focuses on a few areas for improvement regarding safety checks that were missed during the refactoring and some code cleanup.

gemini-code-assist · 2026-02-26T11:02:12Z

+      py::memoryview mv = memviews[i].cast<py::memoryview>();
+      py::buffer_info info = py::buffer(mv).request();
+      bufs.push_back(info.ptr);
+      lens.push_back(static_cast<size_t>(info.size));


The previous implementation of the pybind wrapper included validation to ensure the provided memoryview was 1-dimensional and byte-addressable. This validation is missing here. This is a critical safety check to prevent potential memory corruption or crashes if an incorrectly shaped buffer is passed from Python. Please add these checks back.

A similar change is needed in bind_submit_batch_set on lines 94-97.

py::memoryview mv = memviews[i].cast<py::memoryview>(); py::buffer_info info = py::buffer(mv).request(); if (info.ndim != 1) { throw std::runtime_error("memoryview must be 1D"); } if (info.itemsize != 1) { throw std::runtime_error("memoryview must be byte addressable"); } bufs.push_back(info.ptr); lens.push_back(static_cast<size_t>(info.size));

I agree with this opinion. Good to have for future debugging.
Since, info.size is number of elements, not bytes.

gemini-code-assist · 2026-02-26T11:02:12Z

+### 3. create python wrapper
+
+inherit from `connector_client_base.py` which provides asyncio integration, future management, sync/async methods.  
+reference: `lmcache/v1/storage_backend/resp_client.py`


The file path referenced here is outdated due to the refactoring. lmcache/v1/storage_backend/resp_client.py has been moved and refactored. The correct reference should be to the new resp_client.py wrapper.

Suggested change

reference: `lmcache/v1/storage_backend/resp_client.py`

reference: `lmcache/v1/storage_backend/native_clients/resp_client.py`

DongDongJu · 2026-02-26T19:39:12Z

Impressive! I will review in this night. Thanks @sammshen

sammshen · 2026-02-26T23:52:15Z

try to unify with MP mode: #2569

DongDongJu

@sammshen Great work. it seems you made the gem.
I left few comments.

DongDongJu · 2026-02-27T15:27:43Z

+instantiated in connector_base.h and further overridden by custom storage
+connectors
+*/
+class IStorageConnector {


What is the I meaning in this class name?

InterfaceStorageConnector, sorry if it is unclear

DongDongJu · 2026-02-27T15:31:37Z

+      py::memoryview mv = memviews[i].cast<py::memoryview>();
+      py::buffer_info info = py::buffer(mv).request();
+      bufs.push_back(info.ptr);
+      lens.push_back(static_cast<size_t>(info.size));


I agree with this opinion. Good to have for future debugging.
Since, info.size is number of elements, not bytes.

DongDongJu · 2026-03-02T03:24:15Z

-
-        # future_id -> (Future, op_name)
-        # we support both types of futures since we only their basic interface
        self._pending: Dict[


It seems does not hold references to the submitted memoryviews or their backing objects.
when a task awaiting a Future is cancelled, the awaited Future can be cancelled too, and the coroutine can unwind, dropping the last Python references while C++ worker threads still hold raw pointers.

Can we store keepalive refs per future_id, e.g. _pending[fid] = (fut, op, tuple(bufs))?
And we should only drop those refs after completion is handled or native shutdown is fully done.

great point, added extra lifetime to the pending operations that will be freed either on failure or on completion

DongDongJu · 2026-03-02T03:26:42Z

+
+RedisConnector::~RedisConnector() { close(); }
+
+WorkerConn RedisConnector::create_connection() {


what happened if this func was failed?

good point, right now we just stay dead but we can add reconnection logic

maobaolong

@sammshen Could you also add a pure c++ implemented fs connector?

sammshen · 2026-03-04T02:11:32Z

@@ -0,0 +1,297 @@
+# SPDX-License-Identifier: Apache-2.0


@ApostaC the new adapter for native MP mode adapters

sammshen · 2026-03-04T02:12:11Z

Thanks for review @maobaolong. I think this should be a follow up PR

maobaolong · 2026-03-06T12:04:48Z

+    # Cleanup
+    # -------------------------------------------------------------------
+
+    def close(self) -> None:


if self._closed: return self._closed = True

Suggest to add the closed flag to avoid double close.

maobaolong

Left a minor nit, otherwise, this PR looks good.

maobaolong · 2026-03-06T12:13:50Z

+        )
+
+
+register_l2_adapter_type("resp", RESPL2AdapterConfig)


#2704

I submitted a PR to do a refactor, after that, add a new coming l2_adapter, we never need to change any existing file.

ApostaC

Please make the fixes following our offline discussions. Give an approve here.

Refactors the C++ Redis connector into a generic storage backend framework and adds an L2 adapter bridge for MP mode. Follows the plugin/auto-discovery pattern. - Generalize csrc/redis -> csrc/storage_backends with ConnectorBase<T> template, IStorageConnector interface, and pybind macro - Add NativeConnectorL2Adapter bridging any native connector to L2 - Add RESPL2AdapterConfig with self-registration (plugin pattern) - Refactor ConnectorClientBase as generic base for non-MP mode

maobaolong

LGTM @sammshen Thanks for this great feature, will extend some fs native adapter base on your effort!

Signed-off-by: Samuel Shen <slshen@tensormesh.ai>

Refactors the C++ Redis connector into a generic storage backend framework and adds an L2 adapter bridge for MP mode. Follows the plugin/auto-discovery pattern. - Generalize csrc/redis -> csrc/storage_backends with ConnectorBase<T> template, IStorageConnector interface, and pybind macro - Add NativeConnectorL2Adapter bridging any native connector to L2 - Add RESPL2AdapterConfig with self-registration (plugin pattern) - Refactor ConnectorClientBase as generic base for non-MP mode

Refactors the C++ Redis connector into a generic storage backend framework and adds an L2 adapter bridge for MP mode. Follows the plugin/auto-discovery pattern. - Generalize csrc/redis -> csrc/storage_backends with ConnectorBase<T> template, IStorageConnector interface, and pybind macro - Add NativeConnectorL2Adapter bridging any native connector to L2 - Add RESPL2AdapterConfig with self-registration (plugin pattern) - Refactor ConnectorClientBase as generic base for non-MP mode Signed-off-by: Aaron Wu <aaron.wu@dell.com>

Refactors the C++ Redis connector into a generic storage backend framework and adds an L2 adapter bridge for MP mode. Follows the plugin/auto-discovery pattern. - Generalize csrc/redis -> csrc/storage_backends with ConnectorBase<T> template, IStorageConnector interface, and pybind macro - Add NativeConnectorL2Adapter bridging any native connector to L2 - Add RESPL2AdapterConfig with self-registration (plugin pattern) - Refactor ConnectorClientBase as generic base for non-MP mode Signed-off-by: baoloongmao <baoloongmao@tencent.com>

Refactors the C++ Redis connector into a generic storage backend framework and adds an L2 adapter bridge for MP mode. Follows the plugin/auto-discovery pattern. - Generalize csrc/redis -> csrc/storage_backends with ConnectorBase<T> template, IStorageConnector interface, and pybind macro - Add NativeConnectorL2Adapter bridging any native connector to L2 - Add RESPL2AdapterConfig with self-registration (plugin pattern) - Refactor ConnectorClientBase as generic base for non-MP mode

sammshen requested review from DongDongJu and maobaolong February 26, 2026 10:54

gemini-code-assist Bot reviewed Feb 26, 2026

View reviewed changes

sammshen mentioned this pull request Feb 26, 2026

[Refactor]: Southbound #2632

Open

DongDongJu reviewed Mar 2, 2026

View reviewed changes

maobaolong reviewed Mar 3, 2026

View reviewed changes

sammshen commented Mar 4, 2026

View reviewed changes

sammshen changed the title ~~[Southbound]: Create a Native Protocol~~ [Southbound]: Create a Native Protocol for MP and non-MP Mar 4, 2026

maobaolong reviewed Mar 6, 2026

View reviewed changes

sammshen requested a review from deng451e March 10, 2026 01:05

ApostaC approved these changes Mar 10, 2026

View reviewed changes

deng451e approved these changes Mar 10, 2026

View reviewed changes

sammshen mentioned this pull request Mar 12, 2026

Add filesystem-backed L2 adapter with auto-discovery plugin mechanism #2704

Merged

2 tasks

sammshen force-pushed the sb-interface branch 3 times, most recently from 460f985 to 341bf0f Compare March 12, 2026 08:16

sammshen force-pushed the sb-interface branch from d839b31 to bcb7946 Compare March 13, 2026 06:32

sammshen enabled auto-merge (squash) March 13, 2026 06:51

github-actions Bot added the full Run comprehensive tests on this PR label Mar 13, 2026

maobaolong approved these changes Mar 13, 2026

View reviewed changes

add documentation

332ba83

Signed-off-by: Samuel Shen <slshen@tensormesh.ai>

sammshen merged commit e97bd21 into LMCache:dev Mar 13, 2026
33 of 37 checks passed

	reference: `lmcache/v1/storage_backend/resp_client.py`
	reference: `lmcache/v1/storage_backend/native_clients/resp_client.py`


		RedisConnector::~RedisConnector() { close(); }

		WorkerConn RedisConnector::create_connection() {

		)


		register_l2_adapter_type("resp", RESPL2AdapterConfig)

Conversation

sammshen commented Feb 26, 2026

Uh oh!

gemini-code-assist Bot commented Feb 26, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

DongDongJu commented Feb 26, 2026

Uh oh!

sammshen commented Feb 26, 2026

Uh oh!

DongDongJu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maobaolong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sammshen commented Mar 4, 2026

Uh oh!

maobaolong Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maobaolong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ApostaC left a comment

Choose a reason for hiding this comment

Uh oh!

maobaolong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

maobaolong Mar 6, 2026 •

edited

Loading