Add filesystem-backed L2 adapter with auto-discovery plugin mechanism by maobaolong · Pull Request #2704 · LMCache/LMCache

maobaolong · 2026-03-06T12:10:23Z

What this PR does / why we need it:

Add FSL2Adapter — a filesystem-backed L2 adapter that stores KV cache chunks
as raw tensor files on disk. This enables persistent L2 caching across restarts.

What changed:

New fs_l2_adapter.py: async disk I/O via aiofiles, atomic writes (tmp + rename),
O_DIRECT support, read-ahead, and startup file scanning.
Refactored config.py and __init__.py to use a registry + pkgutil.iter_modules
auto-discovery mechanism. Adding a new L2 adapter no longer requires modifying any
existing code — just drop a new module in the l2_adapters/ package.
Moved MockL2AdapterConfig and FSL2AdapterConfig into their respective adapter
modules for self-contained registration.

Config options (all align with FSConnector naming):

{ "type": "fs", "base_path": "/path/to/l2_cache", "relative_tmp_dir": ".tmp", "read_ahead_size": 65536, "use_odirect": true }

Test

Start.sh

#!/bin/bash
# LMCache Server 启动
python3 -m lmcache.v1.multiprocess.server \
    --host localhost \
    --port 15555 \
    --chunk-size 256 \
    --l1-size-gb 5 \
    --eviction-policy LRU \
    --max-workers 1 \
    --l2-adapter '{"type":"fs","base_path":"/data1/baoloongmao/data/20260306/mp_mode/l2_cache"}' \
    2>&1 | tee /data1/LMCache/logs/lmcache_server.log &

# vLLM Server 启动
KV_TRANSFER_CONFIG='{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.port":15555}}'

python3 -m vllm.entrypoints.cli.main serve /data1/model/DeepSeek-V2-Lite-Chat/ \
    -tp 8 \
    --load-format dummy \
    --trust-remote-code \
    --served-model-name vllm_cpu_offload \
    --gpu_memory_utilization 0.5 \
    --max-num-seqs 64 \
    --no-enable-prefix-caching \
    --port 8000 \
    --kv-transfer-config "${KV_TRANSFER_CONFIG}" \
    2>&1 | tee /data1/LMCache/logs/vllm_server.log &

Logs


 ls /data1/baoloongmao/data/20260306/mp_mode/l2_cache/
-SEP-data1-SEP-model-SEP-DeepSeek-V2-Lite-Chat-SEP-@0x01000100@f8466387aa1f22723b242e09cfca1311eff3945905bfd40fb67307b6b6fbae8f.data


[2026-03-06 22:09:12,569] LMCache INFO: Initialized FSL2Adapter with base_path=/data1/baoloongmao/data/20260306/mp_mode/l2_cache, relative_tmp_dir=None, scan_on_startup=True, read_ahead_size=None, use_odirect=False, found 1 existing files (fs_l2_adapter.py:277:lmcache.v1.distributed.l2_adapters.fs_l2_adapter)
===
[2026-03-06 22:09:14,842] LMCache INFO: LMCache cache server is running... (server.py:671:__main__)

Special notes for your reviewers:

If applicable:

this PR contains user facing changes - docs added
this PR contains unit tests

gemini-code-assist · 2026-03-06T12:10:44Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the LMCache system by introducing a robust filesystem-backed L2 adapter, providing persistent storage for KV cache data. Concurrently, it implements a flexible auto-discovery mechanism for L2 adapters, transforming the architecture into a plug-and-play system. This change streamlines the process of integrating new storage backends and improves the overall extensibility and resilience of the caching layer.

Highlights

Filesystem-backed L2 Adapter (FSL2Adapter): Introduced a new L2 adapter that stores KV cache chunks as raw tensor files on disk, enabling persistent caching across restarts. It uses aiofiles for asynchronous I/O, implements atomic writes via temporary files and renames, and supports O_DIRECT for bypassing the page cache, along with read-ahead capabilities. The adapter also scans existing files on startup to rebuild its in-memory index.
Auto-discovery Plugin Mechanism for L2 Adapters: Refactored the L2 adapter factory to use a registry and pkgutil.iter_modules for automatic discovery. New L2 adapters can now be added by simply creating a new module in the l2_adapters/ package and self-registering, without requiring modifications to existing core files like __init__.py or config.py.
Configuration Centralization: Moved MockL2AdapterConfig into its respective mock_l2_adapter.py module. The new FSL2AdapterConfig is also defined within fs_l2_adapter.py, promoting self-contained adapter implementations and simplifying the config.py file.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

lmcache/v1/distributed/l2_adapters/init.py
- Updated docstring to reflect automatic module discovery.
- Added importlib and pkgutil for dynamic module loading.
- Removed direct import of MockL2Adapter and MockL2AdapterConfig.
- Modified create_l2_adapter to delegate adapter creation to the new registry-based factory function.
lmcache/v1/distributed/l2_adapters/config.py
- Added Callable and Any to imports for type hinting.
- Introduced _L2_ADAPTER_FACTORY_REGISTRY to store adapter factory functions.
- Implemented register_l2_adapter_factory to register adapter creation functions.
- Added create_l2_adapter_from_registry to centralize adapter instantiation via the factory registry.
- Updated get_type_name_for_config docstring to reflect new adapter types.
- Removed MockL2AdapterConfig class definition and its registration, moving it to its dedicated module.
lmcache/v1/distributed/l2_adapters/fs_l2_adapter.py
- Added new file for the FSL2Adapter implementation.
- Defined utility functions _object_key_to_filename and _filename_to_object_key for reversible filename generation.
- Implemented FSL2AdapterConfig for configuring the filesystem adapter, including base_path, relative_tmp_dir, scan_on_startup, read_ahead_size, and use_odirect.
- Developed FSL2Adapter class, inheriting from L2AdapterInterface, with async I/O using aiofiles.
- Included logic for atomic writes, O_DIRECT support, read-ahead, and startup file scanning.
- Integrated self-registration for FSL2AdapterConfig and FSL2Adapter using the new registry mechanism.
lmcache/v1/distributed/l2_adapters/mock_l2_adapter.py
- Moved MockL2AdapterConfig class definition from config.py to this module.
- Added self-registration calls for MockL2AdapterConfig and MockL2Adapter using the new registry functions.
tests/v1/distributed/test_distributed_storage_manager.py
- Updated import path for MockL2AdapterConfig to its new location.
tests/v1/distributed/test_mock_l2_adapter.py
- Updated import path for MockL2Adapter and MockL2AdapterConfig to reflect their new module.
tests/v1/distributed/test_prefetch_controller.py
- Updated import path for MockL2Adapter and MockL2AdapterConfig to reflect their new module.
tests/v1/distributed/test_prefetch_policy.py
- Updated import path for MockL2AdapterConfig to its new location.
tests/v1/distributed/test_store_controller.py
- Updated import path for MockL2Adapter and MockL2AdapterConfig to reflect their new module.
tests/v1/distributed/test_store_policy.py
- Updated import path for MockL2AdapterConfig to its new location.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a significant architectural improvement by adding a filesystem-backed L2 adapter and refactoring the adapter loading mechanism to a plugin-based auto-discovery system, greatly enhancing modularity. While the implementation of the FSL2Adapter is comprehensive, security concerns have been identified regarding the handling of filesystem paths and permissions. Specifically, the cache directory and files are created with overly permissive default permissions, potentially leading to information disclosure. Additionally, the temporary directory configuration lacks proper sanitization against path traversal and relies on assert for validation, which is insecure. Beyond these security aspects, the review also includes suggestions to improve robustness and correctness, particularly around file I/O and parsing logic.

maobaolong · 2026-03-06T13:28:54Z

@ApostaC @sammshen Would you like to take a look at this PR? Thanks!

chunxiaozheng · 2026-03-10T06:15:13Z

+    ) -> None:
+        bitmap = Bitmap(len(keys))
+        for i, key in enumerate(keys):
+            if key not in self._known_keys:


@maobaolong Thanks for the great contribution! But I have a question, it will only be searched from memory index?

Thanks for the remind, this in memory metadata related data structures should be removed to achieve metadata sharing between nodes.

maobaolong · 2026-03-10T07:10:49Z

@chunxiaozheng Thanks for your review, removed all motioned metadata cache purpose data structures. PTAL.

sammshen · 2026-03-10T17:59:25Z

+_FILE_BACKENDS = ("GDS", "GDS_MT", "POSIX", "HF3FS")
+
+
+class NixlStoreL2AdapterConfig(L2AdapterConfigBase):


do you want to move this too?

Yeah thanks for remind me.

ApostaC

Small changes needed. Otherwise LGTM

ApostaC · 2026-03-10T17:59:30Z

+    """
+    if name in _L2_ADAPTER_FACTORY_REGISTRY:
+        raise ValueError(f"L2 adapter factory already registered: {name!r}")
+    _L2_ADAPTER_FACTORY_REGISTRY[name] = factory


It's better to put the adapter creatio in a separate file instead of in config.py.

Also, let's have a more "constrained" signature for the factory callable. Something like Callable[[L2AdapterConfigBase], L2AdapterInterface]?

ApostaC · 2026-03-10T17:59:42Z

+
+# Self-register config type and adapter factory
+register_l2_adapter_type("mock", MockL2AdapterConfig)
+register_l2_adapter_factory("mock", lambda config, **kwargs: MockL2Adapter(config))


Suggested change

register_l2_adapter_factory("mock", lambda config, **kwargs: MockL2Adapter(config))

register_l2_adapter_factory("mock", lambda config: MockL2Adapter(config))

ApostaC · 2026-03-10T20:04:43Z

we also need to change the nixl l2 adapters's config and initialization to use the new code

ApostaC

Did another pass on fs_l2_adapter.py, please see the comments.

Additionally, can we add the UT for this as well?

ApostaC · 2026-03-10T20:12:46Z

Please add a customized report_status() implementation for the FSL2Adapter.

ApostaC · 2026-03-10T20:16:16Z

+            if file_path.exists():
+                bitmap.set(i)


This is a FS I/O operation. Do we want to use aiofiles, making it async, and running it in the asyncio loop?

ApostaC · 2026-03-10T20:18:20Z

+                        "FSL2Adapter failed to store %s",
+                        file_path,
+                    )
+                    if await aiofiles.os.path.exists(tmp_path):


If self._key_to_file_and_tmp_path(key) raises before the tmp_path is initialized, here may have another NameError.

add tmp_path not none check

ApostaC · 2026-03-10T20:19:30Z

+    Fields:
+    - base_path: directory for storing KV cache files.
+    - relative_tmp_dir: optional relative sub-dir for
+      temp files (same as fs_connector_relative_tmp_dir).


Can we have the docstring for read_ahead_size and use_odirect as well?

ApostaC · 2026-03-10T20:29:24Z

Oh, almost forget. Please update the docs as well, thanks!

maobaolong · 2026-03-11T03:56:08Z

@ApostaC @sammshen @chunxiaozheng Thanks for the previous review, addressed all the suggested comments from you, PTAL.

ApostaC

Should be some final comments. Otherwise LGTM.

ApostaC · 2026-03-11T19:52:26Z

Thanks for the doc update, can we also update the https://docs.lmcache.ai/mp/l2_storage.html to tell people how to use the new FS L2 adapter (i.e., how to write the config in the commandline)?

ApostaC · 2026-03-11T19:57:25Z

+    if l1_memory_desc is not None:
+        config.l1_memory_desc = l1_memory_desc  # type: ignore[attr-defined]


This is a bit hacky. Let's pass the l1_memory_desc all the way down to the L2 adapter's __init__ function.
This means that we probably need to change the definition of L2AdapterFactory from Callable[["L2AdapterConfigBase"], "L2AdapterInterface"] to Callable[["L2AdapterConfigBase", "L1MemoryDesc"], "L2AdapterInterface"]

ApostaC · 2026-03-11T19:58:20Z

+logger = init_logger(__name__)
+
+# Type alias for factory callables
+L2AdapterFactory = Callable[["L2AdapterConfigBase"], "L2AdapterInterface"]


Suggested change

L2AdapterFactory = Callable[["L2AdapterConfigBase"], "L2AdapterInterface"]

L2AdapterFactory = Callable[["L2AdapterConfigBase", "L1MemoryDesc"], "L2AdapterInterface"]

ApostaC · 2026-03-11T19:58:54Z

+        self.backend = backend
+        self.backend_params = backend_params
+        self.pool_size = pool_size
+        self.l1_memory_desc: Optional[L1MemoryDesc] = None


We don't need this if we are going to pass the l1_memory_desc via the factory's arguments.

ApostaC · 2026-03-11T19:59:05Z

+    l1_memory_desc = getattr(config, "l1_memory_desc", None)
+    if l1_memory_desc is None:
+        raise ValueError("l1_memory_desc is required to create a NixlStoreL2Adapter.")


We don't need this if we are going to pass the l1_memory_desc via the factory's arguments.

maobaolong · 2026-03-12T03:05:57Z

@ApostaC Thanks for the new review, addressed the comment, PTAL.

sammshen · 2026-03-12T07:07:30Z

+``fs`` -- File-system backed storage
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A pure file-system L2 adapter using async I/O.  Does not require NIXL.


why do you need to specify does not require NIXL here?

Remove it. Added since before this fs adapter, there is only nixl adapter.

sammshen

LGTM

sammshen · 2026-03-12T07:54:29Z

I rebased #2642 to follow this PR. Either one can be merged first

Refactors the C++ Redis connector into a generic storage backend framework and adds an L2 adapter bridge for MP mode. Follows the plugin/auto-discovery pattern from LMCache#2704. - Generalize csrc/redis → csrc/storage_backends with ConnectorBase<T> template, IStorageConnector interface, and pybind macro - Add NativeConnectorL2Adapter bridging any native connector to L2 - Add RESPL2AdapterConfig with self-registration (plugin pattern) - Refactor ConnectorClientBase as generic base for non-MP mode - Move MockL2AdapterConfig to mock_l2_adapter.py (per LMCache#2704 pattern) - Update __init__.py to use pkgutil auto-discovery Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Refactors the C++ Redis connector into a generic storage backend framework and adds an L2 adapter bridge for MP mode. Follows the plugin/auto-discovery pattern from LMCache#2704. - Generalize csrc/redis → csrc/storage_backends with ConnectorBase<T> template, IStorageConnector interface, and pybind macro - Add NativeConnectorL2Adapter bridging any native connector to L2 - Add RESPL2AdapterConfig with self-registration (plugin pattern) - Refactor ConnectorClientBase as generic base for non-MP mode - Move MockL2AdapterConfig to mock_l2_adapter.py (per LMCache#2704 pattern) - Update __init__.py to use pkgutil auto-discovery

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

Refactors the C++ Redis connector into a generic storage backend framework and adds an L2 adapter bridge for MP mode. Follows the plugin/auto-discovery pattern from LMCache#2704. - Generalize csrc/redis → csrc/storage_backends with ConnectorBase<T> template, IStorageConnector interface, and pybind macro - Add NativeConnectorL2Adapter bridging any native connector to L2 - Add RESPL2AdapterConfig with self-registration (plugin pattern) - Refactor ConnectorClientBase as generic base for non-MP mode - Move MockL2AdapterConfig to mock_l2_adapter.py (per LMCache#2704 pattern) - Update __init__.py to use pkgutil auto-discovery

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

chunxiaozheng

LGTM!

ApostaC

LGTM!

…LMCache#2704) * Add filesystem-backed L2 adapter with auto-discovery plugin mechanism Signed-off-by: baoloongmao <baoloongmao@tencent.com> Signed-off-by: Aaron Wu <aaron.wu@dell.com>

…LMCache#2704) * Add filesystem-backed L2 adapter with auto-discovery plugin mechanism Signed-off-by: baoloongmao <baoloongmao@tencent.com>

maobaolong requested a review from ApostaC March 6, 2026 12:10

gemini-code-assist Bot reviewed Mar 6, 2026

View reviewed changes

maobaolong mentioned this pull request Mar 6, 2026

[Southbound]: Create a Native Protocol for MP and non-MP #2642

Merged

maobaolong force-pushed the l2_fs_adapter branch 2 times, most recently from 8f8a4bd to 963902c Compare March 9, 2026 11:19

sammshen requested a review from deng451e March 10, 2026 01:05

chunxiaozheng reviewed Mar 10, 2026

View reviewed changes

Comment thread lmcache/v1/distributed/l2_adapters/fs_l2_adapter.py Outdated

maobaolong force-pushed the l2_fs_adapter branch from 3fa242c to 97c5757 Compare March 10, 2026 07:09

sammshen reviewed Mar 10, 2026

View reviewed changes

ApostaC requested changes Mar 10, 2026

View reviewed changes

ApostaC reviewed Mar 10, 2026

View reviewed changes

ApostaC mentioned this pull request Mar 10, 2026

Plugin L2 Adapter Framework for MP Mode #2715

Merged

2 tasks

maobaolong requested review from ApostaC, chunxiaozheng and sammshen March 11, 2026 03:55

ApostaC reviewed Mar 11, 2026

View reviewed changes

maobaolong force-pushed the l2_fs_adapter branch from 677e756 to 07d0abc Compare March 12, 2026 03:05

sammshen reviewed Mar 12, 2026

View reviewed changes

sammshen approved these changes Mar 12, 2026

View reviewed changes

Add filesystem-backed L2 adapter with auto-discovery plugin mechanism

0bd4e82

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

maobaolong added 4 commits March 12, 2026 16:04

Address gemini's comment

a30cd40

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

Fix style

77007e4

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

Address comment from yihua.

85aa73b

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

fix test

3412751

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

Address comments from chunxiao

94d509f

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

maobaolong force-pushed the l2_fs_adapter branch from e1f8407 to 94d509f Compare March 12, 2026 08:16

maobaolong requested a review from ApostaC March 12, 2026 08:17

maobaolong added 2 commits March 12, 2026 16:56

Add missing files

a84be67

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

Add missing docstring

79a618d

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

maobaolong enabled auto-merge (squash) March 12, 2026 11:17

chunxiaozheng approved these changes Mar 12, 2026

View reviewed changes

github-actions Bot added the full Run comprehensive tests on this PR label Mar 12, 2026

ApostaC disabled auto-merge March 12, 2026 17:58

github-actions Bot removed the full Run comprehensive tests on this PR label Mar 12, 2026

ApostaC enabled auto-merge (squash) March 12, 2026 17:58

github-actions Bot added the full Run comprehensive tests on this PR label Mar 12, 2026

ApostaC approved these changes Mar 12, 2026

View reviewed changes

ApostaC merged commit 0d23dbd into LMCache:dev Mar 12, 2026
29 of 31 checks passed

		_FILE_BACKENDS = ("GDS", "GDS_MT", "POSIX", "HF3FS")


		class NixlStoreL2AdapterConfig(L2AdapterConfigBase):

	register_l2_adapter_factory("mock", lambda config, **kwargs: MockL2Adapter(config))
	register_l2_adapter_factory("mock", lambda config: MockL2Adapter(config))

		if l1_memory_desc is not None:
		config.l1_memory_desc = l1_memory_desc # type: ignore[attr-defined]

	L2AdapterFactory = Callable[["L2AdapterConfigBase"], "L2AdapterInterface"]
	L2AdapterFactory = Callable[["L2AdapterConfigBase", "L1MemoryDesc"], "L2AdapterInterface"]

Conversation

maobaolong commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Mar 6, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maobaolong commented Mar 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

maobaolong commented Mar 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ApostaC left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ApostaC left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ApostaC commented Mar 10, 2026

Uh oh!

maobaolong commented Mar 11, 2026

Uh oh!

ApostaC left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maobaolong commented Mar 6, 2026 •

edited

Loading

sammshen commented Mar 12, 2026 •

edited

Loading