[MP] Support buffer only mode for MP mode by maobaolong · Pull Request #2760 · LMCache/LMCache

maobaolong · 2026-03-12T13:59:17Z

What this PR does / why we need it:

L1_SIZE_GB=0.03
L2_JSON='{"type": "mock", "max_size_gb": 1.0, "mock_bandwidth_gb": 20.0}'
python3 -m lmcache.v1.multiprocess.server \
    --host localhost --port 15556 \
    --chunk-size 256 --l1-size-gb $L1_SIZE_GB \
    --eviction-policy noop --max-workers 1 \
    --l2-adapter "$L2_JSON"

Special notes for your reviewers:

If applicable:

this PR contains user facing changes - docs added
this PR contains unit tests

gemini-code-assist · 2026-03-12T13:59:43Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new "buffer-only" operational mode for the LMCache's distributed storage system. This mode is specifically designed to optimize performance when the L1 cache acts purely as a write buffer, forwarding data to L2 storage. By implementing a NoOpEvictionPolicy and a BufferOnlyStorePolicy, the system avoids the overhead of unnecessary key tracking and immediately clears L1 after successful L2 writes, streamlining the data flow and improving efficiency in such scenarios.

Highlights

New Eviction Policy: Introduced a NoOpEvictionPolicy that performs no tracking and never evicts, specifically designed for scenarios where L1 acts purely as a write buffer.
New Store Policy: Implemented a BufferOnlyStorePolicy which ensures data is stored to L2 and then immediately removed from L1, optimizing buffer-only operations.
Dynamic Policy Selection: Updated the StorageManager to dynamically select between DefaultStorePolicy and the new BufferOnlyStorePolicy based on the configured eviction policy.
Configuration Updates: Extended EvictionConfig and command-line arguments to support the new noop eviction policy, allowing users to enable this mode.
Comprehensive Testing: Added new unit tests to validate the functionality of the buffer-only mode, including L1 cleanup after L2 store and the behavior of the no-op eviction policy.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

lmcache/v1/distributed/config.py
- The eviction_policy field in EvictionConfig was updated to accept "noop" as a valid option.
- The command-line argument --eviction-policy now includes "noop" as a selectable choice.
lmcache/v1/distributed/eviction_policy/init.py
- NoOpEvictionPolicy was imported and added to the module's __all__ export list.
lmcache/v1/distributed/eviction_policy/factory.py
- NoOpEvictionPolicy was imported.
- The CreateEvictionPolicy function was modified to return a NoOpEvictionPolicy instance when "noop" is specified in the configuration.
lmcache/v1/distributed/eviction_policy/noop.py
- A new file was created defining the NoOpEvictionPolicy class, which implements the EvictionPolicy interface with empty methods, effectively performing no eviction actions.
lmcache/v1/distributed/storage_controllers/store_policy.py
- A new BufferOnlyStorePolicy class was added, which dictates that all keys should be stored to all L2 adapters and then immediately deleted from L1.
lmcache/v1/distributed/storage_manager.py
- BufferOnlyStorePolicy and StorePolicy were imported.
- The StorageManager's constructor was modified to conditionally instantiate either BufferOnlyStorePolicy or DefaultStorePolicy for its _store_controller based on the configured eviction_policy.
tests/v1/distributed/test_store_controller.py
- NoOpEvictionPolicy and BufferOnlyStorePolicy were imported.
- A new test class TestBufferOnlyMode was added, containing tests to verify that L1 is cleaned after L2 store, that the no-op eviction policy never evicts, and that multiple keys are correctly handled in buffer-only mode.

Activity

No specific activity (comments, reviews, progress) has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a "buffer only" mode for the multiprocess (MP) setup, which is a valuable optimization. The implementation, including the new noop eviction policy and BufferOnlyStorePolicy, is logical and well-structured. The accompanying tests are also comprehensive. I have a few suggestions to enhance maintainability and test clarity.

DongDongJu · 2026-03-12T14:53:29Z

this means like support staging mode like this #2370? for mp?

maobaolong · 2026-03-13T01:51:17Z

@DongDongJu Yeah, they are similar.

ApostaC

Thanks @maobaolong ! LGTM in general. Can you also review #2773 first so we don't need to hard-code the store policy in the code.

ApostaC · 2026-03-14T00:58:56Z

+        if config.eviction_config.eviction_policy == "noop":
+            store_policy = BufferOnlyStorePolicy()
+        else:
+            store_policy = DefaultStorePolicy()


We are introducing the new config: --l2-store-policy and --l2-prefetch-policy in PR #2773. Once that's merged, we can use that instead of hard-coding here.

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

ApostaC

Otherwise LGTM!

ApostaC · 2026-03-15T03:34:17Z

+
+
 register_store_policy("default", DefaultStorePolicy)
+register_store_policy("noop", BufferOnlyStorePolicy)


nit: can we set the new store policy type to "skip_l1"? Sounds more informative than "noop"

ApostaC · 2026-03-15T03:36:05Z

@maobaolong Oh, forget one extra thing: please update the user-facing docs (in docs/src/mp) for the newly introduced command line arguments

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

chunxiaozheng

lgtm

maobaolong · 2026-03-16T08:23:40Z

+.. code-block:: bash
+
+    --eviction-policy noop \
+    --l2-store-policy noop \


This should be skip_l1

* [MP] Support buffer only mode for MP mode Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Address comments from gemini bot Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Fix code quality issue Signed-off-by: baoloongmao <baoloongmao@tencent.com> * update docs Signed-off-by: baoloongmao <baoloongmao@tencent.com> --------- Signed-off-by: baoloongmao <baoloongmao@tencent.com>

* [MP] Support buffer only mode for MP mode Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Address comments from gemini bot Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Fix code quality issue Signed-off-by: baoloongmao <baoloongmao@tencent.com> * update docs Signed-off-by: baoloongmao <baoloongmao@tencent.com> --------- Signed-off-by: baoloongmao <baoloongmao@tencent.com> Signed-off-by: Aaron Wu <aaron.wu@dell.com>

* [MP] Support buffer only mode for MP mode Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Address comments from gemini bot Signed-off-by: baoloongmao <baoloongmao@tencent.com> * Fix code quality issue Signed-off-by: baoloongmao <baoloongmao@tencent.com> * update docs Signed-off-by: baoloongmao <baoloongmao@tencent.com> --------- Signed-off-by: baoloongmao <baoloongmao@tencent.com>

gemini-code-assist Bot reviewed Mar 12, 2026

View reviewed changes

Comment thread lmcache/v1/distributed/storage_controllers/store_policy.py Outdated

Comment thread tests/v1/distributed/test_store_controller.py Outdated

Comment thread tests/v1/distributed/test_store_controller.py Outdated

maobaolong added the full Run comprehensive tests on this PR label Mar 13, 2026

ApostaC reviewed Mar 14, 2026

View reviewed changes

maobaolong added 2 commits March 15, 2026 10:11

[MP] Support buffer only mode for MP mode

382732f

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

Address comments from gemini bot

53e2292

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

maobaolong force-pushed the l2DisableHotCache branch from f3c74c1 to 53e2292 Compare March 15, 2026 02:12

maobaolong requested a review from ApostaC March 15, 2026 02:15

Fix code quality issue

36ccf69

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

ApostaC approved these changes Mar 15, 2026

View reviewed changes

update docs

2f20fe2

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

chunxiaozheng approved these changes Mar 15, 2026

View reviewed changes

chunxiaozheng enabled auto-merge (squash) March 15, 2026 04:38

chunxiaozheng merged commit 15cc709 into LMCache:dev Mar 15, 2026
26 of 29 checks passed

maobaolong commented Mar 16, 2026

View reviewed changes



		register_store_policy("default", DefaultStorePolicy)
		register_store_policy("noop", BufferOnlyStorePolicy)

Conversation

maobaolong commented Mar 12, 2026

Uh oh!

gemini-code-assist Bot commented Mar 12, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DongDongJu commented Mar 12, 2026

Uh oh!

maobaolong commented Mar 13, 2026

Uh oh!

ApostaC left a comment

Choose a reason for hiding this comment

Uh oh!

ApostaC Mar 14, 2026

Choose a reason for hiding this comment

Uh oh!

ApostaC left a comment

Choose a reason for hiding this comment

Uh oh!

ApostaC Mar 15, 2026

Choose a reason for hiding this comment

Uh oh!

maobaolong Mar 15, 2026

Choose a reason for hiding this comment

Uh oh!

ApostaC commented Mar 15, 2026

Uh oh!

chunxiaozheng left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

maobaolong Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants