Add argument --prefetch-max-in-flight to fix hardcode by maobaolong · Pull Request #2789 · LMCache/LMCache

maobaolong · 2026-03-16T12:16:27Z

What this PR does / why we need it:

Special notes for your reviewers:

If applicable:

this PR contains user facing changes - docs added
this PR contains unit tests

Note

Low Risk
Low risk: exposes an existing prefetch concurrency limit as a CLI/config option and wires it through to PrefetchController, with default behavior unchanged. Main risk is operational misconfiguration (setting the limit too high could increase L1 memory pressure).

Overview
Adds a new --l2-prefetch-max-in-flight CLI flag (default 8) to control how many concurrent L2 prefetch/load requests the PrefetchController can issue, replacing the previously fixed limit.

Wires the new setting through StorageManagerConfig into StorageManager when constructing PrefetchController, and updates multiprocess docs/examples to document the flag and its memory/throughput tradeoff.

^{Written by Cursor Bugbot for commit 88039a4. This will update automatically on new commits. Configure here.}

gemini-code-assist · 2026-03-16T12:16:47Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the flexibility of the LMCache system by externalizing the control over prefetch concurrency. By introducing a command-line argument and a corresponding configuration field, it allows administrators to dynamically adjust the maximum number of in-flight prefetch requests, moving away from a fixed, hardcoded limit. This change provides better operational control and resource management for distributed storage operations.

Highlights

Configurability: Introduced a new configuration option, --prefetch-max-in-flight, allowing users to specify the maximum number of concurrent prefetch requests.
Hardcoded Value Removal: Replaced a previously hardcoded max_in_flight value within the PrefetchController with the new configurable argument.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

lmcache/v1/distributed/config.py
- Added prefetch_max_in_flight attribute with a default value to the StorageManagerConfig class.
- Included --prefetch-max-in-flight as a new command-line argument in add_storage_manager_args.
- Updated parse_args_to_config to pass the new prefetch_max_in_flight argument to the configuration.
lmcache/v1/distributed/storage_controllers/prefetch_controller.py
- Removed a TODO comment related to the max_in_flight parameter, as it is now configurable.
- Ensured the _max_in_flight attribute is initialized directly from the constructor parameter.
lmcache/v1/distributed/storage_manager.py
- Modified the __init__ method to pass the newly configured prefetch_max_in_flight value to the PrefetchController.

Activity

maobaolong opened this pull request to add a configurable argument for prefetch concurrency.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a command-line argument --prefetch-max-in-flight to make the maximum number of in-flight prefetch requests configurable, removing a hardcoded value. The changes are well-implemented across the configuration, storage manager, and prefetch controller. My feedback focuses on improving the naming consistency of the new argument to align with existing patterns in the codebase.

ApostaC

LGTM!

ApostaC · 2026-03-18T00:23:55Z

Please update the docs in docs/src/mp to reflect the new command line interface. Thanks!

Signed-off-by: baoloongmao <baoloongmao@tencent.com> LMCache#2789

chunxiaozheng

LGTM!

ApostaC · 2026-03-19T17:24:42Z

@maobaolong can we add some docs in docs/src/mp for the new cli interface? Thanks!

maobaolong · 2026-03-23T06:23:13Z

@ApostaC Sure, thanks for remind me.

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

* Add argument --prefetch-max-in-flight to fix hardcode Signed-off-by: baoloongmao <baoloongmao@tencent.com> * add l2 prefix Signed-off-by: baoloongmao <baoloongmao@tencent.com> --------- Signed-off-by: baoloongmao <baoloongmao@tencent.com>

maobaolong requested a review from ApostaC March 16, 2026 12:16

gemini-code-assist Bot reviewed Mar 16, 2026

View reviewed changes

Comment thread lmcache/v1/distributed/config.py

Comment thread lmcache/v1/distributed/config.py Outdated

ApostaC approved these changes Mar 18, 2026

View reviewed changes

yoo-kumaneko pushed a commit to yoo-kumaneko/LMCache that referenced this pull request Mar 18, 2026

Backport update: Add argument --prefetch-max-in-flight to fix hardcode

78f52ab

Signed-off-by: baoloongmao <baoloongmao@tencent.com> LMCache#2789

maobaolong added the full Run comprehensive tests on this PR label Mar 19, 2026

chunxiaozheng approved these changes Mar 19, 2026

View reviewed changes

chunxiaozheng enabled auto-merge (squash) March 19, 2026 09:42

maobaolong disabled auto-merge April 1, 2026 11:15

github-actions Bot removed the full Run comprehensive tests on this PR label Apr 1, 2026

maobaolong added 2 commits April 1, 2026 19:16

Add argument --prefetch-max-in-flight to fix hardcode

07a3723

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

add l2 prefix

f99e984

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

cursor Bot reviewed Apr 1, 2026

View reviewed changes

Comment thread lmcache/v1/distributed/config.py

Add document

88039a4

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

maobaolong enabled auto-merge (squash) April 1, 2026 11:39

github-actions Bot added the full Run comprehensive tests on this PR label Apr 1, 2026

maobaolong force-pushed the max_in_flight_cfg branch from a43ba63 to 88039a4 Compare April 1, 2026 11:40

cursor Bot reviewed Apr 1, 2026

View reviewed changes

Comment thread lmcache/v1/distributed/config.py

maobaolong merged commit 6774d50 into LMCache:dev Apr 1, 2026
34 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add argument --prefetch-max-in-flight to fix hardcode#2789

Add argument --prefetch-max-in-flight to fix hardcode#2789
maobaolong merged 3 commits intoLMCache:devfrom
maobaolong:max_in_flight_cfg

maobaolong commented Mar 16, 2026 •

edited by cursor Bot

Loading

Uh oh!

gemini-code-assist Bot commented Mar 16, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

ApostaC left a comment

Uh oh!

ApostaC commented Mar 18, 2026

Uh oh!

chunxiaozheng left a comment

Uh oh!

ApostaC commented Mar 19, 2026

Uh oh!

maobaolong commented Mar 23, 2026

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

maobaolong commented Mar 16, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Mar 16, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

ApostaC left a comment

Choose a reason for hiding this comment

Uh oh!

ApostaC commented Mar 18, 2026

Uh oh!

chunxiaozheng left a comment

Choose a reason for hiding this comment

Uh oh!

ApostaC commented Mar 19, 2026

Uh oh!

maobaolong commented Mar 23, 2026

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

maobaolong commented Mar 16, 2026 •

edited by cursor Bot

Loading