[feat] add observability stack to MP mode by royyhuang · Pull Request #2638 · LMCache/LMCache

royyhuang · 2026-02-25T01:02:56Z

This PR introduces observability stack for our multiprocess mode via newly introduced EventListener class.

Special notes for your reviewers:

If applicable:

this PR contains user facing changes - docs added
this PR contains unit tests

Signed-off-by: royyhuang <roy.y.huang@gmail.com>

gemini-code-assist · 2026-02-25T01:03:20Z

Summary of Changes

Hello @royyhuang, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates an observability stack into the multi-process (MP) mode, specifically targeting the distributed storage components. It establishes a clear listener pattern for various storage events, enabling the collection of metrics and insights into cache operations. The changes lay the groundwork for monitoring the performance and behavior of the L1 cache and overall storage manager interactions by providing granular event notifications and a dedicated Prometheus controller.

Highlights

New Listener Interface: Introduced a new StorageManagerListener abstract base class to standardize event notifications across storage layers, providing detailed success and failure key lists for each event.
L1 Manager Listener Refactoring: Refactored L1ManagerListener methods by adding an on_l1_ prefix to clearly distinguish L1-specific events and updated L1Manager to call these new prefixed methods.
Prometheus Controller Integration: Implemented PrometheusController to initiate and manage the collection of storage-related metrics, registering a StorageStatsListener to capture L1 manager events.
Storage Manager Observability: Updated StorageManager to register and notify StorageManagerListener instances for key lifecycle events, including reserve_write, finish_write, read_prefetched_results, and submit_prefetch_task.
New Observability Dataclasses: Added new dataclasses (MPServerStats, StorageManagerStats, VLLMIntegratorStats) to structure observability data for various components.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

.gitignore
- Added uv.lock to ignore uv lock files.
- Added several .md files for coding agent instructions to be ignored.
lmcache/v1/distributed/eviction.py
- Renamed L1 manager listener methods to include an on_l1_ prefix.
- Added no-op implementations for new StorageManagerListener methods.
lmcache/v1/distributed/internal_api.py
- Defined a new StorageManagerListener abstract base class with methods for read/write/prefetch events, including success/failure keys.
- Modified L1ManagerListener to inherit from StorageManagerListener.
- Renamed all L1ManagerListener abstract methods to include an on_l1_ prefix.
- Added a placeholder L2ManagerListener class.
lmcache/v1/distributed/l1_manager.py
- Updated all calls to L1ManagerListener methods to use the new on_l1_ prefixed names.
lmcache/v1/distributed/observability/logger/storage_stats_logger.py
- Added StorageStatsListener class, inheriting from L1ManagerListener.
lmcache/v1/distributed/observability/prometheus_controller.py
- Added PrometheusController class, which initializes and registers a StorageStatsListener with the L1 manager, and runs a periodic monitoring thread.
lmcache/v1/distributed/observability/stats/mp_server_stats.py
- Added MPServerStats dataclass.
lmcache/v1/distributed/observability/stats/storage_manager_stats.py
- Added StorageManagerStats dataclass.
lmcache/v1/distributed/observability/stats/vllm_integrator_stats.py
- Added VLLMIntegratorStats dataclass.
lmcache/v1/distributed/storage_manager.py
- Imported StorageManagerListener.
- Added a list to store registered StorageManagerListener instances.
- Implemented register_listener method to add new listeners.
- Integrated calls to on_sm_reserved_write, on_sm_write_finished, on_sm_read_prefetched_finished, and on_sm_read_prefetched to notify registered listeners about storage events, distinguishing between successful and failed keys.

Activity

No human activity recorded for this pull request.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces an observability stack for the multi-process mode by adding a listener pattern to track storage events. The changes are well-structured, introducing StorageManagerListener and updating related components. However, there are a few critical issues in the implementation of the new listener interfaces. Specifically, StorageStatsListener is missing implementations for abstract methods, and EvictionPolicy has incorrect method signatures for the new listener methods, both of which will cause runtime errors. There are also some minor issues with a misleading comment and an incomplete loop in the new PrometheusController.

Signed-off-by: royyhuang <roy.y.huang@gmail.com>

ApostaC

Otherwise LGTM

Signed-off-by: royyhuang <roy.y.huang@gmail.com>

ApostaC

LGTM. Some future tasks:

Move the observability module to something like lmcache/v1/mp_observability (and the configuration).
Update the Prometheus controller implementation to enable a more flexible way to initialize the loggers (see the detail below)
More stats loggers for L2, mp servers, vLLM integrations

ApostaC · 2026-02-27T18:47:54Z

+    def __init__(
+        self,
+        storage_manager: "StorageManager",
+        l1_manager: L1Manager,
+        log_interval: float,
+    ):
+        super().__init__(storage_manager, l1_manager)
+
+        self._log_interval = log_interval
+        self.all_loggers: List[PrometheusLogger] = []
+
+        self.sm_stats_logger: StorageManagerStatsLogger = StorageManagerStatsLogger()
+        self.get_storage_manager().register_listener(self.sm_stats_logger)
+        self.all_loggers.append(self.sm_stats_logger)
+
+        self.l1_stats_logger: L1ManagerStatsLogger = L1ManagerStatsLogger()
+        self.get_l1_manager().register_listener(self.l1_stats_logger)
+        self.all_loggers.append(self.l1_stats_logger)


In the future, we can let each module (L1Manager, L2Manager, etc) to intialize their own loggers, and register those loggers into the Prometheus controller.

The Prometheus controller should be a singleton, and can be accessed by other components for logger registration.

In this case, the Prometheus controller itself does not storage_manager and l1_manager during intialization

* [feat] adding storage manager listener and observability skeleton Signed-off-by: royyhuang <roy.y.huang@gmail.com> * remove local uv.lock and coding agent markdown Signed-off-by: royyhuang <roy.y.huang@gmail.com> * [feat] add prometheus logger interface and prometheus controller Signed-off-by: royyhuang <roy.y.huang@gmail.com> * remove misleading no ops in eviction policy Signed-off-by: royyhuang <roy.y.huang@gmail.com> * fix code quality check and prometheus naming conflicts Signed-off-by: royyhuang <roy.y.huang@gmail.com> * refactor storage manager stats logger into stats logger per storage tier Signed-off-by: royyhuang <roy.y.huang@gmail.com> * address comments Signed-off-by: royyhuang <roy.y.huang@gmail.com> * fix tests Signed-off-by: royyhuang <roy.y.huang@gmail.com> --------- Signed-off-by: royyhuang <roy.y.huang@gmail.com>

* [feat] adding storage manager listener and observability skeleton Signed-off-by: royyhuang <roy.y.huang@gmail.com> * remove local uv.lock and coding agent markdown Signed-off-by: royyhuang <roy.y.huang@gmail.com> * [feat] add prometheus logger interface and prometheus controller Signed-off-by: royyhuang <roy.y.huang@gmail.com> * remove misleading no ops in eviction policy Signed-off-by: royyhuang <roy.y.huang@gmail.com> * fix code quality check and prometheus naming conflicts Signed-off-by: royyhuang <roy.y.huang@gmail.com> * refactor storage manager stats logger into stats logger per storage tier Signed-off-by: royyhuang <roy.y.huang@gmail.com> * address comments Signed-off-by: royyhuang <roy.y.huang@gmail.com> * fix tests Signed-off-by: royyhuang <roy.y.huang@gmail.com> --------- Signed-off-by: royyhuang <roy.y.huang@gmail.com> Signed-off-by: Ofer Kiselov Nahman <ofer.kiselovnahman@weka.io>

* [feat] adding storage manager listener and observability skeleton Signed-off-by: royyhuang <roy.y.huang@gmail.com> * remove local uv.lock and coding agent markdown Signed-off-by: royyhuang <roy.y.huang@gmail.com> * [feat] add prometheus logger interface and prometheus controller Signed-off-by: royyhuang <roy.y.huang@gmail.com> * remove misleading no ops in eviction policy Signed-off-by: royyhuang <roy.y.huang@gmail.com> * fix code quality check and prometheus naming conflicts Signed-off-by: royyhuang <roy.y.huang@gmail.com> * refactor storage manager stats logger into stats logger per storage tier Signed-off-by: royyhuang <roy.y.huang@gmail.com> * address comments Signed-off-by: royyhuang <roy.y.huang@gmail.com> * fix tests Signed-off-by: royyhuang <roy.y.huang@gmail.com> --------- Signed-off-by: royyhuang <roy.y.huang@gmail.com>

* [feat] adding storage manager listener and observability skeleton Signed-off-by: royyhuang <roy.y.huang@gmail.com> * remove local uv.lock and coding agent markdown Signed-off-by: royyhuang <roy.y.huang@gmail.com> * [feat] add prometheus logger interface and prometheus controller Signed-off-by: royyhuang <roy.y.huang@gmail.com> * remove misleading no ops in eviction policy Signed-off-by: royyhuang <roy.y.huang@gmail.com> * fix code quality check and prometheus naming conflicts Signed-off-by: royyhuang <roy.y.huang@gmail.com> * refactor storage manager stats logger into stats logger per storage tier Signed-off-by: royyhuang <roy.y.huang@gmail.com> * address comments Signed-off-by: royyhuang <roy.y.huang@gmail.com> * fix tests Signed-off-by: royyhuang <roy.y.huang@gmail.com> --------- Signed-off-by: royyhuang <roy.y.huang@gmail.com> Signed-off-by: shaoxiawjc <wjc2800@163.com>

* [feat] adding storage manager listener and observability skeleton Signed-off-by: royyhuang <roy.y.huang@gmail.com> * remove local uv.lock and coding agent markdown Signed-off-by: royyhuang <roy.y.huang@gmail.com> * [feat] add prometheus logger interface and prometheus controller Signed-off-by: royyhuang <roy.y.huang@gmail.com> * remove misleading no ops in eviction policy Signed-off-by: royyhuang <roy.y.huang@gmail.com> * fix code quality check and prometheus naming conflicts Signed-off-by: royyhuang <roy.y.huang@gmail.com> * refactor storage manager stats logger into stats logger per storage tier Signed-off-by: royyhuang <roy.y.huang@gmail.com> * address comments Signed-off-by: royyhuang <roy.y.huang@gmail.com> * fix tests Signed-off-by: royyhuang <roy.y.huang@gmail.com> --------- Signed-off-by: royyhuang <roy.y.huang@gmail.com> Signed-off-by: Aaron Wu <aaron.wu@dell.com>

* [feat] adding storage manager listener and observability skeleton Signed-off-by: royyhuang <roy.y.huang@gmail.com> * remove local uv.lock and coding agent markdown Signed-off-by: royyhuang <roy.y.huang@gmail.com> * [feat] add prometheus logger interface and prometheus controller Signed-off-by: royyhuang <roy.y.huang@gmail.com> * remove misleading no ops in eviction policy Signed-off-by: royyhuang <roy.y.huang@gmail.com> * fix code quality check and prometheus naming conflicts Signed-off-by: royyhuang <roy.y.huang@gmail.com> * refactor storage manager stats logger into stats logger per storage tier Signed-off-by: royyhuang <roy.y.huang@gmail.com> * address comments Signed-off-by: royyhuang <roy.y.huang@gmail.com> * fix tests Signed-off-by: royyhuang <roy.y.huang@gmail.com> --------- Signed-off-by: royyhuang <roy.y.huang@gmail.com>

royyhuang added 2 commits February 25, 2026 00:56

[feat] adding storage manager listener and observability skeleton

bda337e

Signed-off-by: royyhuang <roy.y.huang@gmail.com>

remove local uv.lock and coding agent markdown

f037478

Signed-off-by: royyhuang <roy.y.huang@gmail.com>

royyhuang added the mp_mode label Feb 25, 2026

royyhuang marked this pull request as draft February 25, 2026 01:03

gemini-code-assist Bot reviewed Feb 25, 2026

View reviewed changes

Comment thread lmcache/v1/distributed/eviction.py Outdated

Comment thread lmcache/v1/distributed/observability/logger/storage_stats_logger.py Outdated

Comment thread lmcache/v1/distributed/eviction.py Outdated

Comment thread lmcache/v1/distributed/observability/prometheus_controller.py Outdated

[feat] add prometheus logger interface and prometheus controller

7e0015c

Signed-off-by: royyhuang <roy.y.huang@gmail.com>

royyhuang marked this pull request as ready for review February 25, 2026 23:15

royyhuang added 3 commits February 25, 2026 23:18

remove misleading no ops in eviction policy

f695b33

Signed-off-by: royyhuang <roy.y.huang@gmail.com>

fix code quality check and prometheus naming conflicts

e5b657a

Signed-off-by: royyhuang <roy.y.huang@gmail.com>

refactor storage manager stats logger into stats logger per storage tier

aa1a162

Signed-off-by: royyhuang <roy.y.huang@gmail.com>

ApostaC reviewed Feb 26, 2026

View reviewed changes

royyhuang added 3 commits February 27, 2026 00:16

address comments

36627c1

Signed-off-by: royyhuang <roy.y.huang@gmail.com>

merge upstream

918ac02

Signed-off-by: royyhuang <roy.y.huang@gmail.com>

fix tests

919c7c7

Signed-off-by: royyhuang <roy.y.huang@gmail.com>

ApostaC added the full Run comprehensive tests on this PR label Feb 27, 2026

Merge branch 'dev' into dev

0d7946b

royyhuang enabled auto-merge (squash) February 27, 2026 08:53

ApostaC approved these changes Feb 27, 2026

View reviewed changes

Oasis-Git approved these changes Feb 27, 2026

View reviewed changes

royyhuang merged commit 2ff6f2b into LMCache:dev Feb 27, 2026
25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat] add observability stack to MP mode#2638

[feat] add observability stack to MP mode#2638
royyhuang merged 10 commits intoLMCache:devfrom
royyhuang:dev

royyhuang commented Feb 25, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Feb 25, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ApostaC left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ApostaC left a comment •

edited

Loading

Uh oh!

ApostaC Feb 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

royyhuang commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Feb 25, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ApostaC left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ApostaC left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ApostaC Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

royyhuang commented Feb 25, 2026 •

edited

Loading

ApostaC left a comment •

edited

Loading