Skip to content

[MP] Lazy start heartbeat thread when first req coming#2943

Merged
maobaolong merged 2 commits intoLMCache:devfrom
maobaolong:lazy_start_hbt
Apr 11, 2026
Merged

[MP] Lazy start heartbeat thread when first req coming#2943
maobaolong merged 2 commits intoLMCache:devfrom
maobaolong:lazy_start_hbt

Conversation

@maobaolong
Copy link
Copy Markdown
Collaborator

@maobaolong maobaolong commented Apr 3, 2026

What this PR does / why we need it:

I'm sorry #2798 is not good enough, the current PR make it lazy start heartbeat thread when first req coming.

Special notes for your reviewers:

If applicable:

  • this PR contains user facing changes - docs added
  • this PR contains unit tests

Note

Medium Risk
Changes the timing of heartbeat thread startup in the vLLM multiprocess adapter, which can affect health detection and degraded-mode behavior during startup and early requests. Concurrency/thread-lifecycle adjustments may introduce subtle race or ordering issues.

Overview
Defers starting the LMCache HeartbeatThread in LMCacheMPWorkerAdapter until the first submit_store_request or submit_retrieve_request, instead of starting it immediately after register_kv_caches completes.

Replaces the eager _start_heartbeat call with an idempotent _ensure_heartbeat_started helper and updates comments to clarify the new “start on first use” behavior once vLLM initialization (model load, KV allocation, warmup/CUDA graph capture) is complete.

Written by Cursor Bugbot for commit 2775a48. This will update automatically on new commits. Configure here.

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements lazy initialization for the heartbeat thread in the VLLMMultiProcessAdapter, moving its start from the registration phase to the first store or retrieve request. A review comment identifies an inaccuracy in the class constructor's documentation regarding which method triggers the heartbeat, providing a suggestion to align the comment with the actual implementation.

Comment thread lmcache/integration/vllm/vllm_multi_process_adapter.py Outdated
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
@maobaolong
Copy link
Copy Markdown
Collaborator Author

@cursor review

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

@maobaolong maobaolong added the full Run comprehensive tests on this PR label Apr 3, 2026
@maobaolong
Copy link
Copy Markdown
Collaborator Author

@sammshen @chunxiaozheng Would you like to take a look at this ? I've tested locally.

Copy link
Copy Markdown
Contributor

@sammshen sammshen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense to me.

cc @ApostaC

Copy link
Copy Markdown
Collaborator

@chunxiaozheng chunxiaozheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

maobaolong added a commit to maobaolong/LMCache that referenced this pull request Apr 8, 2026
…che#2943

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
@maobaolong maobaolong merged commit 0e0dc12 into LMCache:dev Apr 11, 2026
35 checks passed
Oasis-Git pushed a commit to Oasis-Git/LMCache that referenced this pull request Apr 13, 2026
* [MP] Lazy start heartbeat thread when first req coming

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

* Update comment

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

---------

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
ftian1 pushed a commit to ftian1/LMCache that referenced this pull request Apr 20, 2026
* [MP] Lazy start heartbeat thread when first req coming

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

* Update comment

Signed-off-by: baoloongmao <baoloongmao@tencent.com>

---------

Signed-off-by: baoloongmao <baoloongmao@tencent.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full Run comprehensive tests on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants