[MP] Fault Tolerance CI by Oasis-Git · Pull Request #2764 · LMCache/LMCache

Oasis-Git · 2026-03-13T01:52:17Z

What this PR does / why we need it:

Special notes for your reviewers:

If applicable:

this PR contains user facing changes - docs added
this PR contains unit tests

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

Signed-off-by: yuweia <ayw.sirius19@gmail.com>

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

…branch) Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

gemini-code-assist · 2026-03-13T01:52:34Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves the robustness of the LMCache system by introducing a dedicated fault tolerance test within the CI pipeline. It also refines the vLLM integration to ensure that request completion is accurately managed, even in the event of LMCache server disruptions, thereby enhancing the overall reliability and stability of the caching mechanism.

Highlights

New Fault Tolerance CI Test: A new Buildkite CI script has been introduced to test the fault tolerance of LMCache. This test simulates LMCache server failure during a vLLM benchmark and verifies that vLLM requests still complete successfully.
vLLM Adapter Robustness Improvements: The vLLM multi-process adapter has been enhanced to prevent re-reporting of finished request IDs and to correctly handle scenarios where LMCache becomes unavailable mid-request, ensuring accurate tracking of request completion.
CI Script Integration: The main multiprocessing test script now incorporates the new fault tolerance test as a critical step, replacing a previous LMCache status check, and removes an unused environment variable.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

.buildkite/scripts/multiprocessing-test/run-fault-tolerance.sh
- Added a new shell script to implement a fault tolerance test for LMCache, which involves killing the LMCache container mid-benchmark.
.buildkite/scripts/multiprocessing-test/run-mp-test.sh
- Removed the LMCACHE_HTTP_PORT environment variable export as it is no longer used.
- Replaced the LMCache server status check with the execution of the new fault tolerance test script.
lmcache/integration/vllm/vllm_multi_process_adapter.py
- Added a new _returned_finished set to track request IDs already reported as finished, preventing duplicate reporting.
- Modified _process_finished_stores to utilize the _returned_finished set to avoid re-processing already reported finished requests.
- Updated get_finished to ensure requests with pending retrieves are not also reported as finished sending if they completed without LMCache after the server died.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a fault tolerance test to the CI pipeline, which simulates an LMCache server failure and verifies that vLLM requests can still complete. The changes include a new test script, modifications to the main test runner to incorporate this new test, and fixes in the Python adapter to correctly handle this fault tolerance scenario. My review focuses on improving the robustness and maintainability of the new test script and ensuring consistent error handling. I've identified a potential bug in how JSON payloads are constructed in the shell script and suggested improvements for simplification and consistency. The Python changes for fault tolerance appear correct and well-reasoned.

gemini-code-assist · 2026-03-13T01:54:28Z

+        if ! curl -sf --max-time 120 \
+            "http://localhost:${VLLM_PORT}/v1/completions" \
+            -H "Content-Type: application/json" \
+            -d "{
+                \"model\": \"$MODEL\",
+                \"prompt\": \"Question: What is $i + $i?\\nAnswer:\",
+                \"max_tokens\": 32,
+                \"temperature\": 0
+            }" > /dev/null 2>&1; then
+            echo "Request $i failed - vLLM became unresponsive"
+            exit 1
+        fi


Constructing JSON with string concatenation is fragile and can lead to invalid JSON if variables like $MODEL contain special characters (e.g., quotes). Using a heredoc to create the JSON payload is a more robust and readable approach.

Suggested change

if ! curl -sf --max-time 120 \

"http://localhost:${VLLM_PORT}/v1/completions" \

-H "Content-Type: application/json" \

-d "{

\"model\": \"$MODEL\",

\"prompt\": \"Question: What is $i + $i?\\nAnswer:\",

\"max_tokens\": 32,

\"temperature\": 0

}" > /dev/null 2>&1; then

echo "Request $i failed - vLLM became unresponsive"

exit 1

fi

json_payload=$(cat <<EOF

{

"model": "$MODEL",

"prompt": "Question: What is $i + $i?\\nAnswer:",

"max_tokens": 32,

"temperature": 0

}

EOF

)

if ! curl -sf --max-time 120 \

"http://localhost:${VLLM_PORT}/v1/completions" \

-H "Content-Type: application/json" \

-d "$json_payload" > /dev/null 2>&1; then

echo "Request $i failed - vLLM became unresponsive"

exit 1

fi

gemini-code-assist · 2026-03-13T01:54:29Z

+    docker kill "$LMCACHE_CONTAINER_NAME" 2>/dev/null || true
+    docker rm -f "$LMCACHE_CONTAINER_NAME" 2>/dev/null || true


The docker rm -f command forcefully removes a running container, which includes stopping it. Therefore, the preceding docker kill command is redundant. You can simplify this to a single docker rm -f command.

Suggested change

docker kill "$LMCACHE_CONTAINER_NAME" 2>/dev/null || true

docker rm -f "$LMCACHE_CONTAINER_NAME" 2>/dev/null || true

docker rm -f "$LMCACHE_CONTAINER_NAME" 2>/dev/null || true

gemini-code-assist · 2026-03-13T01:54:29Z

+if ! "$SCRIPT_DIR/run-fault-tolerance.sh"; then
+    echo "❌ fault tolerance test failed"
+    TEST_RESULT=1
+    exit 1
+fi


The script exits immediately with exit 1 upon failure of the fault tolerance test. This is inconsistent with how other test failures are handled in this script (e.g., the run-long-doc-qa.sh test), which only set TEST_RESULT=1 and allow the script to continue. While this is the last test, exiting immediately could prevent any future cleanup steps from running. For consistency and to ensure any final cleanup logic is executed, consider removing exit 1.

Suggested change

if ! "$SCRIPT_DIR/run-fault-tolerance.sh"; then

echo "❌ fault tolerance test failed"

TEST_RESULT=1

exit 1

fi

if ! "$SCRIPT_DIR/run-fault-tolerance.sh"; then

echo "❌ fault tolerance test failed"

TEST_RESULT=1

fi

ApostaC

Need to update k3s as well

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

ApostaC · 2026-03-14T00:57:46Z

@sammshen Please also take a look at this PR!

sammshen

LGTM!

ApostaC

LGTM!

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

* health check Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * lint Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * fix ut Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * fix Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * dev Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * timeout Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * fix ut Signed-off-by: yuweia <ayw.sirius19@gmail.com> * add comment Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * add test Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * Remove fault tolerance CI step (will be added in separate fault-t-ci branch) Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * Rename HEALTH_CHECK to PING, add timeout params, extract helper Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * fix Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * fix ut Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * fix timeout Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * fix Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * ci Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> * add k3s test Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> --------- Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com> Signed-off-by: yuweia <ayw.sirius19@gmail.com> Co-authored-by: Samuel Shen <slshen@tensormesh.ai>

Oasis-Git added 24 commits February 27, 2026 05:50

health check

ab4ef13

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

lint

3bb3467

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

Merge branch 'dev' into health

4ddfb30

fix ut

8ee2c40

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

fix

dc562fa

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

dev

7c8c846

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

merge main

03245ae

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

timeout

16b1e85

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

fix ut

04bbec2

Signed-off-by: yuweia <ayw.sirius19@gmail.com>

Merge branch 'dev' into fault-t

f16bd3c

add comment

91b3d18

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

add test

8e0ad15

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

Merge branch 'dev' into fault-t

3e31aec

Remove fault tolerance CI step (will be added in separate fault-t-ci …

1b23ccd

…branch) Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

Merge branch 'dev' into fault-t

a6a95a1

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

Rename HEALTH_CHECK to PING, add timeout params, extract helper

99a0656

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

fix

248ace9

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

fix ut

f713620

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

Merge branch 'dev' into fault-t

0de8f76

fix timeout

edda88f

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

merge main

2fd8fab

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

fix

b06982d

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

ci

b288833

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

merge main

c5d218d

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

Oasis-Git added the full Run comprehensive tests on this PR label Mar 13, 2026

gemini-code-assist Bot reviewed Mar 13, 2026

View reviewed changes

Oasis-Git changed the title ~~Fault Tolerance CI~~ [MP] Fault Tolerance CI Mar 13, 2026

ApostaC reviewed Mar 13, 2026

View reviewed changes

add k3s test

7312f70

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

Merge branch 'dev' into fault-t

301ddb3

sammshen approved these changes Mar 16, 2026

View reviewed changes

sammshen enabled auto-merge (squash) March 16, 2026 07:05

ApostaC approved these changes Mar 17, 2026

View reviewed changes

ApostaC and others added 3 commits March 23, 2026 15:31

Merge branch 'dev' into fault-t

e4e4700

Merge branch 'dev' into fault-t

68d6def

fix ci

a31e309

Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>

sammshen merged commit cbc9dd8 into LMCache:dev Mar 24, 2026
24 of 25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MP] Fault Tolerance CI#2764

[MP] Fault Tolerance CI#2764
sammshen merged 29 commits intoLMCache:devfrom
Oasis-Git:fault-t

Oasis-Git commented Mar 13, 2026

Uh oh!

gemini-code-assist Bot commented Mar 13, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Mar 13, 2026

Uh oh!

gemini-code-assist Bot Mar 13, 2026

Uh oh!

gemini-code-assist Bot Mar 13, 2026

Uh oh!

ApostaC left a comment

Uh oh!

ApostaC commented Mar 14, 2026

Uh oh!

sammshen left a comment

Uh oh!

ApostaC left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		docker kill "$LMCACHE_CONTAINER_NAME" 2>/dev/null \|\| true
		docker rm -f "$LMCACHE_CONTAINER_NAME" 2>/dev/null \|\| true

Conversation

Oasis-Git commented Mar 13, 2026

Uh oh!

gemini-code-assist Bot commented Mar 13, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

ApostaC left a comment

Choose a reason for hiding this comment

Uh oh!

ApostaC commented Mar 14, 2026

Uh oh!

sammshen left a comment

Choose a reason for hiding this comment

Uh oh!

ApostaC left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants