[Core] [GDS] Improve GDS backend error handling and retry logic by oferki · Pull Request #2675 · LMCache/LMCache

oferki · 2026-03-03T23:36:45Z

What this PR does / why we need it:

Add comprehensive error handling for cuFile operations
Implement retry logic with exponential backoff for allocate/batched_allocate
Improve error reporting with detailed context
Add proper cleanup on allocation failures
Enhance logging for better debugging

Special notes for your reviewers:
Changes are somewhat conservative (mainly try/catch addition). Consider potential failures and how caller would get them while reviewing.

- Add comprehensive error handling for cuFile operations - Implement retry logic with exponential backoff for allocate/batched_allocate - Improve error reporting with detailed context - Add proper cleanup on allocation failures - Enhance logging for better debugging Signed-off-by: Ofer Kiselov Nahman <ofer.kiselovnahman@weka.io>

gemini-code-assist · 2026-03-03T23:37:04Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the reliability and resilience of the GDS backend by integrating robust error handling, retry mechanisms, and improved logging. The changes address potential failures during cuFile operations, memory allocations, and metadata management, ensuring more stable and debuggable cache interactions. It also refines how cache keys are handled in file paths for better compatibility.

Highlights

Enhanced Error Handling: Implemented comprehensive try-except blocks for cuFile operations and metadata reading, improving robustness against file system issues and data corruption.
Retry Mechanism for Allocations: Introduced retry logic with exponential backoff for allocate and batched_allocate methods to handle transient memory allocation failures.
Improved Logging and Error Reporting: Added detailed logging for various failure scenarios, including cuFile write failures, metadata read errors, and allocation issues, providing better debugging context.
Refined Key String Encoding: Switched to urllib.parse.quote and unquote for handling key strings in file paths, replacing a custom replace mechanism and rand_suffix for better URL-safe encoding.
Allocation Failure Cleanup: Added logic to clean up cache entries if metadata write operations fail after a successful GDS write.
Metadata Scan Synchronization: Ensured that the initial metadata scan completes before the GDS backend is closed, preventing potential race conditions.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

lmcache/v1/storage_backend/gds_backend.py
- Imported urllib.parse and uuid, removed random and string imports.
- Removed an unnecessary empty line.
- Updated rand_suffix function to use uuid.uuid4().hex for generating random suffixes.
- Added max_alloc_attempts and alloc_attempt_delay_secs configuration parameters to the init method for controlling retry behavior.
- Removed the self.rand attribute and stored the _scan_metadata coroutine future for later synchronization.
- Modified _scan_metadata_subdir to use urllib.parse.unquote for decoding key strings from filenames.
- Expanded error handling in _try_to_read_metadata to catch FileNotFoundError, PermissionError, OSError, IOError, and general Exception during metadata file reads.
- Removed an assertion for _ in key strings and implemented urllib.parse.quote for encoding key strings into file paths in _key_to_path.
- Wrapped _async_save_bytes_to_disk logic in a try...finally block, added error handling for GDS/cuFile write failures and POSIX metadata write failures, including cache cleanup on metadata write errors.
- Changed a logger.debug call to logger.error when memory allocation fails during synchronous disk loads.
- Changed a logger.info call to logger.debug in batched_get_blocking for thread pool implementation.
- Added comprehensive error handling to _load_gds, including a file size check and logging for CuFile read failures.
- Implemented a retry loop with exponential backoff for the allocate method, based on max_alloc_attempts and alloc_attempt_delay_secs.
- Implemented a retry loop with exponential backoff for the batched_allocate method, based on max_alloc_attempts and alloc_attempt_delay_secs.
- Added logic in the close method to wait for the _scan_metadata_future to complete, with a timeout.

Activity

No human activity (comments, reviews, etc.) has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request significantly improves the error handling and robustness of the GDS backend by adding comprehensive try-except blocks, implementing a retry mechanism, and ensuring proper cleanup. While these changes enhance system resilience, a security vulnerability has been identified: new log statements log the full CacheEngineKey object, which contains a request_configs dictionary that may hold sensitive data, leading to potential information leakage. It is recommended to use the to_string() method of the key object for sanitized logging. Additionally, there are suggestions to further improve atomicity and code maintainability.

Signed-off-by: Ofer Kiselov Nahman <ofer.kiselovnahman@weka.io>

…into gds_error_handling_clean

…r handling Signed-off-by: Ofer Kiselov Nahman <ofer.kiselovnahman@weka.io>

Signed-off-by: Ofer Kiselov Nahman <ofer.kiselovnahman@weka.io>

oferki · 2026-03-20T17:55:36Z

Hi @sammshen @DongDongJu @deng451e
Will appreciate your review of this PR
Many thanks!

sammshen

LGTM!

deng451e

LGTM!

…che#2675) * Improve GDS backend error handling and retry logic - Add comprehensive error handling for cuFile operations - Implement retry logic with exponential backoff for allocate/batched_allocate - Improve error reporting with detailed context - Add proper cleanup on allocation failures - Enhance logging for better debugging Signed-off-by: Ofer Kiselov Nahman <ofer.kiselovnahman@weka.io> * Fix code review coments Signed-off-by: Ofer Kiselov Nahman <ofer.kiselovnahman@weka.io> * [Core] Added exception handling in async task + unit testing for error handling Signed-off-by: Ofer Kiselov Nahman <ofer.kiselovnahman@weka.io> * [Test] pre-commit fixes... Signed-off-by: Ofer Kiselov Nahman <ofer.kiselovnahman@weka.io> --------- Signed-off-by: Ofer Kiselov Nahman <ofer.kiselovnahman@weka.io>

Merge branch 'dev' into gds_error_handling_clean

49f3992

gemini-code-assist Bot reviewed Mar 3, 2026

View reviewed changes

oferki added 10 commits March 3, 2026 19:01

Fix code review coments

1aba8f7

Signed-off-by: Ofer Kiselov Nahman <ofer.kiselovnahman@weka.io>

Merge branch 'gds_error_handling_clean' of github.com:oferki/LMCache …

1a933f5

…into gds_error_handling_clean

Merge branch 'dev' into gds_error_handling_clean

bd0ceb9

Merge branch 'dev' into gds_error_handling_clean

8bc8190

Merge branch 'dev' into gds_error_handling_clean

c20f1ef

Merge branch 'dev' into gds_error_handling_clean

fd4c972

[Core] Added exception handling in async task + unit testing for erro…

aa3fbaf

…r handling Signed-off-by: Ofer Kiselov Nahman <ofer.kiselovnahman@weka.io>

[Test] pre-commit fixes...

3ed9c18

Signed-off-by: Ofer Kiselov Nahman <ofer.kiselovnahman@weka.io>

Merge branch 'dev' into gds_error_handling_clean

d40869f

Merge branch 'dev' into gds_error_handling_clean

1519dfc

sammshen approved these changes Mar 21, 2026

View reviewed changes

oferki added 3 commits March 21, 2026 17:38

Merge branch 'dev' into gds_error_handling_clean

76bd8f0

Merge branch 'dev' into gds_error_handling_clean

e7da4be

Merge branch 'dev' into gds_error_handling_clean

8bcbace

deng451e approved these changes Mar 24, 2026

View reviewed changes

oferki added 5 commits March 23, 2026 20:02

Merge branch 'dev' into gds_error_handling_clean

3746667

Merge branch 'dev' into gds_error_handling_clean

d88e15a

Merge branch 'dev' into gds_error_handling_clean

19ad796

Merge branch 'dev' into gds_error_handling_clean

75e1ab5

Merge branch 'dev' into gds_error_handling_clean

a9f86b1

deng451e enabled auto-merge (squash) March 25, 2026 22:45

github-actions Bot added the full Run comprehensive tests on this PR label Mar 25, 2026

deng451e merged commit 750a2c2 into LMCache:dev Mar 25, 2026
35 checks passed

oferki deleted the gds_error_handling_clean branch March 26, 2026 14:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] [GDS] Improve GDS backend error handling and retry logic#2675

[Core] [GDS] Improve GDS backend error handling and retry logic#2675
deng451e merged 20 commits intoLMCache:devfrom
oferki:gds_error_handling_clean

oferki commented Mar 3, 2026

Uh oh!

gemini-code-assist Bot commented Mar 3, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oferki commented Mar 20, 2026

Uh oh!

sammshen left a comment

Uh oh!

deng451e left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

oferki commented Mar 3, 2026

Uh oh!

gemini-code-assist Bot commented Mar 3, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oferki commented Mar 20, 2026

Uh oh!

sammshen left a comment

Choose a reason for hiding this comment

Uh oh!

deng451e left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants