• lmcache/integration/vllm/vllm_multi_process_adapter.py
  • Replaced _lookup_job_ids and _finished_lookup_jobs with a single _lookup_hit_counts dictionary to store synchronous lookup results.
  • Updated maybe_submit_lookup_request to send a SYNC_LOOKUP request and directly store the returned hit count.
  • Simplified check_lookup_result to retrieve the hit count directly from _lookup_hit_counts without any network calls or polling.
  • Modified cleanup_lookup_result to simply remove the request ID from _lookup_hit_counts.
• lmcache/v1/distributed/storage_controllers/prefetch_controller.py
  • Imported the time module for busy-polling in synchronous operations.
  • Added synchronous_lookup method to submit lookup_and_lock tasks to L2 adapters and busy-poll for their results.
  • Implemented execute_load_phase to synchronously perform the L2-to-L1 data loading, including plan selection, L1 buffer reservation, and L2 load task submission.
  • Introduced unlock_lookup_results and an internal _unlock_results helper to release L2 pins from synchronous lookups without proceeding to load.
• lmcache/v1/distributed/storage_manager.py
  • Imported Bitmap for handling bitmap operations.
  • Added synchronous_lookup_and_lock to perform a combined L1 and L2 prefix lookup, pinning L2 objects and returning hit counts.
  • Implemented execute_prefetch_load to facilitate L2-to-L1 data movement based on prior synchronous lookup results.
  • Added unlock_l2_lookups to release L2 pins when a synchronous lookup is not followed by a load.
• lmcache/v1/multiprocess/protocols/base.py
  • Added SYNC_LOOKUP as a new RequestType enum member.
• lmcache/v1/multiprocess/protocols/engine.py
  • Exported SYNC_LOOKUP in the module's __all__ list.
  • Defined the SYNC_LOOKUP protocol, specifying its payload (KeyType, int) and response (int) and marking it as a BLOCKING handler type.
• lmcache/v1/multiprocess/server.py
  • Imported Bitmap for use in lookup state.
  • Defined _PendingLookupState dataclass to store state for synchronous lookup handoff.
  • Initialized _pending_lookups dictionary and _pending_lookups_lock to manage synchronous lookup state.
  • Modified the retrieve method to check for and execute pending L2 prefetch loads from _pending_lookups.
  • Added sync_lookup method to handle the new SYNC_LOOKUP request, performing L1 and L2 lookups and storing pending state.
  • Updated free_lookup_locks to release any pending L2 pins stored in _pending_lookups.
  • Registered engine.sync_lookup as the handler for the SYNC_LOOKUP request type.
• tests/v1/multiprocess/test_free_locks.py
  • Updated test assertions to reference _lookup_hit_counts instead of _lookup_job_ids.
  • Adjusted mock return values to reflect hit counts (chunks) instead of job IDs.

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.