[3/N][Feat]Persist metadata on device and fix raw-device benchmark setup by DongDongJu · Pull Request #2614 · LMCache/LMCache

DongDongJu · 2026-02-18T02:12:07Z

What this PR does / why we need it:

Replaces rust raw block backend metadata persistence path to use on-device checkpointing
Adds metadata format/version config support and robust load/apply validation behavior.
Updates rust raw block backend unit tests to match the new metadata persistence/recovery semantics.
Fixes storage backend microbenchmark behavior for real raw devices by skipping truncate() on /dev/* paths.

This is needed to keep metadata colocated with raw block storage, simplify recovery semantics, and avoid invalid benchmark setup on real block devices.

If applicable:

this PR contains user facing changes - docs added
this PR contains unit tests

…tests Replace file-manifest fallback with on-device metadata checkpoint handling and update rust raw block backend tests for the new recovery behavior and config surface. Signed-off-by: DongDongJu <commisori28@gmail.com> Signed-off-by: Dongjoo Seo <dongjoo.seo1@samsung.com>

Only create/truncate backing files for non-/dev paths so raw-device benchmark runs do not issue file truncate operations on block devices. Signed-off-by: DongDongJu <commisori28@gmail.com> Signed-off-by: Dongjoo Seo <dongjoo.seo1@samsung.com>

gemini-code-assist · 2026-02-18T02:12:30Z

Summary of Changes

Hello @DongDongJu, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly refactors the metadata persistence strategy for the Rust raw block storage backend. By moving from file-based manifests to on-device checkpointing, the system gains improved resilience and simplifies recovery processes. The changes also introduce configurable metadata formats and robust validation during loading, ensuring data integrity. Additionally, a critical fix was implemented for the storage backend microbenchmark to prevent unintended data truncation on actual raw devices, making the benchmarking process safer and more accurate.

Highlights

On-device Metadata Persistence: The Rust raw block backend now persists metadata directly on the device using a checkpointing mechanism, replacing the previous file-based manifest persistence. This improves recovery semantics and colocates metadata with the storage.
Robust Metadata Handling: Introduced support for metadata format/version configuration, along with robust load and apply validation behavior. This includes checks for magic bytes, version, sequence numbers, and CRC for data integrity.
Benchmark Setup Fix: The storage backend microbenchmark was updated to correctly handle real raw devices by skipping the truncate() operation on /dev/* paths, preventing potential data loss or errors during benchmarking.
Updated Unit Tests: Existing unit tests for the Rust raw block backend were updated to align with the new on-device metadata persistence and recovery semantics, and new tests were added to cover specific scenarios like data offsets and torn checkpoints.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

benchmarks/storage_backend_io/storage_backend_io_benchmark.py
- Modified the _bench_rust_raw_block function to conditionally skip truncate() calls for paths starting with /dev/, preventing unintended data loss on raw devices during benchmarks.
lmcache/v1/storage_backend/plugins/rust_raw_block_backend.py
- Added imports for struct, time, and zlib for new metadata handling.
- Defined _DEFAULT_META_MAGIC, _DEFAULT_META_VERSION, and _META_HEADER_STRUCT for on-device metadata structure.
- Updated class docstrings to reflect the change from 'Manifest persistence' to 'On-device metadata checkpoint'.
- Removed manifest_path configuration and related logic.
- Introduced new configuration options for on-device metadata, including meta_total_bytes, meta_magic, meta_version, meta_checkpoint_interval_sec, meta_idle_quiet_ms, meta_enable_periodic, and meta_verify_on_load.
- Added internal state variables _meta_seq, _meta_dirty_total, _meta_persisted, _inflight_io_count, and _last_io_ts for managing metadata checkpoints.
- Implemented _ensure_capacity_and_layout to calculate effective device capacity and the base offset for data, accounting for the metadata region.
- Modified _allocate_slot, _slot_to_offset, and _offset_to_slot to correctly map slot indices to physical offsets, respecting the metadata region.
- Incremented _meta_dirty_total when slots are freed (e.g., during eviction or removal) to track metadata changes.
- Removed _maybe_save_manifest and _save_manifest methods, replacing them with on-device checkpointing logic.
- Introduced _checkpoint_loop for periodic background metadata writes.
- Added _meta_payload_capacity, _read_meta_header, _load_meta_payload, _snapshot_state, _write_checkpoint, _checkpoint_once, _apply_loaded_state, and _validate_loaded_entries for comprehensive on-device metadata management.
- Modified _submit_write and get_blocking to update _inflight_io_count and _last_io_ts for idle detection.
- Added _decode_slot_header and _read_slot_header for parsing slot headers from the device.
- Updated the close method to gracefully stop the metadata checkpoint thread, wait for pending I/O, and force a final checkpoint.
- Implemented _load_checkpoint_from_device to restore the cache state from the most recent valid on-device metadata checkpoint.
tests/v1/storage_backend/test_rust_raw_block_backend.py
- Added import for the struct module.
- Imported _DEFAULT_META_MAGIC and _DEFAULT_META_VERSION constants.
- Updated test_rust_raw_block_backend_put_get_roundtrip and test_rust_raw_block_backend_eviction_lru to configure meta_total_bytes and disable periodic metadata checkpointing for test isolation.
- Renamed test_rust_raw_block_backend_manifest_roundtrip to test_rust_raw_block_backend_device_checkpoint_roundtrip and adapted its implementation to test the new on-device metadata persistence.
- Added test_rust_raw_block_backend_data_offsets_start_after_metadata to verify that data allocation begins after the reserved metadata region.
- Added test_rust_raw_block_backend_ignores_torn_newer_checkpoint to ensure the backend correctly ignores corrupted or invalid on-device metadata checkpoints.

Activity

Unit tests have been added/updated to cover the new metadata persistence and recovery mechanisms.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request is a significant improvement, replacing the file-based manifest in RustRawBlockBackend with a more robust on-device metadata checkpointing system. The new implementation includes important reliability features like magic numbers, versioning, CRC checks for data integrity, and on-load validation of entries. The related unit tests have been updated and expanded to cover these new features, including scenarios with corrupted data, which is excellent. Overall, this is a high-quality change that enhances the backend's resilience. I have one suggestion to further improve the robustness of the state restoration logic.

- Validate next_slot/free_slots strictly when loading metadata - Deduplicate and range-check free slots - Avoid re-adding duplicate/invalid free slots in runtime paths - Ensure free list excludes slots already used by loaded index - Guard metadata region to require header + payload space Signed-off-by: DongDongJu <commisori28@gmail.com> Signed-off-by: Dongjoo Seo <dongjoo.seo1@samsung.com>

DongDongJu · 2026-02-22T08:53:33Z

/gemini review

gemini-code-assist

Code Review

This pull request is a significant improvement, replacing the file-based manifest persistence in RustRawBlockBackend with a more robust on-device metadata checkpointing system. This change co-locates metadata with data, enhancing recovery semantics. The implementation is well-executed, featuring a new binary metadata format with versioning and CRC checks, a background thread for periodic checkpointing, and thorough validation of loaded metadata. The unit tests have been effectively updated to cover the new functionality, including scenarios for data offsets and corruption handling. Overall, this is a high-quality contribution that greatly improves the reliability of the raw block backend. I have one minor suggestion to enhance logging during metadata parsing to further improve debuggability.

…-metadata-journal # Conflicts: # benchmarks/storage_backend_io/storage_backend_io_benchmark.py # lmcache/v1/storage_backend/plugins/rust_raw_block_backend.py

Signed-off-by: DongDongJu <commisori28@gmail.com>

DongDongJu · 2026-03-02T03:38:52Z

/gemini review

gemini-code-assist

Code Review

This pull request significantly improves the RustRawBlockBackend by replacing the file-based manifest with a robust on-device metadata checkpointing system. This change is well-implemented, featuring versioning, integrity checks (magic numbers, CRCs), and thorough validation on load, which greatly enhances recovery and data safety. The shutdown process is also more robust, ensuring a final checkpoint is written. The unit tests have been updated accordingly, with new tests that validate the new on-device layout and recovery from corruption. Overall, this is an excellent enhancement. I've included a few minor suggestions to improve code clarity and exception handling.

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Dongjoo Seo <commisori28@gmail.com>

Signed-off-by: DongDongJu <commisori28@gmail.com>

DongDongJu · 2026-03-09T00:33:16Z

@maobaolong @sammshen Could you take a look by any chance?

sammshen · 2026-03-09T20:20:17Z

@DongDongJu I left two comments previously :)

DongDongJu · 2026-03-09T20:54:17Z

@DongDongJu I left two comments previously :)

I bliv that one was io_uring one. haha. isnt it?

sammshen · 2026-03-09T21:10:30Z

do these not show up?

DongDongJu · 2026-03-09T21:31:33Z

do these not show up?

Thanks for the review! It seems pended. Could you click submit the review?

sammshen

LGTM!

sammshen · 2026-03-04T05:04:14Z

+
+        raw = self._rawdev()
+
+        raw.pwrite_from_buffer(payload_off, payload, payload_len, payload_total_len)


checkpoint payload is written in-place (payload_off) before header commit. if the process/device crashes during that payload write, the previous payload is partially overwritten while the old header is still present. on restart CRC check fails and the loader drops metadata entirely so restart recovery can be lost after a single torn write.

@DongDongJu

You are right. Thanks! I added the fix for this in here e76aa46

sammshen · 2026-03-04T05:04:48Z

-                    shape_list = e.get("shape")
-                    fmt_name = e.get("fmt")
+
+                    offset = int(entry.get("offset", 0))


offset/size from checkpoint entries are accepted without checking:

offset >= data_base_offset
slot alignment ((offset - data_base_offset) % slot_bytes == 0)
size > 0 and size <= slot_bytes - header_bytes

@DongDongJu

Thanks! I added _is_valid_checkpoint_entry func.

sammshen · 2026-03-09T21:44:25Z

my bad, I forgot to click "submit review", thx for the reminder!

Signed-off-by: DongDongJu <commisori28@gmail.com> Signed-off-by: Dongjoo Seo <dongjoo.seo1@samsung.com>

DongDongJu · 2026-03-16T15:43:04Z

@sammshen @ApostaC PTAL!

DongDongJu · 2026-03-16T15:45:08Z

It seems weird. wha is the latest patch touch cache engine?
UT for cache_engine broke tests/v1/test_cache_engine.py::test_force_store_wait - AssertionError: assert 9472 == 9984.

sammshen

LGTM!

DongDongJu · 2026-03-24T21:24:19Z

@deng451e Could you take a look by any chance?

ApostaC

Some quick question, otherwise LGTM!

ApostaC · 2026-03-26T01:21:20Z

-            return
+        return snapshot, dirty_total
+
+    def _write_checkpoint(self, payload: bytes, dirty_total_snapshot: int) -> bool:


Should we make this function being thread-safe? This will be called by _checkpoint_once, which will be called in both _checkpoint_loop and close

Good catch.
The intended flow here is single-writer: close() stops the periodic checkpoint thread and joins it before issuing the final _checkpoint_once(force=True), so under the normal shutdown path _checkpoint_loop and close() should not overlap.

That said, relying on join(timeout=5) alone is a bit brittle, since it does not strictly guarantee the background thread has exited.
I agree it’s safer to serialize the checkpoint path explicitly. I’ll tighten this by guarding the checkpoint flow itself (likely at _checkpoint_once(), not only _write_checkpoint()), so the snapshot/write/seq update stays single-writer even during shutdown.

Next one is target to multi io worker. I will dealing this sync issue in next series.
Thanks for the review!

…tup (LMCache#2614) * feat(rust_raw_block): persist metadata on device and update recovery tests Replace file-manifest fallback with on-device metadata checkpoint handling and update rust raw block backend tests for the new recovery behavior and config surface. Signed-off-by: DongDongJu <commisori28@gmail.com> Signed-off-by: Dongjoo Seo <dongjoo.seo1@samsung.com> * fix(benchmark): avoid truncating raw block devices under /dev Only create/truncate backing files for non-/dev paths so raw-device benchmark runs do not issue file truncate operations on block devices. Signed-off-by: DongDongJu <commisori28@gmail.com> Signed-off-by: Dongjoo Seo <dongjoo.seo1@samsung.com> * rust_raw_block: harden metadata checkpoint slot validation - Validate next_slot/free_slots strictly when loading metadata - Deduplicate and range-check free slots - Avoid re-adding duplicate/invalid free slots in runtime paths - Ensure free list excludes slots already used by loaded index - Guard metadata region to require header + payload space Signed-off-by: DongDongJu <commisori28@gmail.com> Signed-off-by: Dongjoo Seo <dongjoo.seo1@samsung.com> * fix style Signed-off-by: DongDongJu <commisori28@gmail.com> * Update lmcache/v1/storage_backend/plugins/rust_raw_block_backend.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Dongjoo Seo <commisori28@gmail.com> * fix style Signed-off-by: DongDongJu <commisori28@gmail.com> * fix(raw-block): harden metadata checkpoint recovery Signed-off-by: DongDongJu <commisori28@gmail.com> Signed-off-by: Dongjoo Seo <dongjoo.seo1@samsung.com> --------- Signed-off-by: Dongjoo Seo <dongjoo.seo1@samsung.com> Signed-off-by: DongDongJu <commisori28@gmail.com> Signed-off-by: Dongjoo Seo <commisori28@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

DongDongJu added 2 commits February 18, 2026 02:08

DongDongJu mentioned this pull request Feb 18, 2026

[RFC] Rust based raw block device support #2394

Closed

DongDongJu closed this Feb 18, 2026

gemini-code-assist Bot reviewed Feb 18, 2026

View reviewed changes

Merge branch 'LMCache:dev' into feature/rust-raw-block-metadata-journal

a0f9da5

DongDongJu reopened this Feb 22, 2026

gemini-code-assist Bot reviewed Feb 22, 2026

View reviewed changes

Comment thread lmcache/v1/storage_backend/plugins/rust_raw_block_backend.py

DongDongJu added 4 commits February 24, 2026 20:07

Merge remote-tracking branch 'origin/dev' into feature/rust-raw-block…

45c64cc

…-metadata-journal # Conflicts: # benchmarks/storage_backend_io/storage_backend_io_benchmark.py # lmcache/v1/storage_backend/plugins/rust_raw_block_backend.py

Merge branch 'LMCache:dev' into feature/rust-raw-block-metadata-journal

f1d94f4

fix style

119f096

Signed-off-by: DongDongJu <commisori28@gmail.com>

Merge branch 'dev' into feature/rust-raw-block-metadata-journal

9b112eb

DongDongJu requested review from ApostaC, maobaolong and sammshen March 2, 2026 03:38

gemini-code-assist Bot reviewed Mar 2, 2026

View reviewed changes

DongDongJu and others added 2 commits March 1, 2026 22:12

Update lmcache/v1/storage_backend/plugins/rust_raw_block_backend.py

d68dc1d

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Dongjoo Seo <commisori28@gmail.com>

fix style

5a1d331

Signed-off-by: DongDongJu <commisori28@gmail.com>

sammshen requested changes Mar 9, 2026

View reviewed changes

fix(raw-block): harden metadata checkpoint recovery

e76aa46

Signed-off-by: DongDongJu <commisori28@gmail.com> Signed-off-by: Dongjoo Seo <dongjoo.seo1@samsung.com>

sammshen approved these changes Mar 17, 2026

View reviewed changes

DongDongJu added 5 commits March 17, 2026 14:45

Merge branch 'dev' into feature/rust-raw-block-metadata-journal

fdd1c46

Merge branch 'dev' into feature/rust-raw-block-metadata-journal

0a8aac3

Merge branch 'dev' into feature/rust-raw-block-metadata-journal

fd3ea23

Merge branch 'dev' into feature/rust-raw-block-metadata-journal

c0cab21

Merge branch 'dev' into feature/rust-raw-block-metadata-journal

ea8cc2b

ApostaC approved these changes Mar 26, 2026

View reviewed changes

DongDongJu enabled auto-merge (squash) March 26, 2026 04:53

github-actions Bot added the full Run comprehensive tests on this PR label Mar 26, 2026

Merge branch 'dev' into feature/rust-raw-block-metadata-journal

578f7f0

DongDongJu disabled auto-merge March 26, 2026 14:04

DongDongJu enabled auto-merge (squash) March 26, 2026 14:04

github-actions Bot added full Run comprehensive tests on this PR and removed full Run comprehensive tests on this PR labels Mar 26, 2026

DongDongJu merged commit 951cd3b into LMCache:dev Mar 26, 2026
36 checks passed

DongDongJu deleted the feature/rust-raw-block-metadata-journal branch March 30, 2026 07:46


		raw = self._rawdev()

		raw.pwrite_from_buffer(payload_off, payload, payload_len, payload_total_len)

Conversation

DongDongJu commented Feb 18, 2026

Uh oh!

gemini-code-assist Bot commented Feb 18, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

DongDongJu commented Feb 22, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

DongDongJu commented Mar 2, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DongDongJu commented Mar 9, 2026

Uh oh!

sammshen commented Mar 9, 2026

Uh oh!

DongDongJu commented Mar 9, 2026

Uh oh!

sammshen commented Mar 9, 2026

Uh oh!

DongDongJu commented Mar 9, 2026

Uh oh!

sammshen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sammshen commented Mar 9, 2026

Uh oh!

DongDongJu commented Mar 16, 2026

Uh oh!

DongDongJu commented Mar 16, 2026

Uh oh!

sammshen left a comment

Choose a reason for hiding this comment

Uh oh!

DongDongJu commented Mar 24, 2026

Uh oh!

ApostaC left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!