Conversation
| } | ||
|
|
||
| let mut stream = response.bytes_stream(); | ||
| let mut total_bytes_downloaded = 0u64; |
There was a problem hiding this comment.
How about we just do file.len() at the end?
There was a problem hiding this comment.
Will have to do file.metadata().await?.len() which can be unreliable due to flushing and will be a sys call. It's fine like this imo. It will measure some response bytes too, which is okay.
📝 WalkthroughWalkthroughThe changes introduce comprehensive download instrumentation across multiple snapshot-related modules. A new public constant BYTES_IN_MB is added to the common module, then utilized throughout snapshot download operations in the collection, storage, and API layers to record and log download timing metrics including duration and speed in Mbps alongside byte tracking. The instrumentation maintains existing error handling and verification logic while adding measurement capabilities. Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes
Areas that may require extra attention during review:
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (4)
lib/segment/src/common/mod.rs (1)
247-248: BYTES_IN_MB constant is correct and consistent
BYTES_IN_MBmatches1024 * 1024and fits existing usage; no behavioral issues. If you want to avoid duplicating the literal, you could optionally define it in terms ofBYTES_IN_KB(e.g.,BYTES_IN_KB * BYTES_IN_KB), but it’s fine as-is.lib/collection/src/operations/snapshot_storage_ops.rs (1)
9-9: Download metrics and directory handling look good; consider small sequencing/type tweaksThe added timing/throughput logging and parent directory creation are solid and improve observability and robustness; the main behavior looks correct.
Two minor follow-ups to consider:
Log after integrity check instead of before it
Right now the"Object storage snapshot download completed"debug log is emitted before the file-size consistency check. If the size mismatch branch is hit, logs will still say “completed”, which can be slightly misleading when debugging corrupted/partial downloads. Reordering so that the debug log comes after thefile_meta.len() != total_size as u64check would keep logs aligned with actual success.Optional: align counter type with on-disk length
total_sizeis inferred asusizeand later compared tofile_meta.len()viaas u64. If you ever expect very large snapshots, you may want to maketotal_sizeau64to matchfile_meta.len()and avoid any theoreticalusizeoverflow on 32‑bit targets. If you do that, you can also drop theas u64cast in the comparison. Based on learnings, this mirrors the overflow‑resilience approach used elsewhere in the repo.If you’d like to apply (1), an example diff for the tail of the function would look like:
- // ensure flush - file.flush() - .await - .map_err(|e| CollectionError::service_error(format!("Failed to flush file: {e}")))?; - - let download_duration = download_start_time.elapsed(); - let total_size_mb = total_size as f64 / BYTES_IN_MB as f64; - let download_speed_mbps = total_size_mb / download_duration.as_secs_f64(); - log::debug!( - "Object storage snapshot download completed: path={}, size={:.2} MB, duration={:.2}s, speed={:.2} MB/s", - target_path.display(), - total_size_mb, - download_duration.as_secs_f64(), - download_speed_mbps - ); - - // check len to file len - let file_meta = tokio_fs::metadata(target_path).await?; + // ensure flush + file.flush() + .await + .map_err(|e| CollectionError::service_error(format!("Failed to flush file: {e}")))?; + + // check len to file len before logging success + let file_meta = tokio_fs::metadata(target_path).await?; if file_meta.len() != total_size as u64 { return Err(CollectionError::service_error(format!( "Downloaded file size does not match the expected size: {} != {}", file_meta.len(), total_size ))); } - Ok(()) + + let download_duration = download_start_time.elapsed(); + let total_size_mb = total_size as f64 / BYTES_IN_MB as f64; + let download_speed_mbps = if download_duration.as_secs_f64() > 0.0 { + total_size_mb / download_duration.as_secs_f64() + } else { + 0.0 + }; + log::debug!( + "Object storage snapshot download completed: path={}, size={:.2} MB, duration={:.2}s, speed={:.2} MB/s", + target_path.display(), + total_size_mb, + download_duration.as_secs_f64(), + download_speed_mbps + ); + + Ok(())Also applies to: 200-258
lib/storage/src/content_manager/snapshots/download.rs (1)
7-7: Remote snapshot download metrics look correct; minor naming/logging nitThe added timing and byte‑counting logic is straightforward and correctly computes MB and MB/s using
BYTES_IN_MB; the overall behavior ofdownload_fileis preserved.One small nit: the variable is called
download_speed_mbps, but the log message says"speed={:.2} MB/s". To avoid confusion between megabits (Mbps) and megabytes (MB/s), consider renaming the variable (e.g.,download_speed_mb_s) or adjusting the log string to match whatever unit you intend to standardize on across the codebase.Also applies to: 33-33, 56-61, 66-75
src/actix/api/snapshot_api.rs (1)
21-21: Partial snapshot download‑from‑peer metrics and flow look goodThe added timing/byte tracking inside
recover_partial_snapshot_fromis wired correctly:
download_start_timescopes the measurement to the remote partial snapshot fetch.total_bytes_downloadedis accumulated per chunk and returned viaStorageResult::Ok((collection, partial_snapshot_temp_path, total_bytes_downloaded)).- The match on
create_partial_snapshot_resultcleanly preserves theEmptyPartialSnapshotfast‑path and error propagation, and metrics are only logged for real downloads.- The debug log includes path, size in MB, duration, speed, and
shard_id, which is useful for tracing.Given that very similar MB/s logging now exists in several places, you might eventually consider a small shared helper (e.g., a utility that takes
start_time,bytes, and a label/context) to avoid copy‑pasting the same conversion and format string, but that’s purely a future cleanup and not required for this PR.Also applies to: 722-813
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
lib/collection/src/operations/snapshot_storage_ops.rs(3 hunks)lib/segment/src/common/mod.rs(1 hunks)lib/storage/src/content_manager/snapshots/download.rs(3 hunks)src/actix/api/snapshot_api.rs(3 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.rs
📄 CodeRabbit inference engine (.github/review-rules.md)
**/*.rs: Prefer explicitSomeType::from(x)over implicitx.into()in Rust
In new code, don't usetransmute_from_u8,transmute_to_u8,transmute_from_u8_to_slice,transmute_from_u8_to_mut_slice,transmute_to_u8_slice- usebytemuckorzerocopycrates instead
Prefer explicit exhaustive matches over catch-all_arm in Rust match expressions, so we don't miss handling new enum variants when they are added later (exceptions: tests and benchmarks, or when sure new variants won't affect logic)
Prefer explicit field ignoring using: _over using..in Rust struct destructuring (exceptions: tests and benchmarks, or when sure new fields won't affect logic)
Files:
lib/segment/src/common/mod.rslib/collection/src/operations/snapshot_storage_ops.rssrc/actix/api/snapshot_api.rslib/storage/src/content_manager/snapshots/download.rs
🧠 Learnings (1)
📚 Learning: 2025-08-15T11:42:00.297Z
Learnt from: IvanPleshkov
Repo: qdrant/qdrant PR: 7043
File: lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs:86-90
Timestamp: 2025-08-15T11:42:00.297Z
Learning: In lib/segment/src/vector_storage/quantized/quantized_mmap_storage.rs, overflow protection for encoded_storage_size computation (quantized_vector_size * vectors_count) is implemented in PR #7048, using checked_mul with u64 casting to prevent silent overflow.
Applied to files:
lib/segment/src/common/mod.rs
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
- GitHub Check: rust-tests (macos-latest)
- GitHub Check: rust-tests (ubuntu-latest)
- GitHub Check: rust-tests (windows-latest)
- GitHub Check: Build Qdrant Edge Python bindings
- GitHub Check: e2e-tests
- GitHub Check: test-consistency
- GitHub Check: test-shard-snapshot-api-s3-minio
- GitHub Check: test-consensus-compose
- GitHub Check: integration-tests-consensus
- GitHub Check: integration-tests
- GitHub Check: lint
* Log snapshot download duration * only 2 digits in filesize * Log for normal snapshots too * fmt * Log download stats for s3 snapshot API and cleanup
Logging file size, duration, and download rate makes the system more transparent. Doing this for full, partial, and s3 snapshot downloads