Conversation
7bc5ead to
3897979
Compare
e20531c to
0e50337
Compare
This comment was marked as resolved.
This comment was marked as resolved.
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (1)
lib/segment/src/vector_storage/vector_storage_base.rs (1)
245-246: Boiler-plate explosion – consider macro-driven enum dispatchAdding
SparseVolatilerequired touching every single match arm in this file.
The pattern has already repeated for the dense & multi-dense volatile variants and is becoming hard to maintain.A small
macro_rules!helper (orenum_dispatch/pastecrates) that generates the match arms based on a list of variants would:
- eliminate this error-prone duplication,
- make future variants cheaper to add,
- keep the file readable.
This is an optional refactor, but it will pay off quickly given the pace of new storages.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (10)
docs/redoc/master/openapi.json(1 hunks)lib/segment/src/index/hnsw_index/gpu/gpu_vector_storage/mod.rs(1 hunks)lib/segment/src/types.rs(1 hunks)lib/segment/src/vector_storage/quantized/quantized_vectors.rs(1 hunks)lib/segment/src/vector_storage/raw_scorer.rs(1 hunks)lib/segment/src/vector_storage/sparse/mod.rs(1 hunks)lib/segment/src/vector_storage/sparse/volatile_sparse_vector_storage.rs(1 hunks)lib/segment/src/vector_storage/tests/test_appendable_multi_dense_vector_storage.rs(1 hunks)lib/segment/src/vector_storage/tests/test_appendable_sparse_vector_storage.rs(2 hunks)lib/segment/src/vector_storage/vector_storage_base.rs(22 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
lib/segment/src/vector_storage/tests/test_appendable_sparse_vector_storage.rs (1)
lib/segment/src/vector_storage/sparse/volatile_sparse_vector_storage.rs (1)
new_volatile_sparse_vector_storage(34-36)
⏰ Context from checks skipped due to timeout of 90000ms (13)
- GitHub Check: test-consistency
- GitHub Check: test-shard-snapshot-api-s3-minio
- GitHub Check: Basic TLS/HTTPS tests
- GitHub Check: test-snapshot-operations-s3-minio
- GitHub Check: test-low-resources
- GitHub Check: integration-tests-consensus
- GitHub Check: integration-tests
- GitHub Check: test-consensus-compose
- GitHub Check: lint
- GitHub Check: storage-compat-test
- GitHub Check: rust-tests (ubuntu-latest)
- GitHub Check: rust-tests (macos-latest)
- GitHub Check: rust-tests (windows-latest)
🔇 Additional comments (10)
lib/segment/src/vector_storage/quantized/quantized_vectors.rs (1)
291-292: Correct handling of the new SparseVolatile variant.The new match arm correctly handles the
VectorStorageEnum::SparseVolatilevariant by returningErr(OperationError::WrongSparse), consistent with other sparse vector storage variants. The#[cfg(test)]attribute ensures this code only appears in test builds.lib/segment/src/index/hnsw_index/gpu/gpu_vector_storage/mod.rs (1)
396-399: Consistent error handling for SparseVolatile on GPU.The added match arm for
SparseVolatilecorrectly returns an error indicating that sparse vectors are not supported on GPU, consistent with how other sparse vector storage types are handled. The#[cfg(test)]attribute appropriately limits this code to test builds.lib/segment/src/vector_storage/sparse/mod.rs (1)
4-5: Well-structured module exposure for test builds.Adding the
volatile_sparse_vector_storagemodule with public visibility and conditional compilation for tests is appropriate. This follows the same pattern used for other volatile storage implementations in the codebase, making it available only for test scenarios.lib/segment/src/vector_storage/tests/test_appendable_multi_dense_vector_storage.rs (1)
94-95: Appropriate match arm for unreachable case.The addition of the
SparseVolatilematch arm marked asunreachable!()is correct, as this test is specifically for multi dense vector storage and shouldn't encounter sparse storage variants. This maintains consistency with how other sparse storage types are handled in this match statement.lib/segment/src/vector_storage/raw_scorer.rs (1)
104-105: Properly implements scoring for the new SparseVolatile vector storage variantThe implementation correctly adds support for the new
SparseVolatilevariant in the vector storage enum, following the same pattern as other sparse vector storage types by delegating to theraw_sparse_scorer_implfunction. The#[cfg(test)]attribute properly ensures this code is only compiled during test builds, consistent with other volatile storage implementations in this file.docs/redoc/master/openapi.json (2)
12018-12018: Refinedon_diskdescription for clarityThe description now explicitly mentions RocksDB storage, matching the underlying implementation and improving documentation accuracy.
12025-12025: Refinedmmapdescription for clarityThe description now explicitly mentions GridStore (memory-mapped) storage, aligning docs with code behavior.
lib/segment/src/vector_storage/tests/test_appendable_sparse_vector_storage.rs (2)
19-19: Import added for the new volatile sparse vector storage functionality.This import is correctly added to use the new volatile sparse vector storage implementation.
136-136: Successfully integrated volatile sparse vector storage for testing.The test has been updated to use the new
new_volatile_sparse_vector_storage()function instead of the previous storage implementation. This change appropriately tests the newly added in-memory volatile sparse vector storage implementation, which is the main objective of this PR.lib/segment/src/types.rs (1)
1384-1384: Improved documentation clarity for sparse vector storage types.The documentation for
SparseVectorStorageTypeenum variants has been enhanced by specifying the backend technology for each storage type:
OnDisk: Clarified that it uses "rocksdb storage"Mmap: Clarified that it uses "gridstore storage"This provides better context about the underlying implementation of each storage option.
Also applies to: 1386-1386
* Add volatile sparse vector storage * Add and use volatile sparse vector test * Update OpenAPI spec * Resolve review remarks
* Add volatile sparse vector storage * Add and use volatile sparse vector test * Update OpenAPI spec * Resolve review remarks
Depends on #6564
Same as #6564, but for sparse vectors.
Tasks
devAll Submissions:
devbranch. Did you create your branch fromdev?New Feature Submissions:
cargo +nightly fmt --allcommand prior to submission?cargo clippy --all --all-featurescommand?