Add SVS-VAMANA index to info fields telemetry [MOD-10104]#6614
Add SVS-VAMANA index to info fields telemetry [MOD-10104]#6614
Conversation
| env.assertEqual(field_info_to_dict(fieldsInfo['search_fields_numeric']), get_search_field_info('Numeric', 2, NoIndex=1)) | ||
| env.assertEqual(field_info_to_dict(fieldsInfo['search_fields_geo']), get_search_field_info('Geo', 1)) | ||
| env.assertEqual(field_info_to_dict(fieldsInfo['search_fields_vector']), get_search_field_info('Vector', 2, Flat=1, HNSW=1)) | ||
| env.assertEqual(field_info_to_dict(fieldsInfo['search_fields_vector']), get_search_field_info('Vector', 4, Flat=1, HNSW=1, SVS_VAMANA=2, SVS_VAMANA_Compressed=1)) |
There was a problem hiding this comment.
Do we have a test that also checks we count gc_marked_deleted_vectors and used_memory_vector_index correctly?
I suggest to incrementally add svs changes and making sure they take effect (for example, create an svs index and ensure the used_memory_vector_index increased)
There was a problem hiding this comment.
Actually, I don't think that we count the marked deleted vector of svs as part of the vector stats today. We should address that and confirm it here after we implement it in vecsim. Regarding the used memory of vector indexes - it should be accounted - I will add a test
There was a problem hiding this comment.
Yep, svs should implement Tombstone
template <typename DataType, typename DistType>
VecSimIndexStatsInfo VecSimTieredIndex<DataType, DistType>::statisticInfo() const {
auto stats = VecSimIndexStatsInfo{
.memory = this->getAllocationSize(),
.numberOfMarkedDeleted = 0, // Default value if cast fails
};
// If backend implements VecSimIndexTombstone, get number of marked deleted
if (auto tombstone = dynamic_cast<VecSimIndexTombstone *>(this->backendIndex)) {
stats.numberOfMarkedDeleted = tombstone->getNumMarkedDeleted();
}
return stats;
}
There was a problem hiding this comment.
or to implement statisticInfo. The number of marked deleted vectors is available via changes_num field of the svs index.
There was a problem hiding this comment.
Right
I opened a ticket for that
Codecov Report❌ Patch coverage is
❌ Your project status has failed because the head coverage (48.88%) is below the adjusted base coverage (86.51%). You can increase the head coverage or adjust the Removed Code Behavior.
Additional details and impacted files@@ Coverage Diff @@
## master #6614 +/- ##
===========================================
- Coverage 87.57% 49.30% -38.28%
===========================================
Files 278 282 +4
Lines 44606 44802 +196
Branches 7571 7712 +141
===========================================
- Hits 39064 22089 -16975
- Misses 5430 22593 +17163
- Partials 112 120 +8
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
* account for vector index in total memory info * Add svs-vamana to info telemetry * revert manually * add svs vamana info test (cherry picked from commit 6ecbd3e)
|
Successfully created backport PR for |
Describe the changes in the pull request
Mark if applicable