feature: save and load in flat index #163
Conversation
|
Awesome, looks like a lot was already there, especially the assemble. Let's wait for comments from @ibhati |
thx, fixed Co-authored-by: ibhati <ishwar.s.bhati@intel.com>
…el/ScalableVectorSearch into feature_flat_index_save_load
bindings/python/src/flat.cpp
Outdated
| data_directory: Directory where the dataset will be saved. | ||
| Note: All directories should be separate to avoid accidental name collision with any | ||
| auxiliary files that are needed when saving the various components of the index. |
There was a problem hiding this comment.
I think we can remove this Note as there is only one directory in the Flat index unlike the Vamana index, which saves config/graph/data in multiple directories
bindings/python/tests/test_flat.py
Outdated
| print("Flat, From Array, Float32") | ||
| flat = svs.Flat(data_f32, svs.DistanceType.L2) | ||
| self._do_test(flat, queries_f32, groundtruth, svs.DistanceType.L2, data_f32) | ||
| save_reload_and_test(flat, queries_f32, groundtruth, data_f32) |
There was a problem hiding this comment.
| save_reload_and_test(flat, queries_f32, groundtruth, data_f32) | |
| self._do_test(flat, queries_f32, groundtruth, svs.DistanceType.L2, data_f32) | |
| save_reload_and_test(flat, queries_f32, groundtruth, data_f32) |
There was a problem hiding this comment.
Please keep the original _do_test(...) call before doing save/load. The same comment applies to other calls below
There was a problem hiding this comment.
Thx, fixed by have save_and_load and a remove_dir called before and after _do_test
tests/integration/exhaustive.cpp
Outdated
| svs::VectorDataLoader<float>(temp_dir), distance_type, index.get_num_threads() | ||
| ); | ||
| } | ||
|
|
There was a problem hiding this comment.
Maybe do this in the end of this function followed by search and recall testing
There was a problem hiding this comment.
Thx, moved to end of the function!
ibhati
left a comment
There was a problem hiding this comment.
@yuejiaointel Thanks for incorporating the feedback. Now, it looks great to me!
Add save and load functions to flat index