Enable mmap-ped Faiss indices (IndexFlatCodes-based and HNSW) by alexanderguzhva · Pull Request #996 · zilliztech/knowhere

alexanderguzhva · 2024-12-18T18:34:46Z

No description provided.

mergify · 2024-12-18T18:35:24Z

@alexanderguzhva 🔍 Important: PR Classification Needed!

For efficient project management and a seamless review process, it's essential to classify your PR correctly. Here's how:

If you're fixing a bug, label it as kind/bug.
For small tweaks (less than 20 lines without altering any functionality), please use kind/improvement.
Significant changes that don't modify existing functionalities should be tagged as kind/enhancement.
Adjusting APIs or changing functionality? Go with kind/feature.

For any PR outside the kind/improvement category, ensure you link to the associated issue using the format: “issue: #”.

Thanks for your efforts and contribution to the community!.

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>

codecov · 2024-12-18T23:31:22Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.88%. Comparing base (3c46f4c) to head (8ea01d5).
Report is 273 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##           main     #996       +/-   ##
=========================================
+ Coverage      0   73.88%   +73.88%     
=========================================
  Files         0       82       +82     
  Lines         0     6919     +6919     
=========================================
+ Hits          0     5112     +5112     
- Misses        0     1807     +1807

see 82 files with indirect coverage changes

foxspy · 2024-12-19T02:22:31Z

thirdparty/faiss/faiss/impl/mapped_io.cpp

+
+        // set 'random' access pattern
+        // todo: check the error
+        madvise(address, filesize, MADV_RANDOM);


a warn log needs to be added here

foxspy · 2024-12-19T02:26:47Z

thirdparty/faiss/faiss/impl/index_read.cpp

@@ -512,7 +594,8 @@ static void read_HNSW(HNSW* hnsw, IOReader* f) {
    READVECTOR(hnsw->cum_nneighbor_per_level);
    READVECTOR(hnsw->levels);


Don't these require mmap? Although these only account for a small part

same question, what do we exactly mmap for hnsw, seems neighbors and raw vectors/quant codes @alexanderguzhva

foxspy · 2024-12-19T02:38:15Z

/kind improvement

foxspy

/lgtm

sre-ci-robot · 2024-12-19T02:38:32Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexanderguzhva, foxspy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [foxspy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…tech#996) * HNSW_SQenable mmap-ped faiss indices (IndexFlatCodes-based and HNSW) Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com> * temporary add IO_FLAG_MMAP_IFC Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com> --------- Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>

Summary: This PR introduces a backport of a combination of zilliztech/knowhere#996 and zilliztech/knowhere#1032 that allow to have memory-mapped and zerocopy indces. The root underlying idea is that we replace certain `std::vector<>` containers with a custom `faiss::MaybeOwnedVector<>` container, which may behave either as `std::vector<>`, or as a view of a certain pointer / descriptor. We don't replace all the instances of `std::vector<>`, but the largest ones. This change affects `IndexFlatCodes`-based and `IndexHNSW` CPU indices. (done) alter IVF lists as well. (done) alter binary indices as well. Memory-mapped index works like this: ```C++ std::unique_ptr<faiss::Index> index_mm( faiss::read_index(filenamename.c_str(), faiss::IO_FLAG_MMAP_IFC)); ``` In theory, it should be ready to be used from Python. All the descriptor management should be working. Zero-copy index works like this: ```C++ #include <faiss/impl/zerocopy_io.h> faiss::ZeroCopyIOReader reader(buffer.data(), buffer.size()); std::unique_ptr<faiss::Index> index_zc(faiss::read_index(&reader)); ``` All the pointer management for `faiss::ZeroCopyIOReader` should be handled manually. I'm not sure how to plug this into Python yet, maybe, some ref-counting is required. (done) some refactoring Pull Request resolved: #4199 Reviewed By: mengdilin Differential Revision: D69972250 Pulled By: mdouze fbshipit-source-id: 98a3f94d6884814873d3534ee25f960892ef1076

Summary: This PR introduces a backport of a combination of zilliztech/knowhere#996 and zilliztech/knowhere#1032 that allow to have memory-mapped and zerocopy indces. The root underlying idea is that we replace certain `std::vector<>` containers with a custom `faiss::MaybeOwnedVector<>` container, which may behave either as `std::vector<>`, or as a view of a certain pointer / descriptor. We don't replace all the instances of `std::vector<>`, but the largest ones. This change affects `IndexFlatCodes`-based and `IndexHNSW` CPU indices. (done) alter IVF lists as well. (done) alter binary indices as well. Memory-mapped index works like this: ```C++ std::unique_ptr<faiss::Index> index_mm( faiss::read_index(filenamename.c_str(), faiss::IO_FLAG_MMAP_IFC)); ``` In theory, it should be ready to be used from Python. All the descriptor management should be working. Zero-copy index works like this: ```C++ #include <faiss/impl/zerocopy_io.h> faiss::ZeroCopyIOReader reader(buffer.data(), buffer.size()); std::unique_ptr<faiss::Index> index_zc(faiss::read_index(&reader)); ``` All the pointer management for `faiss::ZeroCopyIOReader` should be handled manually. I'm not sure how to plug this into Python yet, maybe, some ref-counting is required. (done) some refactoring Pull Request resolved: facebookresearch/faiss#4199 Reviewed By: mengdilin Differential Revision: D69972250 Pulled By: mdouze fbshipit-source-id: 98a3f94d6884814873d3534ee25f960892ef1076

Summary: This PR introduces a backport of a combination of zilliztech/knowhere#996 and zilliztech/knowhere#1032 that allow to have memory-mapped and zerocopy indces. The root underlying idea is that we replace certain `std::vector<>` containers with a custom `faiss::MaybeOwnedVector<>` container, which may behave either as `std::vector<>`, or as a view of a certain pointer / descriptor. We don't replace all the instances of `std::vector<>`, but the largest ones. This change affects `IndexFlatCodes`-based and `IndexHNSW` CPU indices. (done) alter IVF lists as well. (done) alter binary indices as well. Memory-mapped index works like this: ```C++ std::unique_ptr<faiss::Index> index_mm( faiss::read_index(filenamename.c_str(), faiss::IO_FLAG_MMAP_IFC)); ``` In theory, it should be ready to be used from Python. All the descriptor management should be working. Zero-copy index works like this: ```C++ #include <faiss/impl/zerocopy_io.h> faiss::ZeroCopyIOReader reader(buffer.data(), buffer.size()); std::unique_ptr<faiss::Index> index_zc(faiss::read_index(&reader)); ``` All the pointer management for `faiss::ZeroCopyIOReader` should be handled manually. I'm not sure how to plug this into Python yet, maybe, some ref-counting is required. (done) some refactoring Pull Request resolved: facebookresearch#4199 Reviewed By: mengdilin Differential Revision: D69972250 Pulled By: mdouze fbshipit-source-id: 98a3f94d6884814873d3534ee25f960892ef1076

sre-ci-robot requested review from cydrain and hhy3 December 18, 2024 18:34

sre-ci-robot added the size/L label Dec 18, 2024

mergify bot added the dco-passed label Dec 18, 2024

mergify bot added the do-not-merge/missing-related-issue label Dec 18, 2024

alexanderguzhva mentioned this pull request Dec 18, 2024

memory-mapped faiss indices facebookresearch/faiss#4101

Closed

HNSW_SQenable mmap-ped faiss indices (IndexFlatCodes-based and HNSW)

c58252f

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>

alexanderguzhva force-pushed the mmap branch from 370aa42 to c58252f Compare December 18, 2024 20:19

sre-ci-robot added size/XL and removed size/L labels Dec 18, 2024

temporary add IO_FLAG_MMAP_IFC

8ea01d5

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>

mergify bot added the ci-passed label Dec 18, 2024

foxspy reviewed Dec 19, 2024

View reviewed changes

sre-ci-robot added the kind/improvement label Dec 19, 2024

foxspy approved these changes Dec 19, 2024

View reviewed changes

sre-ci-robot assigned foxspy Dec 19, 2024

sre-ci-robot added the lgtm label Dec 19, 2024

sre-ci-robot added the approved label Dec 19, 2024

mergify bot removed the do-not-merge/missing-related-issue label Dec 19, 2024

sre-ci-robot merged commit c24fd21 into zilliztech:main Dec 19, 2024

alexanderguzhva mentioned this pull request Feb 19, 2025

Memory-mapping and Zero-copy deserializers facebookresearch/faiss#4199

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable mmap-ped Faiss indices (IndexFlatCodes-based and HNSW)#996

Enable mmap-ped Faiss indices (IndexFlatCodes-based and HNSW)#996
sre-ci-robot merged 2 commits intozilliztech:mainfrom
alexanderguzhva:mmap

alexanderguzhva commented Dec 18, 2024

Uh oh!

mergify bot commented Dec 18, 2024

Uh oh!

codecov bot commented Dec 18, 2024 •

edited

Loading

Uh oh!

foxspy Dec 19, 2024

Uh oh!

foxspy Dec 19, 2024

Uh oh!

chasingegg Dec 30, 2024 •

edited

Loading

Uh oh!

foxspy commented Dec 19, 2024

Uh oh!

foxspy left a comment

Uh oh!

sre-ci-robot commented Dec 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -512,7 +594,8 @@ static void read_HNSW(HNSW* hnsw, IOReader* f) {
		READVECTOR(hnsw->cum_nneighbor_per_level);
		READVECTOR(hnsw->levels);

Conversation

alexanderguzhva commented Dec 18, 2024

Uh oh!

mergify bot commented Dec 18, 2024

Uh oh!

codecov bot commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

foxspy Dec 19, 2024

Choose a reason for hiding this comment

Uh oh!

foxspy Dec 19, 2024

Choose a reason for hiding this comment

Uh oh!

chasingegg Dec 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

foxspy commented Dec 19, 2024

Uh oh!

foxspy left a comment

Choose a reason for hiding this comment

Uh oh!

sre-ci-robot commented Dec 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Dec 18, 2024 •

edited

Loading

chasingegg Dec 30, 2024 •

edited

Loading