Skip to content

Fast Post-Filtering Library for Vector Search #282

@s-gobriel

Description

@s-gobriel

Is your feature request related to a problem? Please describe.
A filtered vector search consists of a query vector and a filter for a specific field. Current design of VDMS uses the descriptor indices for doing the vector search (KNN) and the filtering is done using the graph store (PMGD, neo4j)

Fast post-filtering library introduces a new layer of filtering that can be used after the vector search and before hitting the graph store to narrow down the candidate list considerably that satisfy both KNN and filter (to avoid sending too many queries to the graph store), and as a result improve the filtered-search throughput

Describe the solution you'd like
1- A new library with an new API to:
(a) create "summaries" which are vectors that matches a certain attribute that the user can apply a fast filter on.
(b) query these "summaries" to return a set of K vectors that satisfy being K-nearest neighbors and satisfy the attribute for filtering

2- Integrate the new library with the KNN flow and graph store search

Metadata

Metadata

Labels

EnhancementNew features or upgrades for functionality, performance, etc

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions