Skip to content

Narek/streaming hnsw index selects#322

Merged
var77 merged 10 commits intomainfrom
narek/streaming-hnsw-index-selects
Jul 15, 2024
Merged

Narek/streaming hnsw index selects#322
var77 merged 10 commits intomainfrom
narek/streaming-hnsw-index-selects

Conversation

@Ngalstyan4
Copy link
Copy Markdown
Contributor

No description provided.

Copy link
Copy Markdown

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Summary

  • Renamed and reorganized SQL update files for better structure
  • Updated triggers and functions for async tasks and vector search
  • Modified HNSW index scan logic with continue_search variable
  • Updated third_party/usearch submodule to a new commit hash
  • Added empty sql/updates/0.3.0--0.3.1.sql file for version tracking

src/hnsw/scan.c Outdated

int ef = ldb_hnsw_ef_search; // 0 if not set, but we pass it into usearch_custom_ef anyway since 0 is also a
// sentinel value there
bool continue_search = false;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Introduced continue_search to manage search state.

src/hnsw/scan.c Outdated
scanstate->usearch_scalar,
k,
ef,
continue_search,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passed continue_search to usearch_search_ef.

@@ -252,6 +254,8 @@ bool ldb_amgettuple(IndexScanDesc scan, ScanDirection dir)
return false;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Set continue_search to true for subsequent searches.

src/hnsw/scan.c Outdated
scanstate->usearch_scalar,
k,
ef,
continue_search,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passed continue_search to usearch_search_ef.

ldb_wal_retriever_area_reset(scanstate->retriever_ctx);

scanstate->count = num_returned;
scanstate->current = 0;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reset scanstate->current to 0 after search.

@@ -1 +1 @@
Subproject commit a20193e1f3e10e18c0a8edd860727b11bf424e44
Subproject commit ece8908a58597ffa6570ca6fc5ef043103d05038
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated submodule commit from a20193e1f3e10e18c0a8edd860727b11bf424e44 to ece8908a58597ffa6570ca6fc5ef043103d05038.

@Ngalstyan4 Ngalstyan4 force-pushed the narek/streaming-hnsw-index-selects branch from 7302bbb to c29f9aa Compare June 20, 2024 20:19
Default to streaming search for index pagination

This used to have a GUC option. I removed streaming search GUC and
defaulted to it becasue the old approach had an issue and could
simetimes skip some index values and never returned them when paginating
@Ngalstyan4 Ngalstyan4 force-pushed the narek/streaming-hnsw-index-selects branch from 53bb3bb to 194e0f1 Compare July 4, 2024 12:18
@Ngalstyan4 Ngalstyan4 force-pushed the narek/streaming-hnsw-index-selects branch from 194e0f1 to 87aa88a Compare July 5, 2024 12:47
@codecov
Copy link
Copy Markdown

codecov bot commented Jul 13, 2024

Codecov Report

Attention: Patch coverage is 50.00000% with 2 lines in your changes missing coverage. Please review.

Files Patch % Lines
src/hnsw/scan.c 50.00% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@var77 var77 merged commit 6c84849 into main Jul 15, 2024
@var77 var77 deleted the narek/streaming-hnsw-index-selects branch July 15, 2024 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants