Skip to content

Feature/bm25 persist#993

Merged
bplatz merged 43 commits intomainfrom
feature/bm25-persist
Mar 24, 2025
Merged

Feature/bm25 persist#993
bplatz merged 43 commits intomainfrom
feature/bm25-persist

Conversation

@bplatz
Copy link
Contributor

@bplatz bplatz commented Mar 19, 2025

This builds on the persistence work done in #942 by @zonotope

This now is fully working with tests that check persistence state on load, and some other improvements.

BM25 can be used with this PR, but there is still work to be done:

  1. Deleting a BM25 index
  2. Preventing updates to BM25 index (in favor of delete/re-create)
  3. additional stop word support for different languages
  4. additional filters in addition to stemmer (e.g. convert to ascii characters before stemming)

@zonotope
park on error channel when there's an error activating an alias
1472ebd
@zonotope
clean up map for legibility
166f11b
@zonotope
add arity for query function that doesn't include fuel tracker
91bbb5e
@zonotope
Merge branch 'feature/bm25-search' into feature/bm25-search-continued
c180ed8
@zonotope
use two functions instead of boolean argument
3c197af
@zonotope
remove (de)serialize methods from virtual graph protocol
cafa5da
@zonotope
move serialization protocol ns up one level
edce126
@zonotope
use set instead of closure for stop words; load as resource
1135d33
@zonotope
add stemmer protocol
76fea24
@zonotope
add ns for bm25 index serialization with term serializer
58cd0b6
@zonotope
add functions to (de)serialize bm25 index state
594f649
@zonotope
deserialize avg-length back to a ratio in clj
f5e7911
@zonotope
add fn to serialize entire bm25 index
474e002
@zonotope
make storage ns clj only; add deserialize fn
72cda51
@zonotope
remove redundant require
cd4a423
@zonotope
break up query parsing fns for reuse
3e80bbb
@zonotope
only consider db alias when generating vg id
0af31df
@zonotope
query-parsed -> parsed-query
b5ff5d7
@zonotope
fn to deserialize bm25 index
cef02e0
@zonotope
dedicated function to detect and extract virtual graph flakes
d07a22a
@zonotope
StorageSerializer -> GraphSerializer
d8d772f
@zonotope
(de)serialize bm25 type
8f3ab05
@zonotope
add protocol to (de)serialize bm25 graphs
0477277
@zonotope
add functions to read and write bm25 graph
240b484
@zonotope
don't overload "serialize" word
e27832a
@zonotope
move protocol ns up one level; add read/write multimethod
3168a01
@zonotope
convert read/write bm25 methods to vg multimethod implementations
ea4e6c1
@zonotope
serialize and store virtual graphs when writing db root
ffde698
@zonotope
deserialize vg map when reading and reifying db roots
036b056
@zonotope
use storage path function to determine storage path
0e07b9b
bplatz added 12 commits January 28, 2025 06:47
@bplatz
Merge remote-tracking branch 'origin/feature/bm25-retractions' into f…
3f0cb29 
…eature/bm25-search-continued

# Conflicts:
#	src/clj/fluree/db/virtual_graph/bm25/index.clj
#	src/clj/fluree/db/virtual_graph/parse.cljc
@bplatz
Merge branch 'main' into feature/bm25-search-continued
eb9bee2 
# Conflicts:
#	src/clj/fluree/db/flake/transact.cljc
#	src/clj/fluree/db/virtual_graph/parse.cljc
@bplatz
Merge branch 'main' into feature/bm25-search-continued
c739704
@bplatz
Persistence updated / working
f4a3929
@bplatz
bm25 test @id in subgraph selector
5ea0936
@bplatz
bm25 prevent select * queries
a91fbc5
@bplatz
bm25 - handle query with selectOne
34fcfde
@bplatz
bm25 persisted test
c14e136
@bplatz
Catch unhandled async exception
d509c82
@bplatz
bm25 in load from disk when no index created
6b2f027
@bplatz
bm25 Test persistence with/without index
f9edd14
@bplatz
Merge remote-tracking branch 'origin/main' into feature/bm25-persist
765337e 
# Conflicts:
#	src/fluree/db/serde.cljc
#	src/fluree/db/serde/protocol.cljc
#	src/fluree/db/virtual_graph.cljc
#	src/fluree/db/virtual_graph/bm25/storage.clj
#	src/fluree/db/virtual_graph/proto.cljc
@bplatz bplatz requested a review from a team March 19, 2025 11:19
zonotope
zonotope approved these changes Mar 21, 2025
View reviewed changes
Copy link
Contributor

@zonotope zonotope left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a few suggestions, but 👍🏾

@bplatz bplatz merged commit 419fb4e into main Mar 24, 2025
6 checks passed
@bplatz bplatz deleted the feature/bm25-persist branch March 24, 2025 17:00
@aaj3f aaj3f mentioned this pull request Mar 28, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zonotope zonotope zonotope approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bplatz @zonotope