Merged
Conversation
bplatz
reviewed
Nov 26, 2024
Contributor
bplatz
left a comment
There was a problem hiding this comment.
Awesome!
Are you thinking we should write out new VG indexes with every commit, or try to rebuild the index from the last FlakeIndex 't' and just hook into that process?
| (loop [[[vg-alias vg] & r] vg-map | ||
| address-map {}] | ||
| (if vg-alias | ||
| (let [address (<? (vg/write-vg index-catalog vg))] |
Contributor
There was a problem hiding this comment.
I'm guessing you are light on optimizations for now intentionally - but maybe at least worth a TODO that this ideally would be parallelized.
| (loop [[[vg-alias address] & r] vg-address-map | ||
| vg-map {}] | ||
| (if vg-alias | ||
| (let [vg (<? (vg/read-vg index-catalog address))] |
Contributor
There was a problem hiding this comment.
Not sure of best strategy, but ideally we delay loads until the index is used.
|
|
||
| #?(:clj (set! *warn-on-reflection* true)) | ||
|
|
||
| (defprotocol GraphSerializer |
Contributor
There was a problem hiding this comment.
Seems too broad, maybe FlakeIndexSerializer?
Merged
…eature/bm25-search-continued # Conflicts: # src/clj/fluree/db/virtual_graph/bm25/index.clj # src/clj/fluree/db/virtual_graph/parse.cljc
# Conflicts: # src/clj/fluree/db/flake/transact.cljc # src/clj/fluree/db/virtual_graph/parse.cljc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This patch adds functions to serialize and persist any virtual graph indexes as part of the normal indexing process and to reload any persisted indexes when a db is loaded from storage.
This pr currently has draft status because the index isn't persisted if the minimum novelty threshold isn't reached and the graph indexing process doesn't kick off. I'm still working on the best way to detect when the bm25 index has changed and needs persisting at commit time, and how to wait until the bm25 indexing process completes to persist the index without holding up the transactor, but I wanted to publish the persistence code now for feedback.