Use the preload hint on completion fields and memory terms dictionaries.#14634
Merged
jpountz merged 1 commit intoapache:mainfrom May 11, 2025
Merged
Use the preload hint on completion fields and memory terms dictionaries.#14634jpountz merged 1 commit intoapache:mainfrom
jpountz merged 1 commit intoapache:mainfrom
Conversation
This enables the `PreloadHint` introduced in apache#14604 on completion fields and memory terms dictionaries, which are both expected to fit in the page cache in practice. I don't have specific interest in these two file formats, I was more interested in having more than one file format that uses `PreloadHint` to make sure it's generally useful and not only to KNN vectors.
Contributor
Author
jpountz
commented
May 9, 2025
| this.fstTermsInput = | ||
| state.directory.openInput(termsFileName, state.context.withHints(FileTypeHint.INDEX)); | ||
| state.directory.openInput( | ||
| termsFileName, state.context.withHints(FileTypeHint.DATA, PreloadHint.INSTANCE)); |
Contributor
Author
There was a problem hiding this comment.
I also changed the FileTypeHint to DATA since this FST effectively stores all the terms data.
jpountz
added a commit
that referenced
this pull request
May 11, 2025
…es. (#14634) This enables the `PreloadHint` introduced in #14604 on completion fields and memory terms dictionaries, which are both expected to fit in the page cache in practice. I don't have specific interest in these two file formats, I was more interested in having more than one file format that uses `PreloadHint` to make sure it's generally useful and not only to KNN vectors.
weizijun
added a commit
to weizijun/lucene
that referenced
this pull request
May 16, 2025
* main: (31 commits) Fix termination condition in TestStressNRTReplication. (apache#14665) deps(java): bump com.gradle.develocity from 3.19 to 3.19.2 (apache#14662) Build: remove hard-coded Java versions from ecj.javadocs.prefs (apache#14651) Update verifier comment to show label (apache#14658) Catch and re-throw Throwable rather than using a success boolean (apache#14633) Mention label in changelog verifier comment (apache#14656) Enable PR actions in changelog verifier (apache#14644) Fix FuzzySet#getEstimatedNumberUniqueValuesAllowingForCollisions to properly account for hashCount (apache#14614) Don't perform additional KNN querying after timeout, fixes apache#14639 (apache#14640) Add instructions to help/IDEs.txt for VSCode and Neovim (apache#14646) build(deps): bump ruff from 0.11.7 to 0.11.8 in /dev-tools/scripts (apache#14603) deps(java): bump de.jflex:jflex from 1.8.2 to 1.9.1 (apache#14583) Use the preload hint on completion fields and memory terms dictionaries. (apache#14634) Clean up FileTypeHint a bit. (apache#14635) Expressions: Improve test to use a fully private class or method Remove deprecations in expressions (apache#14641) removing constructor with deprecated attribute 'onlyLongestMatch (apache#14356) Moving CHANGES entry for apache#14609 from 11.0 to 10.3 (apache#14638) Overrides rewrite in PointRangeQuery to optimize AllDocs/NoDocs cases (apache#14609) Adding benchmark for histogram collector over point range query (apache#14622) ... # Conflicts: # lucene/CHANGES.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This enables the
PreloadHintintroduced in #14604 on completion fields and memory terms dictionaries, which are both expected to fit in the page cache in practice.I don't have specific interest in these two file formats, I was more interested in having more than one file format that uses
PreloadHintto make sure it's generally useful and not only to KNN vectors.