Add a new "lookup" query vector builder by benwtrent · Pull Request #141488 · elastic/elasticsearch

benwtrent · 2026-01-28T23:04:30Z

This adds a new "lookup" query vector builder:

This significantly simplifies the user experience when attempting to find other vectors that are similar to a vector stored within an index.

closes: #141069

…thin an index

benwtrent · 2026-01-28T23:04:50Z

server/src/main/java/org/elasticsearch/search/vectors/LookupQueryVectorBuilder.java

+        client.prepareSearch(index)
+            .setQuery(new IdsQueryBuilder().addIds(id))
+            .setRouting(routing)
+            .setPreference("_local")
+            .setFetchSource(false)
+            .storedFields(_NONE_)
+            .addDocValueField(path)
+            .setSize(1)
+            .execute(ActionListener.wrap(searchResponse -> {


@jimczi is this what you had in mind?

I thought we could start with plain get but this is the optimised version ;).
I am not sure whether the fact that get uses a different thread pool is a feature or not though.
In this case, we would expect that the get hits cold data so we'd leave more room to parallelise the reads.

elasticsearchmachine · 2026-01-28T23:04:57Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

jimczi

LGTM, we can start with _search to do the optimised lookup.
Let's still plan for a follow up to allow doc values retrieval in get so that we can use the natural api to do the lookup?

jimczi · 2026-01-29T15:42:37Z

server/src/main/java/org/elasticsearch/search/vectors/LookupQueryVectorBuilder.java

+        client.prepareSearch(index)
+            .setQuery(new IdsQueryBuilder().addIds(id))
+            .setRouting(routing)
+            .setPreference("_local")
+            .setFetchSource(false)
+            .storedFields(_NONE_)
+            .addDocValueField(path)
+            .setSize(1)
+            .execute(ActionListener.wrap(searchResponse -> {


I thought we could start with plain get but this is the optimised version ;).
I am not sure whether the fact that get uses a different thread pool is a feature or not though.
In this case, we would expect that the get hits cold data so we'd leave more room to parallelise the reads.

github-actions · 2026-01-29T17:30:17Z

🔍 Preview links for changed docs

docs/reference/query-languages/query-dsl/query-dsl-knn-query.md

github-actions · 2026-01-29T17:30:19Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

Mikep86

Heads up that I'm almost certain that InterceptedInferenceKnnVectorQueryBuilder will need to be updated to handle this new query vector builder. That class was written with the assumption that TextEmbeddingQueryVectorBuilder was the only query vector builder type (see here, for example), which is no longer true.

benwtrent · 2026-01-29T19:56:44Z

...ClusterTest/java/org/elasticsearch/search/ccs/KnnVectorQueryBuilderCrossClusterSearchIT.java

    }

+    // TODO is this what we want to test?
+    public void testKnnQueryLookupCcsMinimizeRoundTripsTrue() throws Exception {


@Mikep86 here is the sort of thing I am thinking about. Basically, let's send a lookupqueryvectorbuilder that points to the local cluster (mandatory for lookup) and just points to the knn vector field. It does assume there is a value there (or it will throw a not-found).

These tests look like a good start to me. I'll branch off of this and make some changes to the interception logic to stabilize the tests. Then we can evaluate if we want those interception logic changes to be in a separate PR or merged into this. Does that work?

works great, thank you @Mikep86 !

@Mikep86 friendly ping :)

I know, this one is on my list to do ASAP. Ranger duty and all...

@Mikep86 you good! I just know I forget about things.

I created an issue for this to track my work on it: #142141

Now that #142254 is merged, LookupQueryVectorBuilder should work with semantic search CCS 🎉

I don't think we need to explicitly test LookupQueryVectorBuilder with CCS here, as we have tests with a generic query vector builder in #142254. As long as LookupQueryVectorBuilder follows the contract of throwing in buildVector on incomplete input or an error, we should be good. WDYT?

docs/reference/query-languages/query-dsl/query-dsl-knn-query.md

shainaraskas · 2026-02-03T21:29:04Z

docs/reference/query-languages/query-dsl/query-dsl-knn-query.md

+Elasticsearch supports knn queries with a vector that is stored within an index.
+
+Here is an example utilizing lookup.


can you clarify what the lookup is actually doing here? "that looks up the vector in a separate index using the lookup parameter" maybe?

docs/reference/query-languages/query-dsl/query-dsl-knn-query.md

Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>

docs/reference/query-languages/query-dsl/query-dsl-knn-query.md

Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>

…into vector-look-qvp

This adds a new "lookup" query vector builder: This significantly simplifies the user experience when attempting to find other vectors that are similar to a vector stored within an index. closes: elastic#141069

Adding docs for the new `lookup` query vector builder. Waiting on elastic/elasticsearch#141488 to be merged.

benwtrent added 2 commits January 28, 2026 17:15

Adding lookup query vector builder for querying with vector stored wi…

32bd5e1

…thin an index

Adding tests

55c5685

benwtrent added >enhancement :Search Relevance/Vectors Vector search v9.4.0 labels Jan 28, 2026

benwtrent commented Jan 28, 2026

View reviewed changes

elasticsearchmachine added the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Jan 28, 2026

[CI] Auto commit changes from spotless

10076bb

jimczi approved these changes Jan 29, 2026

View reviewed changes

adding docs and fixing tests

9e35758

benwtrent mentioned this pull request Jan 29, 2026

Adding lookup for knn search elastic/elasticsearch-specification#5981

Merged

benwtrent requested review from a team January 29, 2026 17:40

Mikep86 reviewed Jan 29, 2026

View reviewed changes

benwtrent and others added 7 commits January 29, 2026 13:01

skipping ccs as index for lookup must be local

1a2d3af

Adding tv

a80dcc0

Merge remote-tracking branch 'upstream/main' into vector-look-qvp

36dfca6

tv move

adb0e04

Merge remote-tracking branch 'upstream/main' into vector-look-qvp

2e4fc43

iter

25c0d52

[CI] Auto commit changes from spotless

4794532

benwtrent commented Jan 29, 2026

View reviewed changes

shainaraskas reviewed Feb 3, 2026

View reviewed changes

docs/reference/query-languages/query-dsl/query-dsl-knn-query.md Outdated Show resolved Hide resolved

shainaraskas reviewed Feb 3, 2026

View reviewed changes

docs/reference/query-languages/query-dsl/query-dsl-knn-query.md Show resolved Hide resolved

shainaraskas reviewed Feb 3, 2026

View reviewed changes

benwtrent commented Feb 4, 2026

View reviewed changes

docs/reference/query-languages/query-dsl/query-dsl-knn-query.md Outdated Show resolved Hide resolved

Apply suggestions from code review

a9bc011

Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>

shainaraskas reviewed Feb 4, 2026

View reviewed changes

docs/reference/query-languages/query-dsl/query-dsl-knn-query.md Show resolved Hide resolved

Update docs/reference/query-languages/query-dsl/query-dsl-knn-query.md

ad7183c

Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>

This was referenced Feb 9, 2026

Update intercepted queries to support generic query vector builders #142141

Closed

Update semantic search CCS to support generic query vector builders #142254

Merged

benwtrent mentioned this pull request Feb 17, 2026

Docs for lookup query vector builder in knn.md elastic/docs-content#5179

Merged

Merge remote-tracking branch 'upstream/main' into vector-look-qvp

650c1e2

benwtrent requested a review from a team as a code owner February 26, 2026 17:55

benwtrent added 2 commits February 26, 2026 14:53

fixing tests

9735e29

fixing more tests

6cb7574

benwtrent added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Feb 26, 2026

benwtrent and others added 7 commits February 26, 2026 18:26

Merge remote-tracking branch 'upstream/main' into vector-look-qvp

27c6e78

Merge remote-tracking branch 'upstream/main' into vector-look-qvp

c8c895d

fixing serverless edge cases

f53fd53

Merge remote-tracking branch 'upstream/main' into vector-look-qvp

159a27a

Merge branch 'main' into vector-look-qvp

6b6f63a

Merge remote-tracking branch 'upstream/main' into vector-look-qvp

af1caa1

Merge branch 'vector-look-qvp' of github.com:benwtrent/elasticsearch …

48ae6a7

…into vector-look-qvp

elasticsearchmachine merged commit ce8f5f2 into elastic:main Feb 28, 2026
35 checks passed

benwtrent added a commit to elastic/docs-content that referenced this pull request Mar 5, 2026

Docs for lookup query vector builder in knn.md (#5179)

f383509

Adding docs for the new `lookup` query vector builder. Waiting on elastic/elasticsearch#141488 to be merged.

		Elasticsearch supports knn queries with a vector that is stored within an index.

		Here is an example utilizing lookup.

Conversation

benwtrent commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Jan 28, 2026

Uh oh!

jimczi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

github-actions bot commented Jan 29, 2026

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

Mikep86 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

benwtrent commented Jan 28, 2026 •

edited

Loading

github-actions bot commented Jan 29, 2026 •

edited

Loading