Add dense_vector equality and inequality support in ES|QL by majiayu000 · Pull Request #140005 · elastic/elasticsearch

majiayu000 · 2025-12-26T11:00:26Z

Summary

Adds support for binary comparison operators (==, !=) for dense_vector type in ES|QL.

Resolves #139929

Changes

Added EqualsDenseVectorEvaluator and NotEqualsDenseVectorEvaluator classes to handle dense_vector comparison
Updated Equals.java and NotEquals.java with dense_vector support in evaluatorMap and @param annotations
Added DENSE_VECTOR_EQUALITY capability in EsqlCapabilities
Updated test classes with dense_vector test cases
Added csv-spec tests for dense_vector equality operations

Two dense vectors are equal if they have the same dimensions and all corresponding elements are equal.

Test plan

EqualsTests with dense_vector test cases
NotEqualsTests with dense_vector test cases
EqualsErrorTests updated
NotEqualsErrorTests updated
dense_vector.csv-spec integration tests

cla-checker-service · 2025-12-26T11:00:31Z

💚 CLA has been signed

github-actions · 2025-12-26T11:02:00Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

Signed-off-by: majiayu000 <1835304752@qq.com>

nik9000 · 2025-12-26T13:01:01Z

@carlosdelest, could you look at this one?

elasticsearchmachine · 2025-12-26T13:01:08Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

majiayu000 · 2025-12-26T13:39:52Z

@elasticmachine recheck CLA

carlosdelest

Nice work!

Some comments about adding test cases - can you please address them @majiayu000 ?

carlosdelest · 2026-01-07T14:14:12Z

.../java/org/elasticsearch/xpack/esql/expression/predicate/operator/comparison/EqualsTests.java

+        suppliers.add(
+            new TestCaseSupplier("<dense_vector>, <different dense_vector>", List.of(DataType.DENSE_VECTOR, DataType.DENSE_VECTOR), () -> {
+                List<Float> left = randomDenseVector();
+                List<Float> right = randomDenseVector();


Nice thinking about vectors actually being different! I think it would be better to use other testing constructs for this:

Suggested change

List<Float> right = randomDenseVector();

List<Float> right = randomValueOtherThan(left, EsTestCase::randomDenseVector());

updated in 0fd20d0 ✅

carlosdelest · 2026-01-07T14:15:05Z

...va/org/elasticsearch/xpack/esql/expression/predicate/operator/comparison/NotEqualsTests.java

+                List<Float> left = randomDenseVector();
+                List<Float> right = randomDenseVector();
+                // Make sure vectors are different
+                if (left.equals(right) && left.size() > 0) {


Same as above

updated in 0fd20d0 ✅

x-pack/plugin/esql/qa/testFixtures/src/main/resources/dense_vector.csv-spec

…nd csv-specs

majiayu000 · 2026-01-07T16:39:38Z

Hi @carlosdelest, I have updated the PR to address your review comments

carlosdelest · 2026-01-07T17:07:39Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/dense_vector-bfloat16.csv-spec

+required_capability: dense_vector_field_type_released
+required_capability: dense_vector_agg_metric_double_if_version
+required_capability: l2_norm_vector_similarity_function
+required_capability: generic_vector_format
+required_capability: dense_vector_equality


Please use only required_capability with the needed capabilities for the test to work - in this case:

Suggested change

required_capability: dense_vector_field_type_released

required_capability: dense_vector_agg_metric_double_if_version

required_capability: l2_norm_vector_similarity_function

required_capability: generic_vector_format

required_capability: dense_vector_equality

required_capability: dense_vector_equality

We can skip checking other capabilities as they are superseded by dense_vector_equality, which is newer

x-pack/plugin/esql/qa/testFixtures/src/main/resources/dense_vector.csv-spec

majiayu000 · 2026-01-08T01:09:33Z

Hi @carlosdelest, thanks for the additional feedback! I have updated the CSV specs to:

Simplify the capabilities to only require dense_vector_equality as suggested.
Add test cases for dimension mismatch (e.g., denseVectorCompareDifferentDimensions) in all 4 spec files.
Let me know if this looks good!

carlosdelest · 2026-01-08T08:48:22Z

@elasticsearchmachine test this please

...earch/xpack/esql/expression/predicate/operator/comparison/NotEqualsDenseVectorEvaluator.java

carlosdelest · 2026-01-09T08:40:33Z

@elasticsearchmachine test this please

carlosdelest · 2026-01-09T09:17:15Z

@majiayu000 I've added some commits to address some issues. I see that we have some compile errors in EqualsTests and NotEqualsTests, can you please address them?

Also, I liked these versions better when they were testing equality and inequality. Why did you change that to a fixed amount of random tests? 🤔 The original approach was better IMO

majiayu000 · 2026-01-09T14:38:35Z

@carlosdelest Thanks for the review!

I've fixed the compilation errors in EqualsTests and NotEqualsTests (missing static imports). Regarding the test coverage, you were right — the fixed set of random tests wasn't ideal. I've refactored it to use the TestCaseSupplier.casesCrossProduct approach with a stable set of random vectors. This ensures we test the full cross-product (both equality and inequality scenarios) while keeping the implementation clean.

Ready for another look!

carlosdelest · 2026-01-09T15:25:23Z

@majiayu000 thanks for iterating on this! I'm afraid that I don't see the changes you mention on the tests. Can you please double check?

Also, please don't force push - it's easier for me to review with the full commit history 👍

majiayu000 · 2026-01-10T16:37:45Z

@carlosdelest thanks for checking. I've double checked the commit f6d34e5 and it contains the changes for EqualsTests and NotEqualsTests (imports and randomDenseVector logic). I also pushed a new commit cdfa93d with the same refactoring for InsensitiveEqualsTests.

Please double check f6d34e5 for EqualsTests/NotEqualsTests and cdfa93d for InsensitiveEqualsTests. I didn't force push, so the history is preserved.

mromaios · 2026-03-02T16:35:58Z

Hey @majiayu000, addressed some pending test comments in 0fd20d0 and run the tests, but the csv-spec tests are failing.

From a quick look, it's two issues:

We should wrap the arrays with to_dense_vector() as these operators don't have implicit casting
In some tests where the value in the column is null, the expected one should be null too I think, but it's currently missing from the test: e.g. here

If you can have a look when you get the chance?
Thanks!

mromaios · 2026-03-08T19:36:02Z

@elasticsearchmachine test this please

Signed-off-by: majiayu000 <1835304752@qq.com>

majiayu000 · 2026-03-09T03:29:06Z

@mromaios Both issues fixed — wrapped arrays with to_dense_vector() and added missing null expectations. Rebased on latest main.

mromaios · 2026-03-09T11:45:30Z

Thanks @majiayu000! I think this looks good now. Took care of some compilation errors after an interface move.

Running CI, (hopefully for the last time 🤞)

mromaios · 2026-03-09T11:45:42Z

@elasticsearchmachine test this please

mromaios · 2026-03-09T13:03:43Z

@elasticsearchmachine test this please

mromaios · 2026-03-09T15:41:08Z

@elasticsearchmachine test this please

mromaios · 2026-03-10T10:27:11Z

@elasticmachine run elasticsearch-ci/pr-upgrade-part-3, elasticsearch-ci/rest-compatibility please

carlosdelest

LGTM, thank you for your work @majiayu000 and @mromaios !

mromaios · 2026-03-10T11:38:45Z

Thanks again for the contribution @majiayu000 and for the many rounds of reviews @carlosdelest and @kkharbas.
I'll merge this in once we have a green CI.

mromaios · 2026-03-10T11:47:51Z

@elasticmachine run Elasticsearch Serverless Checks, elasticsearch-ci/rest-compatibility please

mromaios · 2026-03-10T11:50:49Z

@elasticmachine run elasticsearch-ci/rest-compatibility please

mromaios · 2026-03-10T16:12:45Z

@elasticsearchmachine test this please

elasticsearchmachine added v9.4.0 needs:triage Requires assignment of a team area label external-contributor Pull request authored by a developer outside the Elasticsearch team labels Dec 26, 2025

majiayu000 added a commit to majiayu000/elasticsearch that referenced this pull request Dec 26, 2025

Add changelog for PR elastic#140005

808cb36

Signed-off-by: majiayu000 <1835304752@qq.com>

nik9000 requested a review from carlosdelest December 26, 2025 12:58

nik9000 added :Search Relevance/ES|QL Search functionality in ES|QL and removed needs:triage Requires assignment of a team area label labels Dec 26, 2025

elasticsearchmachine added the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Dec 26, 2025

gareth-ellis mentioned this pull request Dec 29, 2025

Add dense_vector support for Equals and NotEquals operators Closes #139929 #139997

Closed

carlosdelest reviewed Jan 7, 2026

View reviewed changes

carlosdelest self-assigned this Jan 7, 2026

carlosdelest added the >enhancement label Jan 7, 2026

majiayu000 added a commit to majiayu000/elasticsearch that referenced this pull request Jan 7, 2026

Address review comments on PR elastic#140005: refactor tests and expa…

3638c23

…nd csv-specs

carlosdelest reviewed Jan 7, 2026

View reviewed changes

kkharbas reviewed Jan 8, 2026

View reviewed changes

...earch/xpack/esql/expression/predicate/operator/comparison/NotEqualsDenseVectorEvaluator.java Outdated Show resolved Hide resolved

majiayu000 force-pushed the dense-vector-equality-support branch 2 times, most recently from c89ab28 to f6d34e5 Compare January 9, 2026 14:47

kkharbas approved these changes Mar 2, 2026

View reviewed changes

majiayu000 and others added 8 commits March 9, 2026 11:25

Add dense_vector equality and inequality support in ES|QL

c7da5ff

Signed-off-by: majiayu000 <1835304752@qq.com>

Refactor InsensitiveEqualsTests to use TestCaseSupplier

cf1d960

Trigger PR update

69f41dc

Fix dense_vector equality tests per review comments

dc5e99b

[CI] Auto commit changes from spotless

0d1ca58

Revert InsensitiveEqualsTests changes unrelated to dense_vector equality

6fb7236

refactor: use existing testing constructs in Equals and NotEquals tests

d1c9d06

Fix dense_vector csv-spec casts and null expectations

a6944a2

Signed-off-by: majiayu000 <1835304752@qq.com>

majiayu000 force-pushed the dense-vector-equality-support branch from a3ced57 to a6944a2 Compare March 9, 2026 03:26

mromaios and others added 2 commits March 9, 2026 12:21

Merge branch 'main' into dense-vector-equality-support

99d8c0e

refactor code after ExpressionEvaluator interface move

9e0207e

Merge branch 'main' into dense-vector-equality-support

93b7a4f

mromaios and others added 2 commits March 9, 2026 16:28

Merge branch 'main' into dense-vector-equality-support

293043e

test: update expected value after dense_vector addition

17fe1db

carlosdelest approved these changes Mar 10, 2026

View reviewed changes

Merge branch 'main' into dense-vector-equality-support

3b5c562

mromaios merged commit 187e39f into elastic:main Mar 11, 2026
37 checks passed

	List<Float> right = randomDenseVector();
	List<Float> right = randomValueOtherThan(left, EsTestCase::randomDenseVector());

Conversation

majiayu000 commented Dec 26, 2025

Summary

Changes

Test plan

Uh oh!

cla-checker-service bot commented Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 26, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

nik9000 commented Dec 26, 2025

Uh oh!

elasticsearchmachine commented Dec 26, 2025

Uh oh!

majiayu000 commented Dec 26, 2025

Uh oh!

carlosdelest left a comment

Choose a reason for hiding this comment

Uh oh!

carlosdelest Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

mromaios Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

carlosdelest Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

mromaios Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

majiayu000 commented Jan 7, 2026

Uh oh!

carlosdelest Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

majiayu000 commented Jan 8, 2026

Uh oh!

carlosdelest commented Jan 8, 2026

Uh oh!

Uh oh!

carlosdelest commented Jan 9, 2026

Uh oh!

carlosdelest commented Jan 9, 2026

Uh oh!

majiayu000 commented Jan 9, 2026

Uh oh!

carlosdelest commented Jan 9, 2026

Uh oh!

majiayu000 commented Jan 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mromaios commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mromaios commented Mar 8, 2026

Uh oh!

majiayu000 commented Mar 9, 2026

Uh oh!

mromaios commented Mar 9, 2026

Uh oh!

mromaios commented Mar 9, 2026

Uh oh!

mromaios commented Mar 9, 2026

Uh oh!

mromaios commented Mar 9, 2026

Uh oh!

mromaios commented Mar 10, 2026

Uh oh!

carlosdelest left a comment

cla-checker-service bot commented Dec 26, 2025 •

edited

Loading

majiayu000 commented Jan 10, 2026 •

edited

Loading

mromaios commented Mar 2, 2026 •

edited

Loading