Support arithmetic operations for dense_vectors: scalar version by kkharbas · Pull Request #141060 · elastic/elasticsearch

kkharbas · 2026-01-21T16:26:34Z

Adds support for arithmetic operations - '+', '-', '*' and '/' when one operand is a dense_vector and other is scalar.

Performs the arithmetic operation on each element of the dense_vector against the scalar operand. The scalar operand should be a constant value, scalar fields are not supported.
✔️ dense_vector + scalar constant
❌ dense_vector + scalar field

Closes #140538

carlosdelest

Hey @kkharbas ! I don't think we need to have a

dense_vector + field

so we operate on each dense_vector field over each scalar field on the other column, but:

dense_vector + scalar constant

so we use the same scalar constant to operate on every dense_vector field row.

So, we could do something like:

EVAL my_negated_vectors = my_vector_field * -1

or

EVAL my_added_vectors = my_vector_field + 3.0

Right now I don't see the use case for operating on a row by row basis for scalars - and I think in any case using a single scalar constant would be a more frequent use case, and simpler to implement for now.

We may come back to this implementation, but I think for now we should reduce the scope and look for single numeric constants to be applied.

What do you think?

kkharbas · 2026-01-22T14:37:49Z

Hey @kkharbas ! I don't think we need to have a

dense_vector + field

so we operate on each dense_vector field over each scalar field on the other column, but:

dense_vector + scalar constant

so we use the same scalar constant to operate on every dense_vector field row.

So, we could do something like:

EVAL my_negated_vectors = my_vector_field * -1

or

EVAL my_added_vectors = my_vector_field + 3.0

Right now I don't see the use case for operating on a row by row basis for scalars - and I think in any case using a single scalar constant would be a more frequent use case, and simpler to implement for now.

We may come back to this implementation, but I think for now we should reduce the scope and look for single numeric constants to be applied.

What do you think?

That makes total sense. I was just focussing on completeness of the operations but usability wise its fair to only support scalar constant.

github-actions · 2026-02-09T20:00:28Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

carlosdelest

I have a question on the approach - instead of dealing separately with scalar operands in a separate class, convert them via an ExpressionEvaluator to dense vectors so we can operate on them with the current infra.

...arch/xpack/esql/expression/predicate/operator/arithmetic/DenseVectorArithmeticOperation.java

carlosdelest · 2026-02-10T08:56:53Z

...search/xpack/esql/expression/predicate/operator/arithmetic/DoubleDenseVectorOpEvaluator.java

+ * {@link EvalOperator.ExpressionEvaluator} implementation for performing arithmetic operations on lhs=double and rhs=dense_vector arguments.
+ *
+ */
+class DoubleDenseVectorOpEvaluator implements EvalOperator.ExpressionEvaluator {


Having a specific class for handling operating with doubles looks like something we should try to avoid.

I wonder about creating a ToFloat internal function for dealing with this, similar to other conversion functions like ToDouble. This would allow us to use ToFloat as part of the evaluator chain that the factory creates when we find a numeric type that we want to convert to floats, and then deal with doing float to float operations.

WDYT?

carlosdelest · 2026-02-10T09:17:36Z

...search/xpack/esql/expression/predicate/operator/arithmetic/DenseVectorDoubleOpEvaluator.java

+ * {@link EvalOperator.ExpressionEvaluator} implementation for performing arithmetic operations on lhs=dense_vector and rhs=double argument.
+ *
+ */
+class DenseVectorDoubleOpEvaluator implements EvalOperator.ExpressionEvaluator {


I wonder if we can avoid having a specific evaluator for doubles, and adding a specific DenseVectorBinaryEvaluator.

I was thinking on having a ExpressionEvaluator that would convert a numeric block into a dense_vector. That way, we could wrap the numeric block using this converter before it is processed by the DenseVectorsEvaluator, and thus reducing the problem to an operation on two dense vectors which we already have solved.

WDYT?

kkharbas · 2026-02-10T17:39:46Z

I have a question on the approach - instead of dealing separately with scalar operands in a separate class, convert them via an ExpressionEvaluator to dense vectors so we can operate on them with the current infra.

I had given this approach some consideration but ran into hiccups.
One is we throw error when vector dimensions don't match. Would it be possible to distinguish between the case when an operand is a scalar converted to vector or a vector with dimension == 1?

I will take a fresh look again at this approach.

kkharbas · 2026-02-11T17:50:51Z

I have a question on the approach - instead of dealing separately with scalar operands in a separate class, convert them via an ExpressionEvaluator to dense vectors so we can operate on them with the current infra.

I had given this approach some consideration but ran into hiccups. One is we throw error when vector dimensions don't match. Would it be possible to distinguish between the case when an operand is a scalar converted to vector or a vector with dimension == 1?

I will take a fresh look again at this approach.

I tried out this approach of using existing conversion evaluators, e.g. ToDenseVectorFromIntEvaluator, its a neat way to use existing dense_vector support.

Few thoughts and questions on this,

Evaluation logic is slightly different for when both arguments are dense_vectors vs one is scalar, e.g. validating dimensions and getting N vs 1 values from the block. The execution path to take depends on the expression and can be injected into the evaluator as isLhsScalar and isRhsScalar boolean parameters. Question/concern is - is it a significant performance concern doing these checks during evaluation (once per page)?
This approach can be applied to scalar fields too (not just constants) without adding much complexity. If the concern was just related to complexity, does it make sense to support non-literals?

elasticsearchmachine · 2026-02-18T17:25:58Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

kkharbas · 2026-02-18T17:26:31Z

test this please

elasticsearchmachine · 2026-02-18T17:27:27Z

Hi @kkharbas, I've created a changelog YAML for you.

kkharbas · 2026-02-19T18:45:41Z

@carlosdelest the PR is ready for another look. Simplified the implementation based on offline discussion. Thank you

carlosdelest

LGTM, great work @kkharbas !

Thanks for iterating on this one ❤️

...arch/xpack/esql/expression/predicate/operator/arithmetic/DenseVectorArithmeticOperation.java

...icsearch/xpack/esql/expression/predicate/operator/arithmetic/DenseVectorScalarEvaluator.java

Adds support for arithmetic operations - '+', '-', '*' and '/' when one operands is a dense_vector and other is scalar. Performs the arithmetic operation on each element of the dense_vector against the scalar operand. Closes elastic#140538

…tic#141060) Support arithmetic operations for dense_vectors: scalar version Adds support for arithmetic operations - '+', '-', '*' and '/' when one operands is a dense_vector and other is scalar. Performs the arithmetic operation on each element of the dense_vector against the scalar operand. Closes elastic#140538 --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>

…on-sliced-reindex * upstream/main: Update docs for v9.3.1 release (elastic#142887) Update docs for v9.2.6 release (elastic#142888) Improves visibility of vector index options and inference configuration (elastic#141653) Disable CAE in microsoft-graph-authz plugin (elastic#142848) Small improvements to `GetSnapshotsIT#testAllFeatures` (elastic#142825) Fix IndexSettingsTests synthetic ID tests (elastic#142654) [Test] Unmute tests of SnapshotShutdownIT (elastic#142921) Fixing metrics_info.json kibana definition file name (elastic#142813) [Packaging] Disable glibc 2.43 malloc huge pages in Wolfi images (elastic#142894) Mute org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsTSDBSyntheticIdIntegTests testSearchableSnapshot elastic#142918 Add shard heap usage to ClusterInfo (elastic#139557) ESQL: Load script fields row-by-row (elastic#142807) ESQL: Consolidate doc values memory tracking (elastic#142816) ES-14124 Create Index Count Limit User documentation Page (elastic#142570) Add a es819 codec test to verify tryRead returns null if may contain duplicates (elastic#142409) Support arithmetic operations for dense_vectors: scalar version (elastic#141060) [Transform] Allow project_routing (elastic#142421) Refactor query rewrite async actions for knn and sparse_vector queries (elastic#142889) Do not mark bulk indexing requests as retried after primary relocations (elastic#142157)

…tic#141060) Support arithmetic operations for dense_vectors: scalar version Adds support for arithmetic operations - '+', '-', '*' and '/' when one operands is a dense_vector and other is scalar. Performs the arithmetic operation on each element of the dense_vector against the scalar operand. Closes elastic#140538 --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>

elasticsearchmachine added the v9.4.0 label Jan 21, 2026

kkharbas requested review from alex-spies, carlosdelest and ioanatia January 21, 2026 17:38

carlosdelest reviewed Jan 22, 2026

View reviewed changes

kkharbas force-pushed the dense_vector-arithmetic-scalar branch from 685c7a9 to 834ba7d Compare February 9, 2026 19:57

kkharbas force-pushed the dense_vector-arithmetic-scalar branch from 834ba7d to f70c477 Compare February 10, 2026 02:08

kkharbas marked this pull request as ready for review February 10, 2026 06:55

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Feb 10, 2026

carlosdelest reviewed Feb 10, 2026

View reviewed changes

carlosdelest removed the request for review from alex-spies February 12, 2026 07:46

kkharbas added Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch :Search Relevance/ES|QL Search functionality in ES|QL labels Feb 18, 2026

elasticsearchmachine removed the needs:triage Requires assignment of a team area label label Feb 18, 2026

kkharbas added the >enhancement label Feb 18, 2026

kkharbas requested a review from a team as a code owner February 19, 2026 17:54

carlosdelest approved these changes Feb 20, 2026

View reviewed changes

...arch/xpack/esql/expression/predicate/operator/arithmetic/DenseVectorArithmeticOperation.java Show resolved Hide resolved

...icsearch/xpack/esql/expression/predicate/operator/arithmetic/DenseVectorScalarEvaluator.java Outdated Show resolved Hide resolved

kkharbas enabled auto-merge (squash) February 23, 2026 16:34

kkharbas disabled auto-merge February 23, 2026 17:08

kkharbas enabled auto-merge (squash) February 23, 2026 17:09

kkharbas added 2 commits February 23, 2026 11:29

simplify by only supporting constant scalar values

84ce39e

kkharbas added 10 commits February 23, 2026 11:29

remove unsigned long tests

429984b

auto generated docs

4b8dd8c

minor update

5370ca3

review comments: move compatiblity checks, unused method

3637215

lot of refactoring, simplifying the implementation

8398d89

Update docs/changelog/141060.yaml

930b41f

new esql capability

d02b779

small bug introducted in earlier PR

42acc68

add some comments

9b4dff6

minor refactor based on review comment

a6ee999

kkharbas force-pushed the dense_vector-arithmetic-scalar branch from 2e234fe to a6ee999 Compare February 23, 2026 19:30

[CI] Auto commit changes from spotless

8888d16

kkharbas merged commit 0cdf499 into elastic:main Feb 23, 2026
35 checks passed

Conversation

kkharbas commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlosdelest left a comment

Choose a reason for hiding this comment

Uh oh!

kkharbas commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 9, 2026

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

carlosdelest left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

carlosdelest Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

carlosdelest Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

kkharbas commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kkharbas commented Feb 11, 2026

Uh oh!

elasticsearchmachine commented Feb 18, 2026

Uh oh!

kkharbas commented Feb 18, 2026

Uh oh!

elasticsearchmachine commented Feb 18, 2026

Uh oh!

kkharbas commented Feb 19, 2026

Uh oh!

carlosdelest left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kkharbas commented Jan 21, 2026 •

edited

Loading

kkharbas commented Jan 22, 2026 •

edited

Loading

kkharbas commented Feb 10, 2026 •

edited

Loading