ES|QL - dense_vector support for COUNT, PRESENT, ABSENT aggregator functions by carlosdelest · Pull Request #139914 · elastic/elasticsearch

carlosdelest · 2025-12-22T19:08:18Z

Adds support for COUNT, PRESENT, ABSENT aggregator functions for the dense_vector type.

…ense-vector-agg-functions' into enhancement/esql-support-dense-vector-agg-functions

elasticsearchmachine · 2025-12-22T19:08:52Z

Hi @carlosdelest, I've created a changelog YAML for you.

github-actions · 2025-12-22T19:10:38Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

…ttps://github.com/carlosdelest/elasticsearch into enhancement/esql-support-dense-vector-agg-functions

…ort-dense-vector-agg-functions

…ort-dense-vector-agg-functions # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/Absent.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/Count.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/Present.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/VerifierTests.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/aggregate/AbsentErrorTests.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/aggregate/CountErrorTests.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/expression/function/aggregate/PresentErrorTests.java

carlosdelest · 2026-01-07T17:03:30Z

@elasticsearchmachine run elasticsearch-ci/bwc-snapshots-part3

…nctions

elasticsearchmachine · 2026-01-08T06:31:18Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

ioanatia · 2026-01-09T15:31:56Z

.../src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/CountOverTime.java

                "cartesian_shape",
                "date",
                "date_nanos",
+                "dense_vector",


don't throw tomatoes at me, but we don't have any CSV tests for absent_over_time/count_over_time/present_over_time 🙈

but since these inherit from absent/count and present I am not 100% sure they are needed?

No 🍅s, thanks for pointing that out. I added tests and data in 5d287e0

Actually, it's not that straightforward to implement aggregations over time. I've decided to open a new issue for supporting them in #140522

ioanatia · 2026-01-09T15:45:24Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/Count.java

        if (field.foldable()) {
            if (field instanceof Literal l) {
-                if (l.value() != null && (l.value() instanceof List<?>) == false) {
+                if (l.value() != null && ((l.value() instanceof List<?>) == false || l.dataType() == DENSE_VECTOR)) {


can we get a csv test for this? I think this is the right behaviour, but I'd like a test to catch any regression.
for example, the following returns 3 as we are dealing with a multi-valued field:

ROW a = [1,2, 3] | STATS b = count(a)

but if we deal with a dense vector we should be returning 1, which is the case in your PR but we need a test:

ROW a = [1,2, 3]::dense_vector | STATS b = count(a)

returns 1 which is correct.

Yeah, we should be pretty careful with these. A paranoid level of csv tests is good here. i haven't checked what you have, but this is absolutely a place where too many tests is better than just enough.

That makes sense - I added a CSV test for this case in 528e2a5

carlosdelest · 2025-12-22T19:09:32Z

...sql/compute/src/main/java/org/elasticsearch/compute/aggregation/CountAggregatorFunction.java

 import java.util.List;

 public class CountAggregatorFunction implements AggregatorFunction {
-    public static AggregatorFunctionSupplier supplier() {


Refactored into a named class so it can be overriden by subclasses

carlosdelest · 2026-01-12T09:43:54Z

.../src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/CountOverTime.java

                "cartesian_shape",
                "date",
                "date_nanos",
+                "dense_vector",


No 🍅s, thanks for pointing that out. I added tests and data in 5d287e0

carlosdelest · 2026-01-12T09:44:28Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/Count.java

        if (field.foldable()) {
            if (field instanceof Literal l) {
-                if (l.value() != null && (l.value() instanceof List<?>) == false) {
+                if (l.value() != null && ((l.value() instanceof List<?>) == false || l.dataType() == DENSE_VECTOR)) {


That makes sense - I added a CSV test for this case in 528e2a5

This reverts commit b891cdf.

This reverts commit 2277a99.

This reverts commit 6a4cd10.

This reverts commit 0336ac0.

This reverts commit 5d287e0.

…ort-dense-vector-agg-functions # Conflicts: # x-pack/plugin/esql/qa/testFixtures/src/main/resources/dense_vector.csv-spec # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

…i-project-tests * upstream/main: (23 commits) Fix `testAckListenerReceivesNacksIfPublicationTimesOut` (elastic#140514) Reduce priority of clear-cache tasks (elastic#139685) Add docs and tests about `StreamOutput` to memory (elastic#140365) ES|QL - dense_vector support for COUNT, PRESENT, ABSENT aggregator functions (elastic#139914) Add release notes for v9.2.4 release (elastic#140487) Add release notes for v9.1.10 release (elastic#140488) Add conncectors release notes for 9.1.10, 9.2.4 (elastic#140499) Add parameter support in PromQL query durations (elastic#139873) Improve testing of STS credentials reloading (elastic#140114) Fix zstd native binary publishing script to support newer versions (elastic#140485) Add FlattenedFieldBinaryVsSortedSetDocValuesSyntheticSourceIT (elastic#140489) Store fallback match only text fields in binary doc values (elastic#140189) [DiskBBQ] Use the new merge executor for intra-merge parallelism (elastic#139942) ESQL: introduce support for mapping-unavailable fields (elastic#140463) Add ESNextOSQVectorsScorerTests (elastic#140436) Disable high cardinality tests on release builds (elastic#140503) ESQL: TRange timezone support (elastic#139911) Directly compressing `StreamOutput` (elastic#140502) ES|QL - fix dense vector enrich bug (elastic#139774) Use CrossProjectModeDecider in RemoteClusterService (elastic#140481) ...

…nctions (elastic#139914)

carlosdelest added 9 commits December 22, 2025 14:15

Add Absent support

9e880b8

Add Present support

43925c1

Add Count support

5c7d1d7

Fix error tests

509e166

Fix OverTime function types

357c2a7

Fix OverTime function types

5eadd02

Fix Count and CountGroupingAggregatorFunction

7ef0c95

Spotless

755ddf1

Merge remote-tracking branch 'carlosdelest/enhancement/esql-support-d…

e2348da

…ense-vector-agg-functions' into enhancement/esql-support-dense-vector-agg-functions

carlosdelest added >enhancement Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch Team:Search - Relevance The Search organization Search Relevance team :Search Relevance/ES|QL Search functionality in ES|QL v9.4.0 labels Dec 22, 2025

Update docs/changelog/139914.yaml

8408f96

elasticsearchmachine and others added 5 commits December 22, 2025 19:16

[CI] Auto commit changes from spotless

287a3e1

Add javadoc

9ca4ea9

[CI] Auto commit changes from spotless

f262ced

Fix test

c06bd5d

Merge branch 'enhancement/esql-support-dense-vector-agg-functions' of h…

0aa2a2a

…ttps://github.com/carlosdelest/elasticsearch into enhancement/esql-support-dense-vector-agg-functions

carlosdelest mentioned this pull request Dec 23, 2025

ES|QL - Support PRESENT, ABSENT, COUNT aggregation functions for dense_vector #135688

Closed

carlosdelest and others added 7 commits December 23, 2025 08:59

Merge remote-tracking branch 'origin/main' into enhancement/esql-supp…

6ef398b

…ort-dense-vector-agg-functions

Fix count for dense_vectors

e9eabe1

Fix tests

b35d054

Fix merge

7286a44

[CI] Auto commit changes from spotless

ea705c4

Fix tests

86d8edc

Merge branch 'main' into enhancement/esql-support-dense-vector-agg-fu…

7c58600

…nctions

carlosdelest added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jan 8, 2026

carlosdelest marked this pull request as ready for review January 8, 2026 06:30

elasticsearchmachine removed Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search - Relevance The Search organization Search Relevance team labels Jan 8, 2026

carlosdelest requested review from a team, afoucret, ioanatia and kkharbas January 8, 2026 06:33

ioanatia approved these changes Jan 9, 2026

View reviewed changes

afoucret approved these changes Jan 12, 2026

View reviewed changes

carlosdelest added 2 commits January 12, 2026 10:30

Add literal counting tests

528e2a5

Add over time data and tests

5d287e0

carlosdelest commented Jan 12, 2026

View reviewed changes

carlosdelest added 11 commits January 12, 2026 12:25

Fix capabilities

0336ac0

Fix mapping

6a4cd10

Fix tests

2277a99

Fix tests

b891cdf

Revert "Fix tests"

21a2a04

This reverts commit b891cdf.

Revert "Fix tests"

45f31c2

This reverts commit 2277a99.

Revert "Fix mapping"

14b7221

This reverts commit 6a4cd10.

Revert "Fix capabilities"

6d58ee5

This reverts commit 0336ac0.

Revert "Add over time data and tests"

424f896

This reverts commit 5d287e0.

Remove dense_vector support from over time functions

ad0baea

carlosdelest merged commit d2f5917 into elastic:main Jan 12, 2026
35 checks passed

spinscale pushed a commit to spinscale/elasticsearch that referenced this pull request Jan 21, 2026

ES|QL - dense_vector support for COUNT, PRESENT, ABSENT aggregator fu…

1178037

…nctions (elastic#139914)

Conversation

carlosdelest commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Dec 22, 2025

Uh oh!

github-actions bot commented Dec 22, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

carlosdelest commented Jan 7, 2026

Uh oh!

elasticsearchmachine commented Jan 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

carlosdelest commented Dec 22, 2025 •

edited

Loading