Support shapes in ST_CENTROID_AGG by craigtaverner · Pull Request #141657 · elastic/elasticsearch

craigtaverner · 2026-02-02T14:00:55Z

ES|QL has an aggregation ST_CENTROID_AGG, which currently only works on points (geo and cartesian), but should work on shapes (geo and cartesian). The _search API supports aggs over shapes, and this PR expands that capability to ES|QL, making use of common code in the CentroidCalculator.

Key Implementation Details

Uses CentroidCalculator from the server module to compute weighted centroids
When shapes of different dimensions are mixed, only the highest dimension contributes (polygons > lines > points)
Currently only implements source values aggregators (doc-values aggregators cancelled as they require additional infrastructure changes)

Fixes #104658

AI-Assisted: This PR was assisted by Cursor using Claude Opus 4.5.
Usage: Used to generate the initial solution by building on existing code (aggs/centroid and esql/centroid)

This first commit was written entirely by Cursor. I will do cleanup commits following this based on my review.

elasticsearchmachine · 2026-02-02T14:01:20Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2026-02-02T14:01:52Z

Hi @craigtaverner, I've created a changelog YAML for you.

github-actions · 2026-02-02T18:04:24Z

🔍 Preview links for changed docs

docs/reference/query-languages/esql/kibana/docs/functions/st_centroid_agg.md

github-actions · 2026-02-02T18:04:25Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

…ional type'

iverase · 2026-02-06T09:38:39Z

Would it be possible to push the function to the data loaders? If I understand correctly we are copying the full doc values into pages which feels very inefficient.

…sticsearch into st_centroid_agg_shapes

craigtaverner · 2026-02-09T17:04:38Z

Would it be possible to push the function to the data loaders? If I understand correctly we are copying the full doc values into pages which feels very inefficient.

Certainly, that is the plan. I wanted to do that as a followup PR, since it is a performance optimization, and relies on newer infrastructure I did not want to involve in this PR.

ncordon

It looks good to me. I left a few questions for my own understanding

ncordon · 2026-02-09T10:52:06Z

...rg/elasticsearch/compute/aggregation/spatial/SpatialCentroidShapeSourceValuesAggregator.java

+/**
+ * This aggregator calculates the centroid of a set of cartesian shapes.
+ * It is assumed that the cartesian shapes are encoded as WKB BytesRef.
+ * This requires that the planner has NOT planned that shapes are loaded from the index as doc-values, but from source instead.


This is because we don't have support for geo / cartesian shapes in doc values right?

We do have support actually, and the doc-values for shapes includes the actual centroid, so it would be much, much better to load that, but does require more complex code, so I wanted to write that as a followup PR.

ncordon · 2026-02-09T11:22:56Z

...aggregation/spatial/SpatialCentroidCartesianShapeSourceValuesAggregatorFunctionSupplier.java

+
+  @Override
+  public String describe() {
+    return "spatial_centroid_cartesian_shape_source of valuess";


There's a typo valuess here that is also present in other ST aggregators, so I would correct them also in this pr if possible

This is in generated code. I assume the code generator is pluralizing the word by adding an s, but the original variable already is plural. Tricky!

Looks like this has been there since Jan 2024! Very old code.

ncordon · 2026-02-09T12:21:22Z

...ute/src/main/java/org/elasticsearch/compute/aggregation/spatial/CentroidShapeAggregator.java

+         * When mixing shapes of different dimensional types, only the highest dimensional type contributes.
+         */
+        public void add(double x, double dx, double y, double dy, double weight, DimensionalShapeType shapeType) {
+            if (Double.isFinite(x) == false || Double.isFinite(y) == false || Double.isFinite(weight) == false || weight <= 0) {


In what cases could this happen?

I think this is defensive coding, but I will double-check that it is not required for null values, which we needed some effort to support for ST_EXTENT_AGG.

ncordon · 2026-02-09T14:26:02Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/spatial_shapes.csv-spec

+FROM countries_bbox
+| WHERE id IN ("NOR", "SWE", "FIN")
+| STATS centroid=ST_CENTROID_AGG(shape)
+;
+
+centroid:geo_point
+POINT (19.306472392149942 63.98699665426565)
+;


Do we have any way to check these results are correct?

Remind me: we treat this as a multi polygon?

Yes, the centroid over a single geometry-collection or multi-polygon is the same as the centroid over many documents, each containing only a single polygon.

ncordon · 2026-02-09T15:57:40Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/spatial_shapes.csv-spec

+;
+
+centroidFromMixedCartesianPointsAndShapes
+required_capability: st_centroid_agg


I think we can drop this capability right? If st_centroid_agg_shapes is there it means this other one is also supported.

But up to you if you want to be more explicit, which this is achieving 😄

It seems historically we just keep adding new required_capability lines, but you are correct we can compact to the most restrictive one. There is in fact a bug where if there are many capabilities and one fails, the error/warning is reporting the first listed capability as the failing one, which is wrong.

ncordon · 2026-02-09T16:49:15Z

...ernalClusterTest/java/org/elasticsearch/xpack/esql/spatial/SpatialCentroidAggregationIT.java

+
+    @Override
+    public void testStCentroidAggregationWithShapes() {
+        assertStCentroidFromIndex("index_geo_shape", 1e-9);


How do we set up this tolerance? Why not lower or higher?

I've found from many other test cases (including in the _search/agg framework) that 1e-9 is usually the threshold of accuracy we achieve in our tests. Actually, if we aggregate over much larger datasets, we accumulate larger errors, so this number kind-of matches the dataset sizes that our randomized testing tends to generate.

ncordon · 2026-02-09T17:00:52Z

...st/java/org/elasticsearch/xpack/esql/expression/function/aggregate/SpatialCentroidTests.java

+        Stream.of(
+            MultiRowTestCaseSupplier.geoShapeCasesWithoutCircle(1, 1000, IncludingAltitude.NO),
+            MultiRowTestCaseSupplier.cartesianShapeCasesWithoutCircle(1, 1000, IncludingAltitude.NO)


Why are we avoiding shapes with circles?

Circles are only partially supported by the whole ES/Lucene stack, so we mostly don't include them in tests. It might be nicer to try get full support, but there is not really any priority for that.

ncordon · 2026-02-09T17:15:37Z

...rc/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/SpatialCentroid.java

+            case DataType.GEO_SHAPE -> new SpatialCentroidGeoShapeSourceValuesAggregatorFunctionSupplier();
+            case DataType.CARTESIAN_SHAPE -> new SpatialCentroidCartesianShapeSourceValuesAggregatorFunctionSupplier();


Shouldn't we throw an exception just in case we have doc values with geo / cartesian shapes? Which in principle should never happen because of the planner but just as a safeguard

I want to add support for EXTRACT_SPATIAL_CENTROID (the optimization mentioned in a few other comments) and in the meantime I just did not care about the fieldExtractPreference. But you are right it would be better to write the current check now, and throw exceptions on everything but NONE and STORED.

…nd cartesian_shape (#142528) The the PR at #141657 we added support for ST_CENTROID_AGG over geo_shape and cartesian_shape. However, this loads from source, which is inefficient since these centroids are actually saved in the doc-values representations. This PR switches to load from doc-values. One thing to keep in mind is that we already load bounds from doc-values for ST_EXTENT_AGG, and it is completely reasonable for the same query to do both ST_EXTENT_AGG and ST_CENTROID_AGG, in which case we want to load both centroid and bounds at the same time. The solution expands on the existing code for loading bounds into a multi-value like block. For bounds we were loading either 4 or 6 integers (for cartesian or geo extents). Now we have the following combinations: * Geo-Bounds as 6 integers: top, bottom, negLeft, negRight, posLeft, posRight * Cartesian-Bounds as 4 integers: min-x, max-x, max-y, min-y * Centroid as 4 doubles: x, y, weight, shape-type * Geo-Bounds and centroid as 10 doubles: top, bottom, negLeft, negRight, posLeft, posRight, x, y, weight, shape-type * Cartesian-Bounds and centroid as 8 doubles: min-x, max-x, max-y, min-y, x, y, weight, shape-type So we moved from 2 combinations to 6, and since centroid weights were stored in doc-values as double, while centroid coordinately and extents were stored as quantized ints, we also moved the conversion from int to double to the block-loader rather than the aggregator. Since the doc-values header does not include more useful data, it is unlikely we will expand from this combination to anything more complex in future. Fixes #142640

…nd cartesian_shape (elastic#142528) The the PR at elastic#141657 we added support for ST_CENTROID_AGG over geo_shape and cartesian_shape. However, this loads from source, which is inefficient since these centroids are actually saved in the doc-values representations. This PR switches to load from doc-values. One thing to keep in mind is that we already load bounds from doc-values for ST_EXTENT_AGG, and it is completely reasonable for the same query to do both ST_EXTENT_AGG and ST_CENTROID_AGG, in which case we want to load both centroid and bounds at the same time. The solution expands on the existing code for loading bounds into a multi-value like block. For bounds we were loading either 4 or 6 integers (for cartesian or geo extents). Now we have the following combinations: * Geo-Bounds as 6 integers: top, bottom, negLeft, negRight, posLeft, posRight * Cartesian-Bounds as 4 integers: min-x, max-x, max-y, min-y * Centroid as 4 doubles: x, y, weight, shape-type * Geo-Bounds and centroid as 10 doubles: top, bottom, negLeft, negRight, posLeft, posRight, x, y, weight, shape-type * Cartesian-Bounds and centroid as 8 doubles: min-x, max-x, max-y, min-y, x, y, weight, shape-type So we moved from 2 combinations to 6, and since centroid weights were stored in doc-values as double, while centroid coordinately and extents were stored as quantized ints, we also moved the conversion from int to double to the block-loader rather than the aggregator. Since the doc-values header does not include more useful data, it is unlikely we will expand from this combination to anything more complex in future. Fixes elastic#142640

Support shapes in ST_CENTROID_AGG

3496244

This first commit was written entirely by Cursor. I will do cleanup commits following this based on my review.

craigtaverner added >enhancement :Analytics/Geo Indexing, search aggregations of geo points and shapes Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL labels Feb 2, 2026

elasticsearchmachine added the v9.4.0 label Feb 2, 2026

Update docs/changelog/141657.yaml

f979711

elasticsearchmachine and others added 3 commits February 2, 2026 14:10

[CI] Auto commit changes from spotless

3fec77c

Fix changelog

1cb58f8

Fix SpatialCentroidTests

8120990

craigtaverner added 4 commits February 3, 2026 11:30

Added integration tests to validate behaviour and license checks

0353746

Merge remote-tracking branch 'origin/main' into st_centroid_agg_shapes

e3a43d2

Merge branch 'main' into st_centroid_agg_shapes

12e2361

Surprisingly points have a higher error than shapes

cecd7d6

craigtaverner requested review from iverase and ncordon February 4, 2026 14:36

craigtaverner added 4 commits February 4, 2026 16:28

Moved tests, added cartesian tests, and fixed terminology for 'dimens…

c27254a

…ional type'

Merge remote-tracking branch 'origin/main' into st_centroid_agg_shapes

0830a0b

Add applies to for the new types support

c1440be

Merge branch 'main' into st_centroid_agg_shapes

8ef3a01

craigtaverner added 5 commits February 9, 2026 12:04

Added applies_to to the types table

6158713

Merge branch 'st_centroid_agg_shapes' of github.com:craigtaverner/ela…

a0e9fce

…sticsearch into st_centroid_agg_shapes

Merge remote-tracking branch 'origin/main' into st_centroid_agg_shapes

be5e6c9

Remove redundant if statements

e48f204

Centroid for shapes uses common code for geo and cartesian

7085a7c

Partially revert removal of redundant if statements

2eac7a7

Remove duplicated code for points too

1a0a5c3

ncordon approved these changes Feb 9, 2026

View reviewed changes

craigtaverner added 2 commits February 9, 2026 19:08

Error on unsupported fieldExtractPreference

feeeab3

Merge remote-tracking branch 'origin/main' into st_centroid_agg_shapes

b37ae28

craigtaverner merged commit 43ed3f7 into elastic:main Feb 9, 2026
35 checks passed

This was referenced Feb 9, 2026

Enhance ST_CENTROID_AGG to read centroids from doc-values #142178

Open

Extract centroid from doc values for ST_CENTROID_AGG over geo_shape and cartesian_shape #142528

Merged

craigtaverner mentioned this pull request Feb 18, 2026

Optimize ST_CENTROID_AGG over shapes using doc-values #142640

Closed

		case DataType.GEO_SHAPE -> new SpatialCentroidGeoShapeSourceValuesAggregatorFunctionSupplier();
		case DataType.CARTESIAN_SHAPE -> new SpatialCentroidCartesianShapeSourceValuesAggregatorFunctionSupplier();

Conversation

craigtaverner commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Feb 2, 2026

Uh oh!

elasticsearchmachine commented Feb 2, 2026

Uh oh!

github-actions bot commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

github-actions bot commented Feb 2, 2026

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

iverase commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

craigtaverner commented Feb 9, 2026

Uh oh!

ncordon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

craigtaverner commented Feb 2, 2026 •

edited

Loading

github-actions bot commented Feb 2, 2026 •

edited

Loading

iverase commented Feb 6, 2026 •

edited

Loading