[ES|QL] Use avg metric for AMD default metric by limotova · Pull Request #141331 · elastic/elasticsearch

limotova · 2026-01-27T08:35:38Z

This commit changes the behavior of the default metric for
aggregate_metric_double in ES|QL to always use the average regardless
of what the default metric is mapped to.

Note that this avg is not the same as the average you would get if you
ran STATS avg(agg_metric), because the aforementioned is the average
of all docs, but the avg metric is an average on a per-doc basis.

Part of #128356

This commit changes the behavior of the default metric for aggregate_metric_double in ES|QL to always use the average regardless of what the default metric is mapped to. Note that this avg is not the same as the average you would get if you ran `STATS avg(agg_metric)`, because the aformentioned is the average of all docs, but the avg metric is an average on a per-doc basis.

gmarouli

Nice :)

x-pack/plugin/esql/qa/testFixtures/src/main/resources/inlinestats.csv-spec

x-pack/plugin/esql/qa/testFixtures/src/main/resources/k8s-timeseries.csv-spec

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateMetricDoubleBlockLoader.java

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/40_tsdb.yml

elasticsearchmachine · 2026-02-04T21:10:51Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateMetricDoubleBlockLoader.java

elasticsearchmachine · 2026-02-11T18:50:23Z

Hi @limotova, I've created a changelog YAML for you.

gmarouli

Looks very promising, I added some questions that will help me better understand the code before I give the LGTM :)

...ugin/esql/qa/testFixtures/src/main/java/org/elasticsearch/xpack/esql/CsvTestsDataLoader.java

gmarouli · 2026-02-12T09:16:24Z

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateMetricDoubleBlockLoader.java

+                return values;
+            }
+            var sortedValues = leafReader.getSortedNumericDocValues(fieldName);
+            return DocValues.unwrapSingleton(sortedValues);


I am not very familiar with the different doc values data structure, so it's not clear to me what we do here.

It looks like:

We try to retrieve leafReader.getNumericDocValues(fieldName) first, why is that? Which cases do we have where this is null?

Then if it's null, we move to leafReader.getSortedNumericDocValues(fieldName), I was under the impression that we used only this before.

Then we DocValues.unwrapSingleton(sortedValues), this method returns null if the doc values are not singleton, are we sure that this will always be a singleton?

Can you elaborate a little bit, if it's just me then I will learn something new, if we think it's very specific for this case we can add some comments.

I don't really have any answers here - I don't know the specifics of this myself, iirc i think it usually goes through the null route (SortedNumericDocValues), but i believe this was how it was written in the very very first iteration and I didn't change it
I think @martijnvg might have a better understanding..?

Looking at the original code, we used to use DocValues.getSortedNumeric(context.reader(), subfieldName(getFieldName(), metric)); (see reference)

I am wondering if we can use this directly in the reader, see comment.

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateMetricDoubleBlockLoader.java

gmarouli · 2026-02-13T08:55:00Z

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateMetricDoubleBlockLoader.java

+            NumericDocValues sumValues = getNumericDocValues(sumFieldType, context.reader());
+            NumericDocValues valueCountValues = getNumericDocValues(countFieldType, context.reader());


More context in comment, I was thinking something like that:

Suggested change

NumericDocValues sumValues = getNumericDocValues(sumFieldType, context.reader());

NumericDocValues valueCountValues = getNumericDocValues(countFieldType, context.reader());

NumericDocValues sumValues = DocValues.getSortedNumeric(context.reader(), sumFieldType.name());

NumericDocValues valueCountValues = DocValues.getSortedNumeric(context.reader(), countFieldType.name());

gmarouli · 2026-02-25T16:34:23Z

@limotova #141877 & #142135 are merged now, can you bring this up to date and see if we can finish this up too?

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateMetricDoubleBlockLoader.java

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateMetricDoubleFieldMapper.java

…tra advanceExact call

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateMetricDoubleFieldMapper.java

gmarouli

LGTM (assuming ci did not detect anything relevant), I only had a nit to suggest.

This commit changes the behavior of the default metric for aggregate_metric_double in ES|QL to always use the average regardless of what the default metric is mapped to. Note that this avg is not the same as the average you would get if you ran `STATS avg(agg_metric)`, because the aforementioned is the average of all docs, but the avg metric is an average on a per-doc basis. Part of elastic#128356

elasticsearchmachine added the v9.4.0 label Jan 27, 2026

limotova force-pushed the use-avg-metric-amd branch from e768f1d to 1a810a6 Compare January 28, 2026 07:25

limotova force-pushed the use-avg-metric-amd branch from 1a810a6 to d594cb2 Compare January 28, 2026 07:47

limotova requested a review from gmarouli January 28, 2026 08:52

gmarouli reviewed Jan 29, 2026

View reviewed changes

limotova added 7 commits February 2, 2026 23:44

Merge branch 'main' into use-avg-metric-amd

63aae18

undo most capability changes and adjust test name

f2b4159

Merge branch 'main' into use-avg-metric-amd

8fa7853

drop a few more capabilities

2bf078a

fix generative test

aee49c5

Merge branch 'main' into use-avg-metric-amd

f90754b

remove conditional adding of k8s-downsampled index

26f6f2a

gmarouli mentioned this pull request Feb 4, 2026

[PoC] Deprecate default metric in aggregate_metric_double and use average instead. #141825

Closed

4 tasks

limotova mentioned this pull request Feb 4, 2026

[ES|QL] Add support for avg as default AMD metric #140011

Closed

limotova marked this pull request as ready for review February 4, 2026 21:02

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Feb 4, 2026

limotova added the :StorageEngine/ES|QL Timeseries / metrics / PromQL / logsdb capabilities in ES|QL label Feb 4, 2026

elasticsearchmachine added Team:StorageEngine and removed needs:triage Requires assignment of a team area label labels Feb 4, 2026

kkrik-es reviewed Feb 5, 2026

View reviewed changes

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateMetricDoubleBlockLoader.java Outdated Show resolved Hide resolved

kkrik-es reviewed Feb 5, 2026

View reviewed changes

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateMetricDoubleBlockLoader.java Outdated Show resolved Hide resolved

This was referenced Feb 5, 2026

Replace default_metric with average in the aggregate_metric_double #128356

Open

Support aggregate_metric_double in binary operators =, !=, <=, >=, < ,> #142094

Closed

limotova added 2 commits February 10, 2026 10:21

Merge branch 'main' into use-avg-metric-amd

333987e

remove cast and change advanceExact

70dba76

limotova added the >enhancement label Feb 11, 2026

Update docs/changelog/141331.yaml

6270715

limotova requested review from gmarouli and kkrik-es February 11, 2026 20:13

Merge branch 'main' into use-avg-metric-amd

193be65

gmarouli reviewed Feb 12, 2026

View reviewed changes

limotova and others added 2 commits February 12, 2026 10:57

remove unused csv test utility function and add exception

5beb9ca

[CI] Auto commit changes from spotless

ffc1015

gmarouli reviewed Feb 13, 2026

View reviewed changes

limotova added 3 commits February 25, 2026 15:36

Merge branch 'main' into use-avg-metric-amd

544699a

get SortedNumericDocValues and test avg when sum and count are missing

46598f0

update capability

5be56c6

kkrik-es reviewed Feb 26, 2026

View reviewed changes

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateMetricDoubleBlockLoader.java Outdated Show resolved Hide resolved

kkrik-es reviewed Feb 26, 2026

View reviewed changes

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateMetricDoubleFieldMapper.java Outdated Show resolved Hide resolved

use TrackingNumericDocValues, move AMD_DEFAULT into switch, remove ex…

a75681c

…tra advanceExact call

limotova requested review from gmarouli and kkrik-es February 27, 2026 11:25

kkrik-es approved these changes Feb 27, 2026

View reviewed changes

gmarouli reviewed Feb 27, 2026

View reviewed changes

...in/java/org/elasticsearch/xpack/aggregatemetric/mapper/AggregateMetricDoubleFieldMapper.java Outdated Show resolved Hide resolved

gmarouli approved these changes Feb 27, 2026

View reviewed changes

limotova added 2 commits February 27, 2026 08:08

change switch statement

09721cf

Merge branch 'main' into use-avg-metric-amd

4b8f9e0

limotova merged commit e9e4844 into elastic:main Feb 28, 2026
35 checks passed

limotova deleted the use-avg-metric-amd branch February 28, 2026 02:51

		NumericDocValues sumValues = getNumericDocValues(sumFieldType, context.reader());
		NumericDocValues valueCountValues = getNumericDocValues(countFieldType, context.reader());

Conversation

limotova commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gmarouli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Feb 4, 2026

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Feb 11, 2026

Uh oh!

gmarouli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gmarouli Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

limotova Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

gmarouli Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gmarouli Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmarouli commented Feb 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gmarouli left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

limotova commented Jan 27, 2026 •

edited

Loading

gmarouli Feb 13, 2026 •

edited

Loading

gmarouli left a comment •

edited

Loading