So we're blindly adding all available dimensions to each metric? I thought we want to get the ones stored along each metric, per tsid.

For METRICS_INFO, we collapse the per tsid result into a row per metric. If a metric has different sets of attributes for different tsids, we'll return the union of all attributes. But we still need to start with extracting the _tsids, and getting the metric name and dimensions for it. Not all metrics in a data stream will have the same set of dimensions.

Right, we discussed that with Kostas and I made the corresponding changes. Now we scan the docs (one doc per _tsid) and gather metadata from them.

For METRICS_INFO, we collapse the per tsid result into a row per metric. If a metric has different sets of attributes for different tsids, we'll return the union of all attributes. But we still need to start with extracting the _tsids, and getting the metric name and dimensions for it. Not all metrics in a data stream will have the same set of dimensions.

We needed a way to deduplicate rows by _tsid. The existing aggregation operators (FirstDocIdGroupingAggregatorFunction) operate within STATS and produce aggregated results. Here we need filtering only. Influenced by FilterOperator.

The query TS k8s | METRICS_INFO | STATS metric_count = COUNT_DISTINCT(metric_name) was failing with:

[STATS metric_count = COUNT_DISTINCT(metric_name)] requires the [@timestamp] field, 
which was either not present in the source index, or has been dropped or renamed

After METRICS_INFO, the data is no longer time series data but metadata with a fixed set of output columns. The @timestamp field doesn't exist in the METRICS_INFO output so we are defaulting to Aggregate instead of TimeSeriesAggregate.

Ah, now I know where to fix this for PROMQL, where I'm seeing the same issue, thanks!

Q: If I understand correctly, the unit is part of the signature metric. In what case would this become multi-valued, if the signature metric is expected to generate two rows?

Unit is multi-valued only if there are conflicting definitions within a data stream. If there are conflicting units for backing indices within the same data stream, the unit will be multi-valued.

In the 2nd example here if the datastream was the same (k8s), the result would be a single line with multi-valued unit.

Here is another example:
Input: Data stream k8s, one metric request.size. Backing index .ds-k8s-2026.01.01-000001 has unit: "bytes"; backing index .ds-k8s-2026.02.01-000001 has unit: "kilobytes" (e.g. mapping changed over time).

Result: Same (metric_name, data_stream) → one row. The signature’s units set is {bytes, kilobytes}, so the unit column is multi-valued.

Thanks for the explanation, I might doing something wrong but for the following

{
  ".ds-metrics-unit-conflict-2026.02.17-000002": {
    "mappings": {
      "_data_stream_timestamp": {
        "enabled": true
      },
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "metric": {
          "properties": {
            "name": {
              "type": "keyword",
              "time_series_dimension": true
            }
          }
        },
        "total_cost": {
          "type": "double",
          "meta": {
            "unit": "eur"
          },
          "time_series_metric": "counter"
        }
      }
    }
  },
  ".ds-metrics-unit-conflict-2026.02.17-000003": {
    "mappings": {
      "_data_stream_timestamp": {
        "enabled": true
      },
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "metric": {
          "properties": {
            "name": {
              "type": "keyword",
              "time_series_dimension": true
            }
          }
        },
        "total_cost": {
          "type": "double",
          "meta": {
            "unit": "eur"
          },
          "time_series_metric": "counter"
        }
      }
    }
  },
  ".ds-metrics-unit-conflict-2026.02.17-000001": {
    "mappings": {
      "_data_stream_timestamp": {
        "enabled": true
      },
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "metric": {
          "properties": {
            "name": {
              "type": "keyword",
              "time_series_dimension": true
            }
          }
        },
        "total_cost": {
          "type": "double",
          "meta": {
            "unit": "usd"
          },
          "time_series_metric": "counter"
        }
      }
    }
  }
}

Response

POST _query
{
    "query": "TS metrics-unit-conflict | METRICS_INFO"
}

I get two rows

  "values": [
    [
      "total_cost",
      "metrics-unit-conflict",
      "eur",
      "counter",
      "double",
      "metric.name"
    ],
    [
      "total_cost",
      "metrics-unit-conflict",
      "usd",
      "counter",
      "double",
      "metric.name"
    ]
  ]

Yeah, this should produce a single row with the multi-valued unit. I'm checking this now, thank you!

@kpatticha , it was a bug with the final grouping of data. The initial grouping takes place on data nodes and the final one on a coordinator node. That final reduction was implemented wrongly and I've already pushed the fix. Thanks for finding it!

I pulled the latest changes and now I only get one row but with one unit

  "values": [
    [
      "total_cost",
      "metrics-unit-conflict",
      "usd",
      "counter",
      "double",
      "metric.name"
    ]
  ]

Here is the gist of what I used: https://gist.github.com/kpatticha/745d4bce942f04c6bdb35bdbf549387e

same applies for the time_series_metric

Hey @kpatticha ! Could you execute the following query, please, to check where your data had landed? I was grappling with the same issue yesterday and it turned out that all the data, even after _rollover, appeared in the 00001 backing index with USD. Check my setup: https://gist.github.com/leontyevdv/3222cf2b0996494b5706cc6afe49ddb6

It's not very convinient to wait 1m between rollover and the next POST operation but 1m is a minimum value allowed and I didn't find the other way to write data to the new backing index so far.

GET {{baseUrl}}/metrics-unit-conflict/_search
Authorization: {{auth}}
Content-Type: application/json

{
  "query": {
    "match_all": {}
  }
}

Can we change to HashImplFactory.newBytesRefHash to use BytesRefSwissHash when available.

Can we also handle the ordinals block, but we should do it in a follow-up.

We should track memory usage for these maps, but we can address that in a follow-up.

why do we need these channel checks?

Can we add input/output attributes like we do in aggregate mode?

Can we pass the scratch?

Can we pass the scratch?

I think metadataSource should have values for every row?

Can we use one map for all the modes?

Can we pass the mode to the operator and use it instead? I think this covers both final and intermediate.

We should track more than just the shallow size, but we can handle that in follow-up PRs since this one is already quite large.